Scrapetor
Ruby gem · v0.1.0 · MIT

High-performance HTML extraction for Ruby.

Scrapetor pairs a native C arena DOM with a streaming extraction engine that compiles a schema DSL directly into a single forward pass over the input. No DOM materialised on the fast path, one Ruby boundary crossing per document, and no external parser dependencies.

gem install scrapetor View on GitHub See it in action
25-27x
faster CSS selectors
vs Lexbor (#id, .class, tag)
26x
faster DOM construction
vs Lexbor (36 KB input)
16x
faster end-to-end extraction
vs Nokolexbor on listings
0
runtime gem
dependencies

What you get

A focused toolkit for scraping at scale, not a kitchen-sink HTML library. Everything is designed around one workload: read HTML, extract structured data, move on.

01

Native streaming extraction

The schema DSL compiles into a flat descriptor that the C engine executes during tokenisation. The DOM is never materialised on this path - that's where the order-of-magnitude wins come from.

02

Native arena DOM

When you do need a DOM (doc.css, mutation, traversal) you get an arena-allocated tree with class/id/tag hash indexes built during the same parse pass. #id lookups become O(1) instead of tree walks.

03

Zero runtime dependencies

No Nokogiri, no Nokolexbor, no libxml2. Pure Ruby plus a single C99 extension that compiles at install time. Just a working C compiler - that's the contract.

04

Structured data, batteries included

JSON-LD, OpenGraph, Twitter Cards, Schema.org filtering, HTML5 Microdata, and RDFa are first-class methods on every document. Page-type detection ships in the gem.

05

Honest encoding handling

Full HTML5 sniffing cascade: BOM, <meta charset>, http-equiv. Non-UTF-8 input is transcoded with replacement before parsing so one bad byte doesn't poison the document.

06

CLI, builder, SAX

A scrapetor binary for one-off extraction, a pure-Ruby builder DSL for HTML construction, a SAX streamer for token-level work - all in the same gem, no extra installs.

Performance

Numbers below are from benchmark/comprehensive.rb in the repo, measured on Apple Silicon (Ruby 2.7.8). Every workload asserts the three engines produce equivalent output before timing.

End-to-end extraction

iterations per second · higher is better
Workload Scrapetor Nokolexbor Nokogiri
Listing50 cards x 4 fields 9,360 573 171
Product detailtop + 3 reviews 30,101 11,636 2,022
Articletop + tags + sections 53,837 25,553 6,338

Raw parse throughput

megabytes per second · higher is better
Input Scrapetor Nokolexbor Nokogiri
Small170 bytes 37 18 11
Article2 KB 279 116 31
Product page3 KB 426 140 31
Listing page36 KB 3,933 154 31
Large document2.5 MB 18,434 136 31

CSS selector evaluation

iterations per second on a pre-parsed listing page · higher is better
Selector Scrapetor Nokolexbor Speedup
#mainid lookup 1,272,698 65,170 19.5x
articletag, 50 matches 1,244,279 65,122 19.1x
.product-cardclass, 50 matches 1,226,004 68,065 18.0x
#main articleid descendant, 50 matches 1,086,604 35,228 30.9x
img.product-imagetag.class, 50 matches 875,901 65,707 13.3x
.product-grid > .product-cardchild, 50 matches 754,486 60,924 12.4x
[data-sku="SKU0001"]attribute, 1 match 516,157 78,101 6.6x
.product-card .pricedescendant, 50 matches 405,062 42,870 9.5x

Allocations per extraction

live Ruby objects per call · lower is better
Workload Scrapetor Nokolexbor Nokogiri
Listing50 cards x 4 fields 363 4,710 9,501
Product detailtop + 3 reviews 96 140 596

Full report including selector micro-benchmarks: benchmark/comprehensive.rb.

Examples

Three workloads that cover roughly 90% of real-world scraping. All copy-paste, no configuration to set up.

A listing page with 50 product cards

require "scrapetor"

doc = Scrapetor::HTML(html, base_url: "https://example.com/")

result = doc.extract do
  repeated ".product-card", as: :products do
    field :title, from: ".title", clean: true
    field :price, from: ".price", type: :money
    field :url,   from: "a",      attr: :href, type: :url, normalize_url: true
    field :image, from: "img",    attr: :src,  type: :url, normalize_url: true
  end
end

result[:products].first
# => { title: "Widget 1", price: 19.99,
#       url: "https://example.com/products/widget-1",
#       image: "https://example.com/img/widget-1.jpg" }

A product detail page (top-level fields plus repeated reviews)

data = Scrapetor::HTML(html, base_url: site).extract do
  field :title,  from: ".product-title", clean: true, required: true
  field :price,  from: ".price", type: :money
  field :rating, from: ".rating", attr: "data-rating", type: :float

  repeated ".review", as: :reviews do
    field :author, from: ".author", clean: true
    field :stars,  from: ".stars", attr: "data-stars", type: :float
    field :body,   from: ".body",  clean: true
    field :date,   from: ".date",  attr: :datetime, type: :date
  end
end

Fetch and structured-data extraction

doc = Scrapetor.fetch("https://example.com/article")

doc.page_type          # => :article
doc.json_ld            # => [{ "@type" => "NewsArticle", ... }]
doc.opengraph          # => { "title" => "...", "image" => "...", ... }
doc.schema_org(type: "NewsArticle")
doc.microdata
doc.twitter_card

Architecture

Two execution paths in one extension. doc.extract(schema) takes the fast lane and never builds a DOM. Everything else uses the arena DOM, which is itself faster than Lexbor on the workloads scrapers care about.

HTML in ─┬─► Native streaming extract (C) │ `doc.extract(schema)` │ Schema runs during tokenisation. No DOM is built. │ One Ruby/C boundary crossing per document. │ └─► Native arena DOM (C) `doc.css(...)`, `doc.at(...)`, mutation, traversal. Class/id/tag indexes built during the parse pass. Zero-copy text and attribute spans into the input.

Both paths share ext/scrapetor/native/. A pure-Ruby DOM is kept as a fallback for environments where the extension can't be loaded.

Install

A working C compiler is the only system requirement. The native extension is built automatically when the gem installs.

Bundler

gem "scrapetor"

Direct

gem install scrapetor

From source

git clone https://github.com/Alaa-abdulridha/scrapetor
cd scrapetor
bundle install
rake compile
rake test

Where to go from here