​

Scraping at Scale

​

Batch — pages 1..N / many URLs

{
  "op": "extract",
  "urls": ["https://example.com/products?page=1", "...&page=2"],
  "base_selector": ".product",
  "fields": [{ "name": "title", "selector": "h3", "type": "text" }],
  "concurrency": 8
}

• fetch — read visible text
• extract — run a CSS/XPath schema (zero-LLM)
• interact — run an async JS interaction in parallel tabs (click Load-More / infinite-scroll / form fill), then optionally extract a schema — one call replaces N serial agent loops
• screenshot — save full-page PNGs to dir

​

Crawl — whole sites

{
  "mode": "spider",
  "seed_url": "https://example.com",
  "max_depth": 3,
  "max_pages": 200,
  "allow": ["/product/"],
  "deny": ["/cart", "/login"],
  "autothrottle": true,
  "crawldir": "crawls/site-a"
}

• allow / deny URL-regex filters prune /cart, /login, images at crawl time
• autothrottle — adaptive per-domain delay: speeds up on fast servers, doubles on a blocked/error page, capped by autothrottle_max_ms
• crawldir — checkpoint/resume: persists {queue, seen, results} atomically every checkpoint_every pages; re-run with the same crawldir to continue (returns prior + new results, never re-fetches). Deleted on clean finish. max_pages counts the whole crawldir — raise it on resume to fetch more.
• concurrency — parallel tabs on real Chrome/obscura (default 4); single-target/lightpanda auto-serializes
• crawl_timeout_secs — hard wall-clock cap
• respect_robots, keywords (BestFirst scoring), no_content (link-only fast path)

​

Sitemap — discover everything first

webrain_crawl(mode="sitemap", url="https://example.com")
→ { urls: [...], count, sources }

​

Validation & filtering

• webrain_crawl (mode: validate) — HEAD-then-GET check; filters 404s/5xx before a batch
• webrain_extract (mode: bm25) — keep only top-k relevant items after extraction