Skip to main content

​
Scraping at Scale

When one page isn’t enough — pages 1..N, whole sites, media-heavy catalogs.

​
Batch — pages 1..N / many URLs

webrain_batch runs one schema over many URLs using concurrent tabs (crawl4ai arun_many + MemoryAdaptiveDispatcher style):
op variants:
  • fetch — read visible text
  • extract — run a CSS/XPath schema (zero-LLM)
  • interact — run an async JS interaction in parallel tabs (click Load-More / infinite-scroll / form fill), then optionally extract a schema — one call replaces N serial agent loops
  • screenshot — save full-page PNGs to dir
Optional cdp_urls (list) fans the batch across N CDP backends round-robin — per-proxy isolation in one call, no subagents. Optional output persists the full payload to disk (survives temp-file GC between turns).
webrain_batch requires a browser backend — if no browser is running it errors fast (5s timeout).

​
Spider — whole sites

webrain_spider crawls from a seed URL with BFS (or DFS / BestFirst):
Highlights:
  • allow / deny URL-regex filters prune /cart, /login, images at crawl time
  • autothrottle — adaptive per-domain delay: speeds up on fast servers, doubles on a blocked/error page, capped by autothrottle_max_ms
  • crawldir — checkpoint/resume: persists {queue, seen} every checkpoint_every pages; re-run with the same crawldir to continue. Deleted on clean finish.
  • crawl_timeout_secs — hard wall-clock cap
  • respect_robots, keywords (BestFirst scoring), no_content (link-only fast path)

​
Sitemap — discover everything first

webrain_sitemap follows robots.txt Sitemap: → sitemap_index → leaf sitemaps → every <loc>. Pure HTTP, no browser:
Feed the URLs into webrain_batch for a full crawl.

​
Validation & filtering

  • webrain_validate_urls — HEAD-then-GET check; filters 404s/5xx before a batch
  • webrain_bm25 — keep only top-k relevant items after extraction