Skip to main content

​
Challenges & Anti-Bot

Gated pages are the #1 reason scrapes fail. Webrain surfaces the gate explicitly and handles it with a persistent profile + real Chrome + session.

​
Read challenge on every navigate

After webrain_navigate(url), check the challenge field in the response:
  • challenge: null → page loaded normally → extract.
  • challenge: cloudflare_challenge (title “Just a moment…”, _cf_chl*, cf-turnstile) or challenge: blocked (403/forbidden) → the page is gated.
obscura CANNOT pass interactive challenges — no layout engine means the challenge iframe never renders, and its V8 watchdog kills the challenge script. lightpanda is in the same boat. Only real Chrome works.

​
The native way (persistent profile + real Chrome + session)

  1. Start real Chrome on a persistent profile: webrain launch <service> <profile> <url> or webrain_session(op=login, service, profile, url) — native vault + TOTP login that auto-waits challenges and claims Turnstile/reCAPTCHA/hCaptcha widgets. Do not reload-spam — that resets the proof.
  2. On a 2FA/approval gate, login returns waiting_for_human:true — the human acts in the headed browser, then call login again.
  3. Re-attach webrain to that Chrome: CDP_URL=http://127.0.0.1:9222 (or webrain_session(op=open, cdp_url=...)). Same profile → session/cookies shared.
  4. webrain_navigate the protected URL → challenge: null, authenticated → extract normally.
No Python sidecar. Interactive Turnstile/hCaptcha the native path can’t claim need a human in the headed browser (2FA/approval gates return waiting_for_human:true). Obscura with --stealth may pass these — always verify via the challenge field before trusting the result. Set the consent cookies once, then every tab skips the dialog:
Log in once in real Chrome (Turnstile/Cloudflare auto-solve), export, and replay into the batch browser:
  1. webrain_session(op=cookies) / webrain cookies --port 9222 --out — export from the live authenticated Chrome.
  2. webrain_session(op=setcookies) on the session connection.
  3. webrain_batch without cdp_urls — set + batch must share one connection, because obscura isolates cookie jars per CDP connection.
Session cookies die on Chrome restart — export on the live authenticated browser, before closing it.