> ## Documentation Index
> Fetch the complete documentation index at: https://docs.webrayn.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Structured Extraction

> From a bare page to clean JSON without hand-written selectors.

# Structured Extraction

Webrain turns an unknown page into structured JSON without you writing a single
selector. The zero-LLM discovery pipeline is:

## 1. Discover the container

```text theme={null}
webrain_autoschema   → candidate base-selectors with occurrence counts
```

`webrain_autoschema` detects repeated container patterns on the page and
returns candidate base-selectors — e.g. `.product`, `li.item`. Zero LLM.

## 2. Probe the structure

```text theme={null}
webrain_eval   → probe descendant tags/classes + samples → pick fields
```

Run small JavaScript to inspect what lives inside a container before you
declare fields.

## 3. Extract

`webrain_extract_json` builds a JSON array from a base selector + field
selectors:

```json theme={null}
{
  "base_selector": ".product",
  "fields": [
    { "name": "title", "selector": "h3 a", "type": "text" },
    { "name": "price", "selector": ".price", "type": "text" },
    { "name": "href", "selector": "a", "type": "attr", "attr": "href" }
  ]
}
```

Set `adaptive: true` to auto-relocate the container when the base selector
matches nothing (site redesigned) — it finds elements still containing ≥2 of
the field selectors.

## Other extractors

| Tool                    | Use for                                                    |
| ----------------------- | ---------------------------------------------------------- |
| `webrain_extract_regex` | emails, phones, prices, dates, IPs + custom patterns       |
| `webrain_table`         | every HTML `<table>` on the page → JSON rows               |
| `webrain_get_jsonld`    | schema.org JSON-LD / microdata — free, zero cost           |
| `webrain_bm25`          | BM25 relevance filter — keep only the top-k relevant items |
| `webrain_clean`         | strip nav/footer/social noise → clean text blob            |

## Token discipline

<Warning>
  **Never reach for `webrain_get_html` for page text.** `webrain_snapshot`,
  `webrain_clean`, `webrain_eval`, and the extractors return text/structure far
  cheaper. `get_html` is the **LAST resort** — only when the task explicitly
  asks for raw markup.
</Warning>

## The full recipe for paginated catalogs

```text theme={null}
1. webrain_navigate(seed)         → read `links` + `challenge`
2. webrain_eval                   → discover pagination hrefs structurally
                                   (same-prefix numeric links + next/prev)
3. derive urls (page range)
4. webrain_autoschema             → container selector
5. webrain_eval                   → descendant tags/classes + samples → fields
6. webrain_batch(op=extract, urls, base_selector, fields, concurrency=8)
```

Proven flow — no class assumptions, no `get_html`.