Catch bad data
before your customers do.
Field-level evals for LLM-powered web extraction pipelines. Compare page-extracted outputs against trusted labels, catch schema-valid JSON that is still wrong, and see which fields broke before they ship.
Plugs into your existing stack.
Run field-level experiments on every pipeline you ship.
Experiments give each critical field its own grader. Choose from LLM graders, fuzzy string matching, numeric ranges, and more. Find regressions before they flow downstream.
Use cases
Label ground truth once.
Import HTML or capture a web snapshot, then pair each page with the JSON you expect. Write it yourself, or click the values on the page with the Visual Labeler.
Renovated Leslieville semi on a walkable stretch of Queen East.
Pick the best accuracy per dollar.
Run models, prompts, and pipelines against the same Truth Set and see accuracy per dollar side by side. The expensive one is often not the accurate one.
| Prompt | Acc. | $/1K | Acc/$ |
|---|---|---|---|
| Baseline | 88% | $1.95 | 45.1 |
| + Few-shot | 93% | $2.10 | 44.3 |
| + Schema-anchored | 97% | $1.95 | 49.7 |
Winner: Haiku 4.5 (tuned) — 97% accuracy at $1.95 / 1K, beating GPT-5.5 (96% @ $7.25) for ~4× less. More expensive ≠ more accurate.
Catch cost spikes and schema drift in production.
Stream cost per page, schema adherence, and null rates from production. When a pattern starts drifting, promote the worst pages straight into a Truth Set.
Production telemetry by URL pattern
| URL pattern | Pages | Cost | Trend | Schema |
|---|---|---|---|---|
| /products/* | 9,841 | $0.021/page | 99.1% | |
| /search/* | 5,212 | $0.088/page | 92.4% | |
| /reviews/* | 2,633 | $0.047/page | 96.8% | |
| /category/* | 3,094 | $0.019/page | 99.4% |
Promote /search?q=running+shoe to 'Catalog' Truth Set?
Own your infra. Use it anywhere.
Drop Truths.dev into CI, batch jobs, or agent loops. Define what correct looks like once, then gate every deploy on field accuracy.
# score one output against ground truth
result = truths.evaluate(
truth_set_id="ts_trailrunner_pages",
output=extractor.run(url),
)
# gate the deploy on field accuracy
assert result.verdict != "failed"
# compare pipelines, monitor production
truths.arena.compare(["exp_prompt_v6", "exp_prompt_v7"])
truths.auditor.events.create(result.telemetry)Why teams choose Truths.dev
Chat eval platforms score conversations. Scraping tools get you the page. Neither tells you whether the JSON you extracted from that page is right, or what it cost you to be wrong.
| Generic LLM evals | Scraping monitors | In-house scripts | Truths.dev | |
|---|---|---|---|---|
| Field-level JSON accuracy | Partial | — | Manual | ✓ |
| Missing vs incorrect failure types | — | — | Hard | ✓ |
| Visual DOM-to-JSON debugging | — | — | — | ✓ |
| Model / prompt / pipeline Arena | Partial | — | Manual | Arena |
| Token cost per page in production | Partial | — | Manual | ✓ |
| Production schema adherence | — | Partial | Manual | ✓ |
| Promote failures into Truths | — | — | — | ✓ |
Common questions
Quick answers about Truth Sets, evals, Arena, Auditor, and how Truths fits your extraction stack.
What is Truths.dev, and how is it different from schema validation?
Truths.dev audits LLM-powered web extraction by scoring extracted JSON against labeled correct values. Schema validation only checks shape and types; Truths scores each field, handles exact numbers, fuzzy text, nulls, and custom tolerances, and separates missing and incorrect values.
What is a Truth Set, and what do I need to get started?
A Truth Set is your golden dataset for one extraction task: a schema, labeled examples, and per-field match rules. Import records from your existing pipeline or capture web snapshots, label a handful of pages, then reuse the set across Experiments, Arena, and production audits.
Does Truths.dev replace my scraper, browser agent, or backend?
No. Truths sits downstream of extraction. Keep your current stack for fetching and parsing, whether that is Playwright, Firecrawl, Browser Use, your own backend, or something else. Use the API to trigger eval runs, fetch field-level verdicts, and gate deploys on accuracy thresholds.
How do Experiments and Arena help me improve extraction quality?
Experiments run your extractor against a Truth Set to establish field-level accuracy. Arena compares models, prompts, and pipelines by field accuracy, failure type, cost, and latency, so you can improve the fields that matter instead of chasing aggregate pass/fail.
What does Auditor monitor, and what do you do with that data?
Auditor receives lightweight telemetry from live extraction runs: token usage, cost, model and prompt version, URL pattern, schema health, null rates, latency, retries, and extraction status. Your extraction data stays yours: we only store data for running evals, never sell it, never use it to train models, and support deletion requests. Read the privacy policy for more detail.