Extraction pulls structured fields out of unstructured input: an invoice into line items, a contract into dates and parties, a support email into a ticket. It is the feature type where an eval suite is cheapest to build and easiest to read, because most of what matters has a right answer that a machine can compare.
What typically breaks in extraction after a model change
The output stops conforming to the schema
An extra key appears. A number arrives as a string. A field that was an object becomes an array with one object in it. Your parser throws, or worse, it does not throw and a downstream system stores the wrong type. This is the loudest failure and the easiest to catch, which is exactly why it should never be the only thing you check.
A field starts coming back empty
The shape stays valid and one field goes null. The invoice number was on the page and the model no longer returns it. Nothing errors. Downstream, a row lands with a gap in it and someone fills it in by hand for a month before anyone connects it to a model update.
Per-field reporting is what surfaces this. An aggregate pass rate across twelve fields moves by a few points when one field collapses entirely.
A value arrives that was never in the document
The field is populated, the type is right, and the value is wrong in a specific way: a plausible date, a plausible reference number, a party name assembled from context. An exact match against the expected value catches it. A schema check does not, and a human skim-reading the output usually does not either.
Formats and normalisation shift
Dates move between formats. Currency arrives with or without a symbol. An enum comes back in a different case, or with a synonym your mapping does not contain. The information is correct and your pipeline rejects it. This is why the scorer (one automated check that turns an output into a number or a pass and fail verdict) set holds both an exact match and a normalised match, with the gap between them reported rather than hidden.
The call itself breaks without any model swap
A provider change can break extraction at the request layer. Anthropic deprecated the temperature, top_p and top_k request parameters from Claude Opus 4.7 onward, and a non-default value now returns a 400 error[src]. [Anthropic] Extraction pipelines are the ones most likely to be sending a pinned low temperature, so this class of change lands here first. Dated entries for each provider are in the change log.
Pinning to a dated snapshot delays the problem without removing it. OpenAI announced on 11 June 2026 that the gpt-5-2025-08-07 snapshot would be shut down on 11 December 2026[src]. [OpenAI] OpenAI says it gives at least 6 months[src] notice before retiring a generally available model, at least 3 months for specialized variants, and that preview models may be retired with much shorter notice, such as 2 weeks, with no minimum committed. [OpenAI] A re-pin (moving a feature from one model id or snapshot to another, then rerunning the suite before the change ships) is a change like any other, and it needs a run before it ships. See EV-RERUN.
Which scorers apply
LangSmith documents four evaluator types: human review, rule based code functions, LLM as judge (a model that scores another model's output against a written standard) in reference free and reference based variants, and pairwise comparison of two application versions. [LangChain] For extraction, the rule based code functions do most of the work, and they are the reason this feature type is cheap to cover well. DeepEval defines a golden (a precursor to a test case holding an input plus the expected result, which becomes a full test case once your application produces the actual output), which is exactly the shape an extraction dataset already has. [Confident AI] Each scorer carries its limit.
- Schema conformance code check. Catches shape, types, required keys, enum membership. Misses every question about whether the values are correct.
- Field level exact match. Catches a wrong or invented value against the expected value. Misses correct values in a different format, which is why it never runs alone.
- Field level normalised match. Catches the same value written differently, after dates, currency and casing are normalised. Misses a normalisation bug in your own mapping, so the gap against exact match is reported.
- Null rate per field. Catches a field that quietly stopped being populated. Misses a field that is populated with something wrong.
- Invented value check. Catches values absent from the source document, for fields where the source text is stored alongside the expected value. Misses inventions that happen to appear somewhere in a long document.
Free text fields are the exception. A clause summary or a reason code written in prose needs a judge, and a judge needs calibrating before its numbers count. Selection is written out in EV-SCORE.
What a baseline includes for this feature type
The eval baseline for extraction is built per field, not per document.
- Real documents from your own intake, including the scanned, the multi-page and the badly formatted ones, because those are where field accuracy falls first.
- An expected value for every field on every document, labelled by your domain expert, stored with the source text the value came from.
- The schema itself as an artifact in the dataset, versioned with it, so a schema change is visible as a dataset change.
- A normalisation map agreed in writing: date formats, currency, enum synonyms, whitespace and casing rules.
- Per-field pass rates and per-field null rates in the report, with the aggregate shown second, so one collapsing field cannot hide behind eleven healthy ones.
- threshold (the number agreed with you at which a measured drop counts as a finding on this dataset) values set per field rather than once for the feature, because an invoice total and a free text note do not deserve the same bar. Galtea recommends a tiered release gate in which blocking failures trigger a deploy block at any rate while regressions (a case that passed on the previous run and fails on this one) gate on the change in fail rate. [Galtea]
Which plan fits
Start with the eval baseline at $1,800 one-time. Extraction is the feature type where a baseline is closest to complete on its own, because the scorers that matter are rule based code checks that need no calibration and produce numbers you can read without us. The caveat is plain: the baseline is a one-time piece of work and includes no reruns, so it tells you where you stand today and nothing about next month. If the extracted fields feed billing, compliance or anything else that costs money to get wrong, One feature at $1,900 a month adds the rerun triggers and the monthly report. Promptfoo documents running evaluations in CI/CD to test changes before they reach production and to fail the build when quality thresholds are not met, which is the shape of the gate described in EV-THRESH. [Promptfoo]