Generation writes something new: marketing copy, a reply draft, a product description, a code snippet. There is no correct output, so there is nothing to match. What there is instead is a standard, written down by the person in your company whose judgment the product actually depends on.
What typically breaks in generation after a model change
Six patterns come up often enough to belong in every rubric (the written scoring standard applied to an output, in your expert's words) we help write. Each shows up as drift (a measured change in a feature's scores between two runs of the same dataset) the moment a fixed set of briefs is rerun.
The voice moves
The copy is still competent and it no longer sounds like your product. More hedging, more enthusiasm, longer sentences, a different register in the first line. Nobody can point at a broken rule, because the rule was never written down, and that is the whole problem. A voice that lives only in your expert's head cannot be scored, so it slips without anyone being able to say when.
The format drifts
Bullet counts change. Headers appear. Markdown arrives where your interface renders plain text. A sign-off is added. These are the cheapest failures to detect and the most frequent after any change, including your own prompt edits.
A prompt change on your side moves this as far as a model update does. In a postmortem published on 23 April 2026, Anthropic attributed six weeks of Claude Code quality reports to three product-layer changes and measured a 3 percent quality drop[src] from one system prompt change on its own eval suite. [Anthropic] Anthropic states that the API was not impacted and model weights were unchanged. The transferable lesson is that a verbosity instruction is a change that deserves a run, which is the trigger list in EV-RERUN.
Claims appear that your legal team has not approved
Open-ended generation invents. In a product description that means a feature you do not have, a comparison you cannot support, a guarantee you never made. This is a rule based code check problem, not a taste problem: the banned phrase and banned claim list belongs in code, and a match is a blocking failure rather than a score.
Outputs converge on one template
A subtler failure. Every draft opens the same way, every description follows the same three-part shape. Users who see many outputs notice before you do. If variety is part of what your feature is for, then sameness is a failure mode, and it has to be written into the rubric explicitly, because no off-the-shelf scorer (one automated check that turns an output into a number or a pass and fail verdict) looks for it.
Behaviour moves with no release on your side
Future AGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump. [Future AGI] That is a vendor's characterisation of the market. The provider's own evidence alongside it is Google's statement that a Gemini latest alias points to the most recent release of a model variation and is hot-swapped with every new release of that variation. [Google] Release cadence is public and dated: Anthropic's own release notes record Claude Opus 4.8 on 28 May 2026, Claude Opus 5 on 24 July 2026 and Claude Fable 5.1 on 1 September 2026[src]. [Anthropic] Entries and sources are in the Anthropic change log.
The judge that applies your rubric moves too
Everything interesting on this page is judged, so the judge (a model that scores another model's output against a written rubric) is part of the measurement. Future AGI writes that a judge model can move 3-5 points[src] on a fixed rubric in a single month with no change on your side. [Future AGI] A score that falls without a recalibrated judge behind it is not yet a finding about your feature.
Which scorers apply
LangSmith documents four evaluator types: human review, rule based code functions, LLM as judge in reference free and reference based variants, and pairwise comparison of two application versions. [LangChain] Generation uses the reference free judge for quality, code checks for everything with a shape, and pairwise comparison when you are choosing between two prompt versions. Each scorer is written down with its limit.
- Rubric judge scorer. Catches drift away from the standard your expert wrote, rule by rule. Misses anything the rubric does not name, and its verdicts mean nothing until it has been calibrated against human labels.
- Format and length code check. Catches structure, word ceilings, headers, markdown your interface cannot render. Misses every question of quality.
- Banned phrase and claim code check. Catches words, claims and guarantees you have decided never to publish. Misses paraphrase, so it is a floor and not a compliance review.
- Required element code check. Catches a missing call to action, a missing disclaimer, a missing product name. Misses whether any of them are any good.
- Pairwise comparison. Catches which of two versions your expert prefers when no absolute scale exists yet. Produces no standing number, so it supports a decision rather than a trend.
Calibration is the whole game here. Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes. [Galtea] Arize writes that a 75 to 90 percent[src] match between judge labels and human labels indicates strong alignment and readiness to scale. [Arize AI] Hamel Husain adds the check that matters more than the match rate: treat human labels as ground truth and report the judge's true positive rate and true negative rate separately. [Hamel Husain] The procedure is EV-JUDGE, and scorer selection is EV-SCORE.
What a baseline includes for this feature type
The eval baseline for a generation feature is mostly the work of turning taste into rules.
- Real briefs and inputs from your own product, including the thin ones, because a generator is judged hardest when it has little to work with.
- A written rubric from your domain expert, rule by rule, covering what a good output does and what it must never do. Hamel Husain writes that the judge is aligned iteratively against that person's own critiques. [Hamel Husain]
- Graded examples at both ends, so the rubric has anchors rather than adjectives. Hamel Husain starts with around 30 examples[src] and keeps going until no new failure modes appear. [Hamel Husain]
- A banned phrase and banned claim list, in code, treated as a blocking failure rather than a score.
- Format constraints as code checks: length, structure, required elements, anything your interface depends on.
- A calibrated judge with more than one agreement statistic reported, plus a threshold (the number agreed with you at which a measured drop counts as a finding on this dataset) per rubric rule, agreed with you.
Which plan fits
One feature at $1,900 a month fits a single generation feature after the baseline. It covers reruns on every tracked provider update, on every prompt change you flag, and weekly, with judge recalibration on a monthly cycle, which matters more for this feature type than for any other because the judge is doing all the interesting work. Two caveats worth stating plainly. The rubric is your standard, not ours, so the quality of every number on this feature depends on how honestly your expert wrote it down. And a rubric score is a statement about the dataset it was measured on, never a promise about the next thing the model writes. The sequence of work is described in how it works, and the closely related problem of compressing a source rather than inventing from one is on the summarisation hub.