In a postmortem published on 23 April 2026, Anthropic attributed six weeks of Claude Code quality reports to three product-layer changes made between 4 March and 20 April 2026, said neither its internal usage nor its evals initially reproduced the issues, and measured a 3 percent quality drop[src] from one system prompt change.[Anthropic] Anthropic states the API was not impacted and model weights were unchanged.
We repeat that last sentence everywhere we cite this incident, because the interesting part of it disappears if you let the reader assume a model got worse. Nothing in the postmortem says that.
What a product-layer change is
The layer in question is everything between your user and the model call: system prompt text, tool definitions, context assembly, truncation rules, retry logic, defaults.
That layer is code you own, or in this case code the product team owned. It ships on your release schedule, not the provider's. It does not appear in any provider changelog, because there is nothing for a provider to announce.
Compare the Anthropic change log for the same weeks. On 14 April 2026 Anthropic notified developers using Claude Sonnet 4 and Claude Opus 4 of their retirement on the Claude API, with both models retiring on 15 June 2026. On 16 April 2026 Anthropic launched claude-opus-4-7 at the same price as Claude Opus 4.6, with task budgets in beta, high resolution image input and an added xhigh effort level. Both are real, sourced entries with actions attached. Neither has anything to do with the six weeks of reports. A change log is a complete record of announcements and a partial record of what can degrade a feature.
Why the 3 percent figure is the useful part
A single verbosity prompt change, and the team's own suite put a number on it.
Hold the figure in its bounds: Anthropic's number, on Anthropic's eval suite, for one specific prompt change. It is not ours and it is not general. We publish no measurements of our own on any model, and the evidence page records the same bound on this item.
What generalises is the size. Three points on a suite is a small movement, and small movements are exactly what a threshold (the agreed number at which a score change is treated as a finding rather than as noise) set carelessly will swallow. Six weeks of user reports accumulated behind a change that a suite could quantify at three points. That gap between what people felt and what an aggregate showed is the same gap in most quality incidents.
The postmortem also says neither internal usage nor the evals initially reproduced the issues. Dogfooding is a sample of one workload, and an eval suite is a sample of the past. Both are worth having. Neither is a monitor.
Gating on the delta, not on the absolute
If a small movement is the thing to catch, the gate has to be built on change rather than on a pass mark.
Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations trigger a deploy block at any rate, while regressions (a case that passed on the previous run and fails on this one) gate on the change in fail rate, and gives the example of a faithfulness fail rate rising from 3 percent to 5 percent[src].[Galtea] That is Galtea's illustrative example in Galtea's own documentation, not a recommended threshold, and we never quote it as one. Your numbers come from your data and are agreed with you, which is what EV-THRESH sets out.
The structure is what carries over. Two classes of failure, two rules. Policy breaches block at any rate. Quality regression gates on the movement against the last passing run.
Where the check has to sit
A prompt edit is a code change, so the check belongs where code changes are reviewed.
Braintrust recommends running evals on every pull request to catch regressions before they reach production.[Braintrust] Promptfoo documents running evaluations in CI/CD to test prompt changes before they reach production and to fail the build when quality thresholds are not met.[Promptfoo] Two good tools, documenting the same placement. Our rerun protocol treats a flagged prompt or retrieval change as a trigger on every plan, and adds a pipeline gate on the platform plan.
One more habit belongs here. When a prompt change moves a score down, the cheapest response is to move the expected answer. Galtea warns against updating ground truth to match model behaviour, and writes that additions to a dataset should require code review while deletions are treated as production risk.[Galtea] A golden dataset (a fixed set of labelled inputs with expected outputs, versioned so two runs can be compared) that gets edited to agree with the latest build has stopped being a measurement. EV-DATA puts the dataset under review for that reason.
We are not claiming we would have caught this. The stated causes were in a product layer we had no access to, on a product we do not evaluate, and the operator states the API and the model weights were unaffected. What this incident supports is narrower: a prompt change can degrade a feature with no model change anywhere, and it is worth scoring on that basis.
What to take from it
Treat your system prompt as versioned code with a test suite behind it. If it is edited in a console by whoever noticed the problem, it has no history and no gate, and a three-point movement will reach production every time.
Then watch both sources of change, separately. Provider announcements are tracked in the change log with a source on every entry. Your own edits are tracked by your pipeline. Most teams watch one of those two, and the postmortem here is a clean example of the other one being the cause.