The first hour after a provider announcement
Most of the work in the hour after an announcement is reading, in a fixed order. Here is the order, and how each answer maps to one of four recommended actions.
Blog
Four categories: what providers shipped and what to do about it, how the evaluation work is actually done, which tools fit which job, and public post-mortems of AI features going wrong, with sources.
Every number here carries a source id. Every provider statement traces to a change log entry with the provider's own announcement.
20 posts
- Model updates6The narrative companion to the change log: what a provider shipped and what to do about it.
- Methods6How the work is actually done: datasets, scorers, judges, thresholds.
- Tooling4Honest comparisons of eval tools, and where a tool stops and a service starts.
- Failures4Public post-mortems of AI feature regressions, with sources.
Latest
Running an eval suite in CI: what a gate does
A gate on every pull request changes review, deploy timing and who argues with whom. Here is the practical shape of it, and the class of change it cannot catch.
When price and context limits move under a tuned feature
Pricing and context-window entries look like finance news. They land in chunking, truncation and cost logic that somebody tuned once and never revisited.
What a deprecation notice actually gives you
A deprecation notice is a dated commitment from one company about one platform. Here is what Anthropic, OpenAI and Google each publish, and how to plan against it.
Reference-free scoring: what replaces the expected answer
Reference based scoring needs an expected result stored next to the input. When none exists, what replaces it is a written rubric, reference free scorers and a labelled sample your own expert graded.
What a golden dataset stops covering
The thing that ages an eval dataset is usually not a model release. It is your own users, asking things the set was never built from, while every run keeps passing.
Everything, newest first
- Model updatesThe first hour after a provider announcement
- ToolingRunning an eval suite in CI: what a gate does
- Model updatesWhen price and context limits move under a tuned feature
- Model updatesWhat a deprecation notice actually gives you
- MethodsReference-free scoring: what replaces the expected answer
- MethodsWhat a golden dataset stops covering
- ToolingEval tool or eval service: which one you need
- FailuresAnthropic's 3 percent drop, from a system prompt change
- MethodsSetting eval thresholds that gate a release
- Model updatesA quarter of tracked model changes, and what they asked for
- FailuresWhen a provider's own evals missed the degradation
- MethodsChoosing a scorer: code checks, similarity or a judge
- ToolingPromptfoo vs DeepEval: choosing a starting point
- Model updatesPinning a model id, or riding a floating alias
- FailuresIncident 1152: what the AI Incident Database records
- MethodsCalibrating an LLM judge against human verdicts
- Model updatesHow to read a provider changelog when you own a feature
- ToolingBraintrust vs Langfuse: what the docs say
- FailuresWhat The Markup reported about the MyCity chatbot
- MethodsHow many examples a golden dataset starts from
Elsewhere on the site