Your AI feature can change every time the model does. Our job is to catch it before your users do.

A monthly service that owns your evals: the dataset, the scorers, the judge, the reruns on every model update, and the report.

See plans See what changed this month

63 tracked provider changes . 45 sourced items . 0 measurements of our own

Who this is for

An engineering lead who owns an AI featureAn update changed behaviour and a user noticed before you did.Someone to own the reruns and say pass or fail.
A head of AI with several featuresYou cannot compare two models or two prompts objectively.A calibrated suite and a report you can share.
A CTO at a small companyAn enterprise customer asked for evaluation evidence.A document and a process, not a demo.
A studio shipping AI features for clientsHandover and retainers, and nobody owns quality after launch.Per-client baselines and white-label reports.

The anchor

The three latest model changes

Each one cites the provider's own announcement and carries one recommended action. If a model you run is named, this is the week to rerun.

Newest first

GoogleDeprecation18 September 2026

gemini-2.5-pro, gemini-2.5-flash

Google limited access to Gemini 2.5 models to projects that are actively using them. The entry points developers to Gemini 3.5 Flash-Lite or Gemini 3.8 Flash instead.

Chat assistant: A forced move changes the model behind every answer. Google's own documentation says most production apps should use a specific stable model, and hedges only that stable models usually do not change.
RAG: When the generating model is replaced, the answer is scored again for faithfulness, which Ragas defines as the share of claims in a response that can be inferred from the retrieved context. It scores grounding in that context, not factual truth.

Plan a migrationCheck whether any project still calls a Gemini 2.5 model id, and open a migration ticket for the ones that do before new projects lose access.

Source: Google, Gemini API changelog . verified 2026-09-22

MetaNew version3 September 2026

muse-voice-transcribe-1.0

Meta made muse-voice-transcribe-1.0 available on the Meta Model API for streaming speech to text with speaker diarization. Meta lists the price as $3.00 per 1,000 minutes and offers a WebSocket streaming endpoint and an HTTP batch endpoint.

Summarisation: A transcription step feeds the summariser its input, so the text being summarised changes even though the summarising model did not. Galtea warns against updating ground truth to match model behaviour when the outputs shift.
Extraction: If fields are pulled out of a transcript, a new transcription model changes the input to every fixture. Braintrust tracks every change to a dataset so experiments can pin to specific versions, which is what keeps a before and after run comparable.

Rerun the suiteIf a transcription step feeds a downstream feature, run your existing audio fixtures through this model and compare the transcripts before switching any pipeline.

Source: Meta, Build with Muse Voice Transcribe on Meta Model API . verified 2026-09-22

OpenAINew version3 September 2026

gpt-6-astra

OpenAI released GPT-6 Astra, which it describes as its most capable model for end to end work, covering reasoning, coding, computer use and tools. It is available in the Responses API and the Chat Completions API.

Agent: A different model plans and calls tools differently. Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal.
Chat assistant: Running the assistant on the new model beside the one you serve is pairwise comparison, one of the four evaluator types LangSmith documents, alongside human review, rule based code functions and LLM as judge.
Summarisation: If a model from this family also runs as your judge, the scoring side is checked against human labels before its verdicts carry weight. Arize writes that a 75 to 90 percent match between judge labels and human labels indicates strong alignment and readiness to scale.

Rerun the suiteAdd gpt-6-astra as a second model in your next scheduled suite run alongside the one you serve today, and read the differences before changing any default.

Source: OpenAI, OpenAI API changelog . verified 2026-09-22

See all 63 tracked changes

Plans

Start with the eval baseline at $1,800, credited to your first month. Then monthly coverage. Every limit is on the plan page.

All five plans, with scope and limits

How it works

The first four weeks, step by step

Twelve months

What providers shipped

Tracked provider changes, Oct 2025 to Sep 2026

New version
Weight refresh
Deprecation
Pricing, context and tool changes share these three colours and always carry a label.

The same timeline as a list, 61 changes

2026-09-18 Google, Deprecation: gemini-2.5-pro, gemini-2.5-flash
2026-09-03 Meta, New version: muse-voice-transcribe-1.0
2026-09-03 OpenAI, New version: gpt-6-astra
2026-09-02 Google, New version: gemini-3.8-flash
2026-09-02 OpenAI, Tool change: slow_down, server_is_overloaded
2026-09-01 Anthropic, New version: claude-fable-5-1, claude-mythos-5-1
2026-08-26 OpenAI, Deprecation: whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize
2026-08-25 Meta, New version: muse-image-1.0
2026-08-21 OpenAI, Pricing: gpt-5.6-sol
2026-08-20 Anthropic, Deprecation: temperature, top_p, top_k
2026-08-13 Google, New version: gemini-3.7-flash
2026-08-12 Meta, New version: meta-models/Muse-Glimmer-30B
2026-08-05 Meta, New version: muse-spark-1.2, muse-spark-1.2-contributor
2026-08-05 OpenAI, Context change: gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna
2026-07-30 OpenAI, Pricing: gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna
2026-07-24 Anthropic, New version: claude-opus-5, claude-opus-4-7
2026-07-21 Google, New version: gemini-3.6-flash, gemini-3.5-flash-lite
2026-07-09 Meta, New version: muse-spark-1.1
2026-07-08 Meta, Pricing: muse-spark-1.1
2026-06-30 Anthropic, New version: claude-sonnet-5
2026-06-29 Anthropic, Tool change: claude-opus-4-6
2026-06-24 Google, Tool change: gemini-3.5-flash
2026-06-24 OpenAI, Weight refresh: chat-latest
2026-06-05 Anthropic, Deprecation: claude-opus-4-1-20250805
2026-06-03 OpenAI, Deprecation: v1/prompts, Evals platform, Agent Builder
2026-05-29 OpenAI, Tool change: prompt_cache_retention
2026-05-28 Anthropic, New version: claude-opus-4-8
2026-05-19 Google, New version: gemini-3.5-flash
2026-04-30 Anthropic, Context change: claude-sonnet-4-5-20250929, claude-sonnet-4-20250514
2026-04-24 OpenAI, New version: gpt-5.5, gpt-5.5-pro
2026-04-16 Anthropic, New version: claude-opus-4-7
2026-04-14 Anthropic, Deprecation: claude-sonnet-4-20250514, claude-opus-4-20250514
2026-04-01 Google, Pricing: Flex, Priority
2026-03-18 Google, Tool change: Built-in Tools and Function Calling Combination, Grounding with Google Maps
2026-03-17 OpenAI, New version: gpt-5.4-mini, gpt-5.4-nano
2026-03-13 Anthropic, Context change: claude-opus-4-6, claude-sonnet-4-6
2026-03-13 OpenAI, Tool change: gpt-5.4
2026-03-09 Google, Deprecation: gemini-3-pro-preview
2026-03-05 OpenAI, Context change: gpt-5.4, gpt-5.4-pro
2026-02-24 OpenAI, New version: gpt-5.3-codex
2026-02-19 Anthropic, Deprecation: claude-3-7-sonnet-20250219, claude-3-5-haiku-20241022
2026-02-19 Google, New version: gemini-3.1-pro-preview, gemini-3.1-pro-preview-customtools
2026-02-17 Anthropic, New version: claude-sonnet-4-6
2026-02-10 OpenAI, Tool change: Responses API
2026-02-05 Anthropic, New version: claude-opus-4-6
2026-02-03 OpenAI, Tool change: gpt-5.2, gpt-5.2-codex
2026-01-29 Anthropic, Tool change: claude-sonnet-4-5-20250929
2026-01-29 Google, Tool change: gemini-3-pro-preview, gemini-3-flash-preview
2026-01-21 Google, Weight refresh: gemini-3-pro-preview, gemini-3-flash-preview
2026-01-14 OpenAI, New version: gpt-5.2-codex
2025-12-19 Google, Tool change: total_thought_tokens
2025-12-17 Google, New version: gemini-3-flash-preview
2025-12-11 OpenAI, New version: gpt-5.2, gpt-5.2-chat-latest
2025-12-05 Google, Pricing: Grounding with Google Search
2025-11-24 Anthropic, New version: claude-opus-4-5, claude-opus-4-5-20251101
2025-11-18 Google, New version: gemini-3-pro-preview
2025-11-18 OpenAI, Deprecation: chatgpt-4o-latest, gpt-5.1-chat-latest
2025-11-13 OpenAI, New version: gpt-5.1, gpt-5.1-codex, gpt-5.1-codex-mini, gpt-5.1-chat-latest
2025-10-28 Anthropic, Deprecation: claude-3-7-sonnet-20250219
2025-10-15 Anthropic, New version: claude-haiku-4-5, claude-haiku-4-5-20251001
2025-10-06 OpenAI, New version: gpt-5-pro, sora-2, sora-2-pro, gpt-realtime-mini, gpt-audio-mini, gpt-image-1-mini

Published in full

The method

Eight protocols, versioned, cited step by step. Read them before you buy and hold us to them after.

EV-DATAv1.0

Golden dataset construction

Building and maintaining the golden dataset that an eval baseline runs against, from the first sample of real traffic to the scheduled refresh.

EV-SCOREv1.0

Scorer selection by feature type

Choosing and recording the scorers that run against a golden dataset for one feature type, and writing down what each scorer does and does not measure.

EV-JUDGEv1.0

Judge calibration and recalibration

Aligning an automated judge to your people's verdicts, reporting its agreement honestly, and recalibrating it on a fixed cycle.

EV-THRESHv1.0

Thresholds and release gating

Setting the pass and fail rules for an eval run and wiring them into your release process so a regression is caught before the change ships.

All eight protocols

What arrives monthly

The drift report

By the 5th business day: what was covered, every run and what triggered it, the per-scorer deltas against the thresholds you agreed, how well the judge agreed with your expert this month, what production sent that the dataset has no case for, and what the dataset does not cover.

See a synthetic example

What we guarantee

First eval baseline within 10 business days of access.
Every tracked provider update triggers a rerun within 2 business days of the provider's announcement.
Every flagged prompt or retrieval change is evaluated before it ships, within the plan's turnaround.
Monthly drift report by the 5th business day.
Judge recalibrated against human verdicts at least monthly.
If a regression above the agreed threshold reaches production on a change we marked as passed, the next month is free.

These six are the whole list. Nothing else on this site is a guarantee.

Questions people ask first

What does this cost?

An eval baseline is $1,800 one-time and takes 10 business days. Monthly coverage starts at $1,900 for one feature. The baseline fee is credited to the first month of any plan started within 60 days. Every price and every limit is published.

Do you replace our eval tool?

No. The tools are good and we use them. Braintrust, Langfuse, Arize Phoenix, DeepEval, promptfoo, Weights & Biases. What teams are missing is not better software, it is somebody whose job it is to keep the dataset, the judge and the reruns current.

Do you publish scores for models?

Never. This site publishes no measurement of any model. The change log restates provider announcements with their sources. Measurements we make belong to the customer whose dataset produced them.

How fast do you react to a provider update?

Your suite reruns within 2 business days of the provider's announcement, on every plan, and you get a written verdict. That is one of six written guarantees.

What can you not do?

We cannot make a model deterministic or guarantee its outputs. We measure against an agreed dataset and thresholds and report what changed. Coverage is bounded by the dataset and the plan, and every report says what the dataset does not cover.

What we cannot do

We cannot make a model deterministic or guarantee its outputs. We measure against an agreed dataset and thresholds and report what changed. We do not train or fine-tune models, and we do not replace human review for policy or legal decisions. Coverage is bounded by the dataset and the plan.

The longer version, with the reasoning

Contact

Email only, answered by a person within 24 hours on business days. Tell us the feature type, the provider and model, roughly how many requests a month, and whatever eval setup you have now.

Start the baseline Other reasons to write

Change-log entries restate provider announcements with sources and dates. Impact notes cite public practitioner reports. The site publishes no measurements of its own on any model.