Every tracked model change from OpenAI, Anthropic, Google and Meta, newest first, each with the provider's own announcement as its source and one recommended action. If a model you run is in here, rerun your suite.
Your AI feature changes when the model does, and the model changes on the provider's schedule. This is the list we watch so that a rerun is triggered by an announcement rather than by a user complaint.
Google limited access to Gemini 2.5 models to projects that are actively using them. The entry points developers to Gemini 3.5 Flash-Lite or Gemini 3.8 Flash instead.
Chat assistant: A forced move changes the model behind every answer. Google's own documentation says most production apps should use a specific stable model, and hedges only that stable models usually do not change.
RAG: When the generating model is replaced, the answer is scored again for faithfulness, which Ragas defines as the share of claims in a response that can be inferred from the retrieved context. It scores grounding in that context, not factual truth.
Plan a migrationCheck whether any project still calls a Gemini 2.5 model id, and open a migration ticket for the ones that do before new projects lose access.
Meta made muse-voice-transcribe-1.0 available on the Meta Model API for streaming speech to text with speaker diarization. Meta lists the price as $3.00 per 1,000 minutes and offers a WebSocket streaming endpoint and an HTTP batch endpoint.
Summarisation: A transcription step feeds the summariser its input, so the text being summarised changes even though the summarising model did not. Galtea warns against updating ground truth to match model behaviour when the outputs shift.
Extraction: If fields are pulled out of a transcript, a new transcription model changes the input to every fixture. Braintrust tracks every change to a dataset so experiments can pin to specific versions, which is what keeps a before and after run comparable.
Rerun the suiteIf a transcription step feeds a downstream feature, run your existing audio fixtures through this model and compare the transcripts before switching any pipeline.
OpenAI released GPT-6 Astra, which it describes as its most capable model for end to end work, covering reasoning, coding, computer use and tools. It is available in the Responses API and the Chat Completions API.
Agent: A different model plans and calls tools differently. Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal.
Chat assistant: Running the assistant on the new model beside the one you serve is pairwise comparison, one of the four evaluator types LangSmith documents, alongside human review, rule based code functions and LLM as judge.
Summarisation: If a model from this family also runs as your judge, the scoring side is checked against human labels before its verdicts carry weight. Arize writes that a 75 to 90 percent match between judge labels and human labels indicates strong alignment and readiness to scale.
Rerun the suiteAdd gpt-6-astra as a second model in your next scheduled suite run alongside the one you serve today, and read the differences before changing any default.
Google released gemini-3.8-flash as generally available. The entry states it brings improvements for long horizon software engineering.
Agent: Long horizon work is mostly tool calls. Arize Phoenix reconstructs two signals from a trace, tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Generation: Generated text has no single correct answer to match against, so the tempting fix when a new model writes differently is to move the expected outputs. Galtea warns against updating ground truth to match model behaviour.
Rerun the suiteQueue gemini-3.8-flash into the next comparison run against the Flash model you use now, and record the result before moving traffic.
OpenAI wrote that a rapid increase in traffic can now return a 429 error with the slow_down code, and that temporary overload can return a 503 error with the server_is_overloaded code.
Rerun the suiteCheck that your client and your eval harness treat these two codes as a signal to back off rather than as failures, then run a batch large enough to exercise that path.
Anthropic launched claude-fable-5-1 and claude-mythos-5-1 with a 1M token context window by default, 128k maximum output tokens and always on adaptive thinking, priced at $10 and $50 per million tokens. The notes state that tool_choice types any and tool are not supported and return a 400 error, and that thinking blocks are preserved only for the same model or a newer one.
Agent: tool_choice types any and tool now return an error, so the way the agent is made to call a tool changes. Arize Phoenix reads tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Extraction: Forcing a tool call is one way to fix an extractor's output shape, and that request now fails. DeepEval defines a golden as an input plus the expected result, which becomes a full test case once the application produces the actual output, so the same set reruns once the shape is settled.
Revisit the promptSearch your codebase for tool_choice values of any or tool, since those requests now fail, and check any code that replays thinking blocks across models.
OpenAI announced on August 26, 2026 that whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-transcribe-diarize are scheduled to shut down on February 26, 2027. The recommended replacements are gpt-live-transcribe and gpt-transcribe.
Summarisation: Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle. A transcript sits in that middle, so a summary scored only on its final text does not show which step changed.
Plan a migrationFind every call to one of the four ids, run a held out set of recordings through gpt-transcribe or gpt-live-transcribe, and keep both transcripts so the step after them can be compared before February 26, 2027.
Meta made muse-image-1.0 available to developers on the Meta Model API at $0.01 per image. Meta describes anchored composition for consistency across a series of images and reasoning driven edits.
Generation: There is no single correct output to match a generation prompt against, so a fixed prompt set and the recorded outputs carry the comparison. Galtea warns against updating ground truth to match model behaviour.
Rerun the suiteIf image generation sits inside a product flow, run your standing prompt set against this model and keep the outputs as a baseline for the next version.
OpenAI reduced GPT-5.6 Sol pricing to $4 per million input tokens and $20 per million output tokens. The entry states the reduced price is promotional through November 21.
Rerun the suiteUpdate the per token rates in your cost model and dashboards, and put the stated promotional end date in the budget calendar.
Anthropic released v1.0 of the Python SDK. It requires Python 3.10 or later and removes long deprecated surface, including the legacy Text Completions API and the temperature, top_p and top_k parameters on Messages methods. Anthropic's deprecations page states that passing them on v1.0 and later raises a TypeError.
Generation: This is caller code breaking with no model swap on your side. Anthropic deprecated temperature, top_p and top_k from Claude Opus 4.7 onward, and a non-default value returns a 400 error. It is a parameter deprecation; nothing is published about an effect on output quality.
Plan a migrationSearch your code for temperature, top_p and top_k on Messages calls and remove them, and pin your Python SDK version until that change is made.
Google released gemini-3.7-flash. The entry states it brings substantial improvements across software engineering.
Chat assistant: Putting the new Flash model beside the one you pin is pairwise comparison of two application versions, one of the four evaluator types LangSmith documents, alongside human review, rule based code functions and LLM as judge.
Rerun the suiteAdd gemini-3.7-flash to the comparison run you already schedule for Flash tier models, and note where its answers differ from the version you pin.
Meta released Muse-Glimmer-30B, a 30 billion parameter open weight model for local agent workloads, under the Apache 2.0 licence with a 128K token default context window. Meta lists quantized builds and support for vLLM, llama.cpp, ExecuTorch, Ollama and LM Studio.
Agent: An open weight model is pinned by you, so the swap happens when you upgrade rather than when the provider ships. Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans deep, so the comparison reads the trace and not the final answer alone.
Rerun the suiteIf you self host a Llama 4 model, run your current suite against Muse-Glimmer-30B on the same hardware and record both quality and throughput before planning a move.
Meta released muse-spark-1.2 with a 1M token context window, priced at $1.25 per million input tokens, $0.15 per million cached input tokens and $4.25 per million output tokens. Meta states the model was co-trained with its harness and trained on whole repository generation and other long running tasks.
Agent: Whole repository work and other long running tasks are scored end to end by default. Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that reasoning, tool selection and the decision path are where the interesting failures sit.
Generation: Running 1.2 and 1.1 in the same pass is pairwise comparison of two application versions, one of the four evaluator types LangSmith documents.
Rerun the suiteRun your coding and long context cases against muse-spark-1.2 and muse-spark-1.1 in the same pass, and note that the contributor tier is rate limited by tokens in a rolling window rather than by request count.
OpenAI states that Fast mode now supports long context requests exceeding 272K tokens for the GPT-5.6 models.
RAG: Dropping the chunking you added for the old ceiling changes what sits in the context window. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the score moves with the context as well as with the model.
Summarisation: Summarising a whole document rather than a set of chunks changes the expected output for every case. Galtea warns against updating ground truth to match model behaviour.
Rerun the suiteIf you split long inputs to stay under the old ceiling, run a long document case end to end before removing that chunking.
OpenAI introduced Fast mode, which it describes as up to 2.5 times faster at twice the price. The entry also states that GPT-5.6 Luna costs 80 percent less and GPT-5.6 Terra 20 percent less.
Rerun the suitePrice the three GPT-5.6 tiers against your current spend per request, then run the cheaper tier through your suite before moving any traffic to it.
Anthropic launched claude-opus-5 at $5 and $25 per million tokens, with a 1M token context window as both the default and the maximum and 128k maximum output tokens. The notes state that thinking is on by default, that disabling thinking is only allowed at effort high or below, which they call a breaking change from Claude Opus 4.8, and that Fast mode was removed for Claude Opus 4.7 so those requests return an error.
Agent: Thinking on by default, and a narrower way to turn it off, change how much the agent works out before it acts. Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal.
Chat assistant: Future AGI recommends refreshing a golden dataset immediately after a model update, monthly by default and weekly during rapid change, sampling 200-500 recent production traces stratified by intent and persona and versioning the result as a new artifact.
Generation: If a model from this family also scores your outputs, the judging side moves too. Future AGI writes that a judge model can move 3-5 points on a fixed rubric in a single month without any change on your side.
Revisit the promptFind any request that disables thinking at effort xhigh or max, since that combination is now rejected, and remove the fast mode flag from Claude Opus 4.7 calls.
Google released gemini-3.6-flash and gemini-3.5-flash-lite as generally available. The same entry deprecates the sampling parameters temperature, top_p and top_k.
Classification: Classifiers usually pin sampling parameters in the request builder, so a deprecation here is a change to caller code rather than to the model you chose. Anthropic made the same kind of change from Claude Opus 4.7 onward, where a non default temperature, top_p or top_k returns a 400 error.
Extraction: Removing sampling parameters changes the request every fixture runs through. Braintrust tracks every change to a dataset so experiments can pin to specific versions, which is what keeps the run before the edit comparable with the run after it.
Revisit the promptSearch your request builders for temperature, top_p and top_k on Gemini calls and plan their removal, then rerun the affected cases with the parameters dropped.
Meta announced Muse Spark 1.1, a multimodal reasoning model built for agentic tasks, with what it calls major gains in tool and computer use, coding and multimodal understanding. Meta states it has a 1 million token context window and launched a public preview of the Meta Model API for developer access.
Agent: Changes in tool and computer use land in the middle of the trace. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Rerun the suiteIf you track Meta models, open an account on the Meta Model API and run your existing suite once so later versions have a baseline to be compared against.
Meta put muse-spark-1.1 in public preview on the Meta Model API for United States developers, priced pay as you go at $1.25 per million input tokens and $4.25 per million output tokens. Meta lists end to end agentic workflows, advanced coding and native multimodal perception as the areas of improvement.
Rerun the suiteAdd the published input and output rates to your cost model, and note that the preview is limited to United States developers when you plan access.
Anthropic launched claude-sonnet-5 at an introductory $2 and $10 per million tokens with a 1M token context window and 128k maximum output tokens. The notes state that adaptive thinking is on by default, that manual extended thinking was removed and returns a 400 error, that non default sampling parameters return a 400 error, and that a new tokenizer produces about 30 percent more tokens.
Classification: Non default sampling parameters now return a 400 error, which breaks caller code without any model swap on your side. Anthropic first made this parameter deprecation from Claude Opus 4.7 onward.
RAG: A tokenizer that produces about 30 percent more tokens changes how much retrieved text fits in one prompt. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so a smaller retrieved context can move the score by itself.
Summarisation: The same input now costs more tokens, so what reaches the summariser can change before the model does anything. Galtea warns against updating ground truth to match model behaviour when the summaries come back different.
Revisit the promptRecalculate token budgets and cost estimates against the new tokenizer, and strip manual thinking and non default sampling parameters from requests before they start failing.
Anthropic removed fast mode for Claude Opus 4.6. Requests to claude-opus-4-6 with speed set to fast no longer run at fast speed or premium pricing: they run at standard speed, are billed at standard rates, and do not return an error. The usage.speed field on the response reports the speed used, and Anthropic says to migrate to Claude Opus 4.8 to continue using fast mode.
Plan a migrationA request that asks for fast mode on this id still succeeds, so read usage.speed in your own logs to see which speed you are getting, and move to Claude Opus 4.8 if the feature depends on it.
Google launched public preview support for the Computer Use tool in Gemini 3.5 Flash.
Agent: Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, writes that a high end to end score can hide bad tool calls, and warns that strong per step scores can still miss that the agent pursued the wrong goal.
Rerun the suiteRun your agent cases with the Computer Use tool enabled on gemini-3.5-flash and keep the recorded tool calls, not only the final answers.
OpenAI updated the chat-latest snapshot to point to the latest Instant model in ChatGPT.
Chat assistant: An alias that is repointed changes the model behind the assistant with no deploy on your side. Google documents the same mechanism for Gemini, where a latest alias points at the most recent release of a model variation and is hot swapped with every new release.
Generation: Future AGI recommends refreshing a golden dataset immediately after a model update, monthly by default and weekly during rapid change, sampling 200-500 recent production traces stratified by intent and persona.
Re-pin the versionReplace any chat-latest alias in production with a dated snapshot id, so a future repoint does not change behaviour without a deploy.
Anthropic announced the deprecation of Claude Opus 4.1, claude-opus-4-1-20250805, with retirement on the Claude API scheduled for August 5, 2026. Anthropic recommended migrating to Claude Opus 4.8.
Chat assistant: Anthropic notified developers on 5 June 2026 that Claude Opus 4.1 would be retired on 5 August 2026, with Claude Opus 4.8 as the recommended replacement. Nothing is published about how the replacement behaves differently, so a forced swap changes the model behind every answer with no provider figure to compare against.
Plan a migrationThe retirement date has passed, so any call still naming claude-opus-4-1-20250805 fails. Confirm nothing does, and record which id your feature runs on now.
OpenAI announced the deprecation of reusable prompt objects, the Evals platform and Agent Builder. The v1/prompts API and reusable prompt objects, the Evals dashboard and API, and Agent Builder are scheduled to shut down on November 30, 2026, and existing evals become read only on October 31, 2026. OpenAI says to move reusable prompt content into your application code, and points to the Agents SDK, ChatGPT Workspace Agents and Promptfoo.
Plan a migrationList every place that loads a prompt by id from v1/prompts or runs a job on the Evals platform, and put the move into application code or another runner on the calendar before October 31, 2026.
OpenAI wrote that for organizations without ZDR, its zero data retention setting, prompt_cache_retention now defaults to 24h instead of in_memory.
Rerun the suiteCheck whether your requests set prompt_cache_retention explicitly, and decide whether the new default is the one you want before your next run.
Anthropic launched claude-opus-4-8 with a 1M token context window by default on the Claude API, Amazon Bedrock, Google Cloud and Microsoft Foundry. The notes state that the effort parameter defaults to high, that the minimum cacheable prompt length was reduced to 1,024 tokens, that adaptive thinking triggers reasoning only when needed, and that mid conversation system messages are available without a beta header.
Agent: An effort parameter that defaults to high changes how much happens between the request and the answer. Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that reasoning, tool selection and the decision path are where the interesting failures sit.
Chat assistant: Mid conversation system messages change the shape of a multi turn request. LangSmith documents four evaluator types for comparing versions, human review, rule based code functions, LLM as judge in reference free and reference based variants, and pairwise comparison.
Rerun the suiteRun your suite once at the default effort of high before you override it, and re-check cache hit rates now that shorter prompts qualify for caching.
Google released gemini-3.5-flash as generally available. The same entry launches Managed Agents in public preview and an antigravity-preview-05-2026 general purpose managed agent.
Chat assistant: Moving a preview pin onto a generally available id is the direction Google's own documentation points, saying most production apps should use a specific stable model and hedging that stable models usually do not change.
Agent: A managed agent in public preview carries a shorter clock than a stable model. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Rerun the suiteMove any preview Flash pin onto the generally available id and run the suite once to see whether the outputs shifted.
Anthropic retired the 1M token context window beta, context-1m-2025-08-07, for Claude Sonnet 4.5 and Claude Sonnet 4. The beta header now has no effect on these models, and requests exceeding the standard 200k token context window return an error. Anthropic says to migrate to Claude Sonnet 4.6 or Claude Opus 4.6, where the 1M context window is included at standard pricing with no beta header required.
RAG: A smaller usable window changes how much retrieved text reaches the model. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the scorer reads what was actually passed in, not what was retrieved.
Plan a migrationFind any request that still sends the context-1m-2025-08-07 header to these two ids, then either shorten the input or move the feature to a model where the 1M window is standard.
OpenAI released GPT-5.5 as a frontier model with a 1M token context window, structured outputs and extended prompt caching, alongside gpt-5.5-pro.
Extraction: Structured outputs decide whether extracted fields validate against your schema. DeepEval defines a golden as an input plus the expected result, which becomes a full test case once the application produces the actual output, so the same goldens rerun against the new model.
RAG: A larger window invites more retrieved text into the prompt. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, and it scores grounding in that context rather than factual truth.
Rerun the suiteRun your structured output cases against gpt-5.5 and check that responses still validate against your schemas before changing the model you serve.
Anthropic launched claude-opus-4-7 at $5 and $25 per million tokens, the same price as Claude Opus 4.6. The notes state that task budgets launched in beta, that high resolution image input is supported up to 2576 pixels on the long edge, and that an xhigh effort level was added.
Agent: Task budgets and a new effort level change how much the agent does before it answers. Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans deep, so the comparison reads the trace rather than the final answer alone.
Extraction: Higher resolution image input means page and screenshot fixtures may be re-encoded, which changes the inputs themselves and not only the model. Braintrust tracks every change to a dataset so experiments can pin to specific versions.
Rerun the suiteRun the suite against claude-opus-4-7 at your current effort level first, then once at xhigh, and record what the extra effort changes before you make it the default.
On April 14, 2026, Anthropic notified developers using Claude Sonnet 4 and Claude Opus 4 of their upcoming retirement on the Claude API. Both models retired on June 15, 2026, with claude-sonnet-4-6 and claude-opus-4-8 listed as the recommended replacements.
Chat assistant: The clock on a retirement belongs to the provider. Anthropic says it gives customers with active deployments at least 60 days' notice before retiring a publicly released model, and its page states that partner platforms such as Amazon Bedrock and Google Cloud set their own schedules.
Summarisation: A recommended replacement writes summaries in its own register. Galtea warns against updating ground truth to match model behaviour, so the expected outputs stay where they are and the gap is what gets reported.
Plan a migrationExport your Claude API usage by model and confirm nothing still calls these two dated ids, since requests to them now return an error.
Google released the Built-in Tools and Function Calling Combination feature, which makes it possible to use Gemini's built-in tools alongside custom function calling tools in a single API call. The same changelog entry says Grounding with Google Maps is now supported for Gemini 3 models going forward.
Agent: Mixing built in tools with custom functions changes which tool is chosen and in what order. Arize Phoenix describes reading tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Rerun the suiteIf you combine a built in tool with your own functions in one call, record the tool calls for a set of cases before and after, so a changed selection order is visible.
OpenAI released GPT-5.4 mini and GPT-5.4 nano to the Chat Completions and Responses APIs.
Classification: Weights & Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions. That is the structure a move to a smaller id is judged on.
Rerun the suiteIf you are considering a cheaper id for routing or tagging, run the same labelled set through gpt-5.4-mini and gpt-5.4-nano and compare it against the model you use today.
Anthropic took the 1M token context window out of beta for Claude Opus 4.6 and Claude Sonnet 4.6, at standard pricing and with no beta header required. The notes state that the dedicated 1M rate limits were removed and that the media limit was raised to 600 images or PDF pages per request.
RAG: A longer window changes how much retrieved text you put in front of the model. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the score follows the context you assemble.
Summarisation: Feeding a whole document instead of chunks changes the input to every case, which makes the old run and the new run two different things unless the inputs are tracked. Braintrust tracks every change to a dataset so experiments can pin to specific versions.
Rerun the suiteRemove the 1M beta header from your clients and rerun any long context case, since the separate rate limit pool that used to back those requests is gone.
OpenAI wrote that it updated its image encoder to fix a small bug with input_image inputs in gpt-5.4. OpenAI describes it as a bug fix and publishes no new model id alongside it.
Rerun the suiteIf a feature sends images to gpt-5.4, run your image cases again after this date and keep both sets of outputs, because the id you call did not change.
RAG: A retrieval feature pinned to a preview id can lose that id on short notice. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Classification: A classifier that runs unattended is the last place a shutdown gets noticed. Google's documentation says most production apps should use a specific stable model, and hedges that stable models usually do not change.
Plan a migrationReplace any gemini-3-pro-preview pin with a current Gemini 3 id and rerun the affected suites, since the preview id no longer serves.
OpenAI released GPT-5.4 as a frontier model with a 1M token context window, tool search, computer use and compaction, alongside gpt-5.4-pro.
Agent: Tool search and computer use add steps that the final answer never shows. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
RAG: Compaction changes what the model still has in front of it when it answers. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so a compacted context is a different thing to be faithful to.
Rerun the suiteRun one long input case against gpt-5.4 before you rely on the larger window, and check how compaction changes what the model still has in front of it.
OpenAI released gpt-5.3-codex to the Responses API. The same entry added the phase parameter, which OpenAI says labels an assistant message as intermediate commentary or final answer.
Agent: Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, writes that a high end to end score can hide bad tool calls, and warns that strong per step scores can still miss that the agent pursued the wrong goal. A label that separates intermediate commentary from the final answer is read at those resolutions.
Rerun the suiteRun your agent cases against gpt-5.3-codex, and decide whether your harness should record the phase label so intermediate messages stop being scored as answers.
Anthropic launched automatic caching for the Messages API, where a single cache_control field caches the last cacheable block and the cache point moves forward as a conversation grows. The same entry retired Claude Sonnet 3.7, claude-3-7-sonnet-20250219, and retired Claude Haiku 3.5, claude-3-5-haiku-20241022, on the Claude API. Anthropic recommends upgrading to Claude Sonnet 4.6 and Claude Haiku 4.5.
RAG: When the generating model is replaced, the answer is scored again for faithfulness, which Ragas defines as the share of claims in a response that can be inferred from the retrieved context. It scores grounding in that context, not factual truth.
Plan a migrationCheck that nothing still names either retired id, since Anthropic says requests to retired models fail, and decide separately whether automatic caching should be switched on for your long conversations.
Google released gemini-3.1-pro-preview and launched a gemini-3.1-pro-preview-customtools variant.
Agent: A custom tools variant changes how the agent reaches its answer, not only what it answers. Future AGI describes scoring an agent at four resolutions, end to end, trajectory, step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal.
Rerun the suiteRun the Pro tier comparison against gemini-3.1-pro-preview, and keep the custom tools variant in a separate run so tool behaviour differences stay visible.
Anthropic launched claude-sonnet-4-6 at $2 and $10 per million tokens with a 1M token context window in beta and extended thinking support. The notes state that web search and web fetch are free when used with code execution, and that web search, web fetch, programmatic tool calling, code execution, tool search, tool use examples and the memory tool are available without a beta header.
Agent: Tools that no longer need a beta header are tools a request can now reach. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
RAG: Where web search or web fetch supplies the documents, the retrieved context changes with the tool path. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, and it scores grounding in that context, not factual truth.
Rerun the suiteRemove the beta headers you no longer need from tool requests and rerun the tool using cases, since header changes can alter which tool path a request takes.
OpenAI launched server side compaction in the Responses API, launched Skills support across local and hosted execution, and released the Hosted Shell tool with support for networking in containers.
Agent: Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle where they cannot be seen, naming reasoning, tool selection and decision path. Compaction and a new hosted tool both sit in that middle.
Rerun the suiteIf you turn on server side compaction or adopt the Hosted Shell tool, run a long session case with the spans recorded rather than only the final answer.
Anthropic launched claude-opus-4-6 at $5 and $25 per million tokens. The notes state that adaptive thinking is recommended and manual thinking is deprecated, that an effort parameter with levels low, medium, high, xhigh and max launched, that a 1M token context window is available in beta with long context pricing above 200k input tokens, and that the media limit rose from 100 to 600 images or PDF pages with the 1M context.
Agent: An effort level replaces a thinking budget you used to set by hand, which changes what happens between the request and the answer. Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that reasoning, tool selection and the decision path are where the interesting failures sit.
Generation: Moving from a manual thinking budget to adaptive thinking plus an effort level is a prompt change, and the text comes back different. Galtea warns against updating ground truth to match model behaviour.
Revisit the promptSwitch requests from manual thinking budgets to adaptive thinking plus an effort level, then rerun the cases where you previously tuned the thinking budget by hand.
Anthropic took structured outputs out of beta on the Claude API for Claude Sonnet 4.5, Claude Opus 4.5 and Claude Haiku 4.5. The notes state that the output_format parameter moved to output_config.format, and that structured outputs remain in public beta on Amazon Bedrock and Microsoft Foundry.
Extraction: The parameter that carries your schema moved, so the request changes while the fields you expect do not. DeepEval defines a golden as an input plus the expected result, which becomes a full test case once the application produces the actual output.
Classification: Where labels come back through a structured output, the shape of the response is part of the feature. Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations block a deploy at any rate, while regressions gate on the change in fail rate.
Revisit the promptMove output_format to output_config.format in your request builders and rerun the schema validation cases on both the Claude API and any partner platform you use.
Google launched support for the Computer Use tool in gemini-3-pro-preview and gemini-3-flash-preview.
Agent: Arize Phoenix describes reconstructing tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path.
Rerun the suiteBoth ids are preview models, so keep the rerun cycle short for anything built on this tool, and record which preview id each run used.
Google changed the latest aliases to point to Gemini 3 models.
Chat assistant: Google says a latest alias points at the most recent release of a model variation and is hot swapped with every new release of that variation, so an assistant served through the alias changes with no action on your side.
RAG: An answer path that reads retrieved documents is easier to reason about when the model id is explicit. Google's documentation says most production apps should use a specific stable model, and hedges that stable models usually do not change.
Generation: Future AGI writes that OpenAI, Anthropic, Google and most frontier providers update model weights without a major version bump. That is one vendor's characterisation of the market, and Google's documented alias behaviour is the provider's own side of it.
Re-pin the versionReplace any latest alias in your configuration with an explicit model id, and rerun the suites that were served through the alias to see what moved.
OpenAI released gpt-5.2-codex to the Responses API and describes it as optimized for agentic coding tasks in Codex.
Agent: Arize Phoenix describes reconstructing tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path. Those are the signals a coding agent is read on.
Rerun the suiteIf a coding or agent feature sits on a pinned Codex id, run your trajectory cases against gpt-5.2-codex before you point anything at it.
Google renamed the total_reasoning_tokens field to total_thought_tokens, saying the new name better aligns with the concept of thoughts in thinking models. The change affects v1beta.
Rerun the suiteUpdate anything that reads total_reasoning_tokens from a v1beta response, including cost dashboards and eval harness logging, then run a batch and check the field is populated.
Google launched gemini-3-flash-preview. The entry describes frontier class performance and upgraded reasoning capabilities.
Summarisation: A summariser pinned to a preview id runs on a shorter clock. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Rerun the suiteAdd gemini-3-flash-preview to your Flash tier comparison run, and keep the preview id out of production pins until a stable id ships.
OpenAI released GPT-5.2 as its flagship model, stating improvements in reasoning, vision, code generation and spreadsheet understanding, alongside gpt-5.2-chat-latest. The same day OpenAI released client side compaction for long running conversations.
Extraction: Vision and spreadsheet inputs are where extracted fields come from. DeepEval defines a golden as an input plus the expected result, which becomes a full test case once the application produces the actual output, so the document fixtures rerun unchanged against the new model.
Generation: Running the new flagship beside the snapshot you pin is pairwise comparison of two application versions, one of the four evaluator types LangSmith documents.
Rerun the suiteRun your suite against gpt-5.2 alongside the GPT-5.1 snapshot you pin, and pay attention to the vision and document cases the entry calls out.
Anthropic launched Claude Opus 4.5, which the release notes describe as its most intelligent model, combining maximum capability with practical performance. The release notes name the family alias claude-opus-4-5. Anthropic's model deprecations table lists the dated snapshot claude-opus-4-5-20251101 as Active, with retirement not sooner than 24 November 2026.
Rerun the suiteA new model is an option rather than an event, unless an alias you already call now resolves to it. Run your own dataset against the model you use today and against this one before deciding to move.
Google launched gemini-3-pro-preview. The entry describes state of the art reasoning and multimodal capabilities.
Chat assistant: A first baseline on a preview id is a baseline with a short life. Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks' notice.
Rerun the suiteSet up a first baseline run against gemini-3-pro-preview so later Gemini 3 versions have something to be compared with.
OpenAI announced the deprecation of the chatgpt-4o-latest snapshot on November 18, 2025, with a shutdown date of February 17, 2026. The recommended replacement is gpt-5.1-chat-latest.
Chat assistant: How much warning a snapshot gets depends on its class. OpenAI says it gives at least 6 months' notice before retiring a generally available model, at least 3 months for specialized variants, and that preview models may be retired with much shorter notice, such as 2 weeks, with no minimum committed.
Generation: Prompts tuned against the old snapshot produce different text on the replacement. Galtea warns against updating ground truth to match model behaviour.
Plan a migrationSearch your configuration for chatgpt-4o-latest and move those calls to gpt-5.1-chat-latest, then rerun the prompts that were tuned for the old snapshot.
OpenAI released GPT-5.1 as a flagship model optimized for steerability, code generation and agentic workflows, and released gpt-5.1-codex and gpt-5.1-codex-mini to the Responses API. The same day OpenAI launched extended prompt cache retention of up to 24 hours.
Agent: Agentic workflows are judged across the whole run. Future AGI writes that agent traces in contemporary frameworks are routinely 15-40 spans deep, naming the OpenAI Agents SDK, LangGraph, CrewAI, AutoGen and Google ADK.
Chat assistant: A family alias resolves to whatever the provider points it at. Google documents this for Gemini, where a latest alias points at the most recent release of a model variation and is hot swapped with every new release, which is the reason to pin an explicit id.
Re-pin the versionPin the exact GPT-5.1 id you intend to serve rather than a family alias, and rerun the prompts you had tuned against GPT-5 before switching.
On October 28, 2025, Anthropic notified developers using Claude Sonnet 3.7 of its upcoming retirement on the Claude API. The model retired on February 19, 2026, with claude-sonnet-4-6 listed as the recommended replacement.
Classification: A retired id stops serving, so an unattended classifier fails outright rather than quietly scoring worse. Anthropic says it gives customers with active deployments at least 60 days' notice before retiring a publicly released model, and states that partner platforms set their own schedules.
Generation: Prompts written for the retired model produce different text on the replacement. Galtea warns against updating ground truth to match model behaviour.
Plan a migrationConfirm that nothing still calls claude-3-7-sonnet-20250219, since requests to the retired id now fail, and rerun the prompts that were written for it.
Anthropic launched Claude Haiku 4.5, which the release notes describe as its fastest and most intelligent Haiku model, with near-frontier performance. The release notes name the family alias claude-haiku-4-5. Anthropic's model deprecations table lists the dated snapshot claude-haiku-4-5-20251001 as Active, with retirement not sooner than 15 October 2026.
Rerun the suiteIf you are considering this model to cut cost or latency, the comparison that matters is your own dataset scored on both models, not a published summary of either.
OpenAI released GPT-5 Pro, which it describes as a version of GPT-5 that uses more compute to think harder. The same DevDay entry launched the v1/videos API with the Sora 2 and Sora 2 Pro models, and released GPT-Realtime mini, gpt-audio-mini and gpt-image-1-mini for cost efficient speech and image work. It also launched Agent Builder, ChatKit and Trace Evals.
Generation: A release that adds generation models gives a team a new option rather than a forced change. Galtea warns against updating ground truth to match model behaviour, so the outputs you already recorded stay the comparison point when a new id is tried.
Rerun the suiteNothing moves until you adopt one of these ids. If you plan to try gpt-5-pro, run your standing prompt set against it first and keep the current model's outputs beside it.
Anthropic launched claude-sonnet-4-5-20250929, which it describes as its best model for complex agents and coding. The notes state that a new stop reason model_context_window_exceeded was added, that the memory tool launched in beta and that context editing launched in beta.
RAG: Context editing and a new stop reason change what actually reaches the model on a long input. Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, so the score follows whatever survived.
Agent: A memory tool adds calls that the final answer never shows. Arize Phoenix reconstructs tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned.
Rerun the suiteAdd handling for the model_context_window_exceeded stop reason so long inputs fail visibly, and rerun the long context cases that previously truncated.
OpenAI announced on September 26, 2025 the deprecation of gpt-3.5-turbo-instruct, babbage-002, davinci-002 and gpt-3.5-turbo-1106, with a shutdown date of September 28, 2026. The listed migration target is gpt-5.6-terra.
Classification: Older ids tend to sit in batch jobs and internal classifiers that nobody watches. OpenAI says it gives at least 6 months' notice before retiring a generally available model, at least 3 months for specialized variants, and that preview models may be retired with much shorter notice, such as 2 weeks, with no minimum committed.
Extraction: Moving an extractor onto a current model is a release like any other. Galtea recommends a tiered release gate in which blocking failures such as safety or policy violations block a deploy at any rate, while regressions gate on the change in fail rate.
Plan a migrationCheck batch jobs and older internal tools for these four ids ahead of the stated shutdown date, and move any remaining traffic to a current model.
Change-log entries restate provider announcements with sources and dates. Impact notes cite public practitioner reports. The site publishes no measurements of its own on any model.
Questions about the log
Where does this log come from?
From the providers themselves. Every entry cites a page on the provider's own domain: a changelog, a release note, a deprecation table or a pricing page. An announcement we cannot reach on a provider domain does not get an entry, whoever else reported it.
Do you test these models?
No. This site publishes no measurement of any model. The log restates what a provider announced, and the impact notes cite published practitioner guides about what that kind of change typically affects. Measurements we make belong to the customer whose dataset produced them.
What does the recommended action mean?
One of four: rerun your suite, re-pin to a fixed model id, revisit the prompt, or plan a migration. It is what a team does this week, not a prediction about your feature.
How often is this updated?
When a provider announces. We run the verifier against their changelog and deprecation pages, add the entries with sources, and rebuild. Customers on a plan get a rerun within 2 business days of the announcement.
A provider announced something and it is not here.
Either we have not reached it yet, or we could not find the announcement on the provider's own domain. Send us the provider's link and we will add it.