What is actually happening
You are asking which model serves your users better, and the material in front of you answers a different question. A public benchmark reports a fixed task, chosen by its authors, scored against their definition of a good answer. That is a real measurement of a real thing, and it is not your feature, your prompt, your inputs or your users. Nothing in it is dishonest. It simply is not about you.
We add no comparison of our own, and that is deliberate. We publish no measurements of any model, and the evidence page lists every external figure the site uses with its source. What we build instead is the thing that can answer your question, which is a measurement on your own cases.
That thing has a published shape. Weights and Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions.[Weights & Biases] LangSmith documents four evaluator types, human review, deterministic rule-based code functions, LLM-as-judge (a model prompted to score outputs against written rules, calibrated against human labels) in reference-free and reference-based variants, and pairwise comparison of two application versions.[LangChain] That last one is the shape of your question exactly, and it needs a golden dataset (a fixed set of labelled inputs and expected outputs, versioned, that every run scores against) before it can run at all.
The half of the decision that scores never touch is the lifecycle. Three providers publish three different notice commitments, and they are three separate promises by three companies rather than an industry norm. Anthropic says customers with active deployments get at least 60 days[src] before a publicly released model is retired, and its page says partner platforms set their own schedules.[Anthropic] OpenAI says at least 6 months[src] for a generally available model, at least 3 months for specialized variants, and that preview models may be retired with much shorter notice, such as 2 weeks, with no minimum committed.[OpenAI] Google says Gemini preview models may be used in production but will be deprecated with at least 2 weeks[src] notice.[Google] A winner on a preview id is a different commitment from a winner on a general release, whatever the scores say.
What to do this week
Day one is writing, not running. Most stalled model decisions are stalled because nobody has written the criterion down, so every new opinion is admissible and none of them settle anything. One sentence naming what the winner must do better, on which inputs, turns an argument into a test.
Days two and three build the only artifact that outlasts this decision. Traces come from your own traffic, spread across the cases you actually serve rather than the interesting ones. Your domain expert labels them, not us: Hamel Husain writes that in most organisations one or two key individuals hold the judgment that matters to the product, and that a judge is aligned iteratively against that person's own critiques.[Hamel Husain] The rubric (the written rules the judge applies, in your own words, agreed with your expert) is written the same day, because a judge with no rubric is just another opinion.
Day four runs both candidates against one pinned dataset with the same scorer (a function that turns one output into a number or a pass and fail verdict) set, so the only variable is the model. Day five is manual and then administrative. Read the cases where the two candidates disagree, because that is where the decision actually lives, and then read the lifecycle page for each candidate before signing up to it. Our dataset, scorer and judge protocols describe each step in full.
What happens without evals
We have no cited case for this situation. Our evidence set contains no published post-mortem of a team that picked the wrong model, and we will not invent one to fill the section.
What we can describe is the mechanism, which is plain enough. Without a dataset, the decision gets made on demos, on a benchmark table, on price, or on whoever argued last. None of those is wrong as an input, and none of them is a measurement of your feature. The cost shows up twice: once when nobody can say what was gained, and again on the next decision, because a choice made this way leaves nothing behind that the following choice can reuse. A dataset leaves something behind. Galtea also warns against updating ground truth to match model behaviour, which is the discipline that keeps a comparison honest once a favourite emerges.[Galtea]
Which plan fits and why
The eval baseline at $1,800 one-time is the right size for a decision. It builds the golden dataset from your real failures and representative inputs, selects the scorers, calibrates the judge against human verdicts with the agreement reported, agrees the thresholds with you, and hands back a first report with current scores and gaps. One of our six guarantees sets the clock: first eval baseline within 10 business days of access. The fee is credited to the first month of any plan started within 60 days, and the plans page has the exit terms.
The honest limit: the baseline covers one feature and includes no reruns after it. A model comparison is a snapshot of two candidates on one date, and providers keep shipping, as the change log shows month by month. If the answer has to stay current, a monthly plan is what keeps it current, and if this decision is really a deadline in disguise then read migrating from one provider to another instead.