An enterprise customer asked for evaluation evidence

What is actually happening

The request looks like a request for a score, and it is not. Whoever sent it has to satisfy a reviewer, and a reviewer is checking whether a method exists that produces the same kind of answer next quarter, run by someone other than the person who built it. A pass rate with no method behind it answers none of that, and a high one invites the next question rather than closing it.

Four things survive that review, and they are all documents. A written method, published and versioned, so the reviewer can read what was done rather than take your word for it. A dataset with a version number, so the figures quote something fixed: Braintrust tracks every change to a dataset so that experiments can pin to specific versions, which is the behaviour we rely on where that tool is in use.[Braintrust] A judge (a model prompted to score outputs against a written rubric, calibrated against human labels) with its agreement stated honestly. And a report that arrives on a schedule rather than when someone asks.

The judge is where most submissions are weakest, because the statistic that gets quoted is the flattering one. Arize writes that a 75 to 90 percent[src] match between judge labels and human labels indicates strong alignment and readiness to scale.[Arize AI] Hamel Husain writes that raw agreement can be misleading when classes are imbalanced, and that the human labels should be treated as ground truth with the judge's true positive rate and true negative rate reported separately.[Hamel Husain] Reporting both is what makes the figure credible to someone who has read either page.

The fourth thing is the limits, written down. A document that claims less and states its own bounds reads as more reliable, not less, and a reviewer who finds an unstated bound stops trusting the rest of the page.

What to do this week

Day one is a question, not work. Enterprise requests arrive compressed, and "send us your evals" can mean a security review, a procurement checklist, a model risk question or one engineer who wants to know if you measure anything. The document you write for a risk function is not the document you write for their platform team, and asking costs an email.

Day two is an inventory of what you can already evidence, with dates. Most teams can evidence more than they think and claim less than they fear. What is missing becomes days three and four rather than a gap in the covering note.

Day three versions the dataset, because everything quoted in the document has to point at something fixed. Day four calibrates the judge against human labels: Galtea writes that a judge must be run against a labelled gold set, with the gap between its verdicts and the human labels measured and the prompt iterated until the gap closes, and recommends versioning judge prompts alongside application code.[Galtea] Day five sends the package, and the limits go in the package rather than in a follow-up. Our methodology is published for exactly this reason, and the report protocol and judge protocol can be sent as they are.

What happens without evals

The published example we can point to is journalism rather than an operator's account, and we describe it as reported. The Markup reported on 29 March 2024 that New York City's MyCity chatbot told users that landlords do not need to accept Section 8 vouchers and that an employer can take a cut of workers' tips, both of which are unlawful.[The Markup] That is a journalistic investigation and not an incident report by the operator, so we attribute no root cause to it and draw no engineering conclusion from it.

What it illustrates is the reviewer's actual fear, in public. The question behind the evidence request is not whether your pass rate is high. It is whether anyone would know before a customer did. A method, a dataset and a dated report are how that question gets an answer that holds up, and the sample report shows the shape of the document without any client data in it.

Which plan fits and why

Up to three features at $4,400 a month fits the usual version of this request, because an enterprise review rarely stops at one feature. It covers three features, 1,500 dataset items across them, 12 scheduled runs a month, a 5,000 request production sample and a call every two weeks. Two of our six guarantees are the ones a reviewer will care about: monthly drift (a change in how a feature behaves from one run to the next, measured against your dataset) report by the 5th business day, and judge recalibrated against human verdicts at least monthly.

The honest limit: the document covers what the dataset covers and nothing else. Agreement statistics describe the calibration set they were measured on, not your whole production traffic, and drift outside the dataset does not appear in the report. A threshold (the number you and we write down as the line a score must not cross) is an agreement about when we raise a flag, not a promise about what a model will do. The full bounds are on what we cannot do, every external figure we cite is listed on the evidence page, and if you are an agency answering this for a client, Agency partner is quoted on client volume.