Skip to content
DriftCaught

Methodology

Eight protocols, published in full, each versioned and cited step by step. They are what we run for every customer on every plan. Read them before you buy, and hold us to them after.

An evaluation is only worth what its dataset, its scorers and its judge are worth. These pages say how we build each of the three, what each one catches, and what it misses.

The protocols

EV-DATAv1.0

Golden dataset construction

Building and maintaining the golden dataset that an eval baseline runs against, from the first sample of real traffic to the scheduled refresh.

EV-SCOREv1.0

Scorer selection by feature type

Choosing and recording the scorers that run against a golden dataset for one feature type, and writing down what each scorer does and does not measure.

EV-THRESHv1.0

Thresholds and release gating

Setting the pass and fail rules for an eval run and wiring them into your release process so a regression is caught before the change ships.

EV-RERUNv1.0

Rerun triggers and procedure

How we decide that an eval run is due, which trigger fired, and how the rerun and its written verdict are produced.

EV-PRODv1.0

Production sampling and drift detection

Sampling live production traffic for a tracked feature, scoring that sample against the agreed rubric, and comparing the result with the previous period.

EV-AGENTv1.0

Span-level evaluation for agents

EV-AGENT scores one agent run at the level of its individual steps, so that a wrong tool call or a wrong decision path is visible even when the final answer reads well.

EV-REPORTv1.0

Drift report structure

EV-REPORT sets the sections of the monthly drift report and the bounds the report states about its own numbers. The structure is DriftCaught's own, not a published standard.

How to read a protocol

  • Scope says what it covers and what it leaves to another protocol.
  • Steps are numbered. A step that asserts a practice carries a citation. A step with no citation describes a mechanical action inside the service.
  • Limits says what the protocol does not catch. It is on every page and it is not decoration.
  • Derivation names the version, the sources and the date.

Evidence

Every citation on these pages resolves to an item on the evidence page, with the publisher, the title, the URL and the date we read it. There are 45 of them.

Read the evidence list

Change-log entries restate provider announcements with sources and dates. Impact notes cite public practitioner reports. The site publishes no measurements of its own on any model.

Questions about the method

Why publish the method at all?
Because an eval you cannot inspect is a number you have to take on faith, and the whole point of buying this is to stop taking your feature's quality on faith. Every protocol is here in full, with the sources it was derived from and a version.
Are these standards?
No. They are our written procedures, derived from published practice and cited step by step. EV-REPORT is our own document structure and says so. Where a step rests on somebody else's published guide, the citation names them.
What happens when a protocol changes?
The version changes with it. A change to the steps bumps the minor version, a change to what the protocol measures bumps the major. Your reports name the protocol version that produced them, so an old report stays readable.
Do you follow these for every customer?
Yes. The method does not change with the price of the plan. What changes between plans is how many features are covered and how fast a flagged change is evaluated.
Which tools do you run these in?
Yours. The protocols are written so they can run in Braintrust, Langfuse, Arize Phoenix, DeepEval, promptfoo, Weights & Biases or a plain test harness. We are tool-agnostic and we work inside your stack.

Related