How it works
Four steps: we build the dataset from your real failures, pick the scorers and calibrate the judge, rerun on every trigger, and report what moved by the 5th business day. The first two weeks are the baseline. After that it runs monthly.
Nothing here is secret. Each step is a published protocol you can read before you buy, with citations and a version number.
The four steps
- 01
We build the dataset from your failures
Your logs and your support queue already contain the cases where the feature was wrong. We turn those into a labelled, versioned set, with your domain expert grading the ambiguous ones. This is the part that decides what everything downstream is worth.
- 02
We pick scorers and calibrate the judge
Rule based code checks where the answer has a right shape, similarity where there is a reference, a model judge where it needs reading. Then the judge is scored against your expert's verdicts and the agreement is reported, so you know what the number is worth before you gate on it.
- 03
We rerun on every trigger
A tracked provider announces, you flag a prompt or retrieval change, the weekly schedule comes round, or a pull request hits the CI gate. Each one reruns the suite and produces a written verdict against the thresholds you agreed.
- 04
We report what moved, and what we cannot see
By the 5th business day: coverage, the run log, the per-scorer deltas, the judge's agreement this month, what production sent that the dataset has no case for, and what the dataset does not cover. Then the recommended actions.
The first four weeks
- Week 1
Access, scope, and the dataset. By the end of it there is a labelled set your expert has graded.
- Week 2
Scorers, judge calibration, thresholds you agree to, and the first report. The baseline ends here.
- Week 3
The first scheduled rerun, and the first flagged change goes through the suite before it ships.
- Week 4
Production sampling starts feeding the dataset. The first monthly report is drafted from real runs.
The baseline is $1,800 and 10 business days, credited to the first month of any plan started within 60 days.
The protocols behind it
Eight of them, published in full. Your reports name the version of each protocol that produced them, so an old report stays readable.
- EV-DATAGolden dataset construction
- EV-SCOREScorer selection by feature type
- EV-JUDGEJudge calibration and recalibration
- EV-THRESHThresholds and release gating
- EV-RERUNRerun triggers and procedure
- EV-PRODProduction sampling and drift detection
- EV-AGENTSpan-level evaluation for agents
- EV-REPORTDrift report structure
What we guarantee
- First eval baseline within 10 business days of access.
- Every tracked provider update triggers a rerun within 2 business days of the provider's announcement.
- Every flagged prompt or retrieval change is evaluated before it ships, within the plan's turnaround.
- Monthly drift report by the 5th business day.
- Judge recalibrated against human verdicts at least monthly.
- If a regression above the agreed threshold reaches production on a change we marked as passed, the next month is free.
These six are the whole list. Nothing else on this site is a guarantee.
What we cannot do
We cannot make a model deterministic or guarantee its outputs. We measure against an agreed dataset and thresholds and report what changed. We do not train or fine-tune models, and we do not replace human review for policy or legal decisions. Coverage is bounded by the dataset and the plan.
Questions about working together
Do you need access to our production systems?
Which tools do you use?
How much of our time does this take?
What happens the day a provider announces something?
What if we stop?
The dataset needs a person who knows your domain. We can do everything else, and we cannot do that. It is the one dependency worth planning around.
Related