Scorer selection by feature type

Scope

This protocol covers how we choose the scorers (the functions that turn one output into a score or a pass and fail verdict) that run against a golden dataset (a fixed set of saved inputs with the expected result for each one, which an eval run scores against), and how we write down what each one measures. It runs once per feature type when a baseline is set up, and again whenever a new failure mode is added to the dataset.

It does not build the dataset. That is EV-DATA. It does not set the pass mark, agree what counts as a regression, or decide what blocks a release. Those are agreed with you in EV-THRESH. What this protocol produces is the list of things that get measured and the written limits of each measurement.

Inputs

• A versioned golden (a precursor to a test case that holds an input plus the expected result, and becomes a full test case once your application produces the actual output) dataset for the feature, with each row carrying its failure mode.
• The label rules your domain expert wrote during labelling.
• The feature type: chat assistant, summarisation, extraction, retrieval augmented generation (the model answers from documents fetched for it, so retrieval is the step that fetches those documents or passages), agent or classification.
• Traces (the ordered record of one run, every step with its timing, arguments and return) that include tool calls and retrieved context, where the feature has them.
• Any policy or legal requirement that a rule based check has to enforce rather than score.

Steps

1. Start from the pair, dataset plus scorers

A dataset is not an evaluation on its own. Weights and Biases documents a Weave Evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against a fixed set of test cases, compare changes over time and identify regressions (a case that passed on the previous run and fails on this one). [Weights & Biases]

We treat the scorer set as part of the same artifact as the dataset. A scorer set with no dataset version is not a baseline, and a dataset with no scorer set produces no number.

2. Work through the published evaluator types before inventing one

LangSmith documents four evaluator types[src]: human review, rule based code functions, LLM as judge (a model that scores another model's output against a written standard) in reference free and reference based variants, and pairwise comparison of two versions of the application. [LangChain] We go through that list in order for every failure mode on the list, and take the cheapest type that can actually catch the mode.

• Human review stays in the set for anything that is a policy or legal decision.
• Pairwise comparison is useful when you are choosing between two prompt or model versions and no absolute scale exists yet.

3. Decide whether each scorer needs a reference

A reference based scorer needs an expected result stored with the input. DeepEval defines a golden, which is the form the dataset already stores. [Confident AI] Reference free scorers read only the input and the output, so they can run on production samples where no expected result exists. [LangChain]

We write down, per scorer, which of the two it is, because that decides where it can run: reference based scorers run on the golden dataset, reference free scorers can also run on the production sample in EV-PROD.

4. Use a code check wherever the output has a checkable shape

For extraction and classification, most of what matters is checkable in code: field presence, type, format, an exact or normalised match against the expected value, a valid enum. LangSmith lists rule based code functions as one of its four evaluator types for exactly this case. [LangChain]

A code check is cheaper to run, easier for you to read, and does not need calibrating. We only reach for a judge on a failure mode that code cannot express.

Where a requirement is a hard policy rule rather than a quality preference, it becomes a code check plus a human review step, not a scored metric.

5. For retrieval augmented features, score grounding against the retrieved context

Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, computed by identifying every claim in the response and checking each one against the context, expressed as supported claims divided by total claims[src] on a 0 to 1 scale. [Ragas]

Read the definition carefully before agreeing to it. Faithfulness scores whether the answer is grounded in the context that retrieval returned. It does not score whether that context is true. A confidently wrong source document produces a faithful answer. Where truth matters, the dataset needs an expected answer and a separate reference based scorer. This is the scorer pattern behind the RAG feature hub.

6. For agents, score at more than one resolution

Future AGI describes scoring an agent at four resolutions[src], end to end, trajectory (the ordered path the agent took, judged as a path rather than by its last step), step and turn, and writes that a high end to end score can hide bad tool calls, while strong per step scores can still miss that the agent pursued the wrong goal. [Future AGI] Both directions matter, so neither resolution is dropped from the set.

Arize Phoenix makes the same point from the other side: end to end evaluation only inspects the two endpoints, the input and the final output, and the interesting failures happen in the middle where they cannot be seen, naming reasoning, tool selection and the decision path. [Arize AI]

7. For agents, name the trace signals the scorers read

Arize Phoenix describes reconstructing two signals from an agent trace: tool_path, the ordered tool calls with their arguments, and tool_io, what each tool was called with and what it returned, and warns that a correct looking answer can come from a broken path. [Arize AI]

We name those two signals explicitly in the scorer set, because they decide what your application has to emit. If a span (one recorded step inside a request, for example a single tool call or retrieval, with its inputs and outputs) does not carry the tool arguments and the returned value, no span level scorer can read it, and the set falls back to end to end scoring with that gap written down.

8. Where a judge is the scorer, agree the rubric and more than one agreement statistic

A judge needs a written rubric (the rules the judge applies to decide whether an output passes, in your expert's words) drawn from the label rules in EV-DATA, and it needs to be checked against human labels before its numbers are used.

Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than a single statistic, and writes that a judge showing 0.9 accuracy with 0.1 Cohen's kappa[src] is essentially guessing. [Galtea] We agree which of those statistics appear in your monthly report at selection time, so the judge is never reported on one number. The calibration procedure itself is EV-JUDGE.

9. Record the set and the gaps

Inside the service, the scorer set is written as a file pinned to the dataset version, listing each scorer, its type, whether it is reference based, which failure modes it covers and which signals it reads. Each run record stores the scorer set version alongside the run id.

Any failure mode with no scorer against it is listed as an uncovered mode in the same file and repeated in the monthly report, so the gap stays visible rather than being read as a pass.

Outputs

• A named scorer set for the feature, pinned to a dataset version.
• One line per scorer: type, reference based or reference free, failure modes covered, signals read.
• A judge rubric and the list of agreement statistics to report, handed to EV-JUDGE.
• The list of trace signals your application needs to emit for the span level scorers to run.
• A written list of failure modes no scorer covers, carried into the monthly report.
• A candidate metric list handed to EV-THRESH, where the pass marks are agreed with you.

Limits

Version and derivation

Version 1.0, derived from DeepEval metrics, Ragas, Arize evaluation guide, written 22 September 2026, by DriftCaught.