Choosing a scorer: code checks, similarity or a judge

Take the cheapest scorer (the function or model that turns one output into a score or a pass and fail verdict) that can actually catch the failure mode you wrote down. Work through the published evaluator types in order, and only reach for a model judge (a model that scores another model's output against written rules) on a failure mode that code cannot express.

Start from the pair, not from the metric

A dataset does not produce a number on its own. Weights and Biases documents an evaluation as a dataset of test examples plus one or more scoring functions, used to measure performance against repeatable test cases, compare changes over time and identify regressions. [Weights & Biases]

That pairing is the unit of work. A scorer with no fixed dataset behind it measures whatever today's traffic happened to be, and a dataset with no scorer set produces nothing at all.

The four published evaluator types

LangSmith documents four evaluator types[src]: human review, rule based code functions, LLM as judge in reference free and reference based variants, and pairwise comparison of two versions of an application. [LangChain]

We walk that list in order for each failure mode, top to bottom on cost, and stop at the first type that can catch the mode.

• Human review stays permanently in the set for anything that is a policy or legal decision.
• Rule based code checks cover anything with a checkable shape.
• A judge covers what remains.
• Pairwise comparison earns its place when you are choosing between two versions and no absolute scale exists yet.

Rule based code checks

For extraction and classification, most of what matters is checkable without a model: field presence, type, format, an exact or normalised match against the expected value, a valid enum, a total that adds up.

A code check is cheap to run, readable by the person who has to act on it, and needs no calibration. When a code check fails, the failure is the answer. Nobody has to argue about whether the checker was having an off day.

The limit is equally plain. A code check cannot tell you that a correctly formatted answer was unhelpful, condescending or off policy. If the failure mode on your list is about tone, completeness or reasoning, code will not reach it.

Where a requirement is a hard policy rule rather than a quality preference, it becomes a code check plus a human review step rather than a scored metric. A number you can trade off is the wrong shape for a rule you cannot break.

Comparison against a reference

A reference based scorer needs an expected result stored beside the input. That is the form a dataset row already takes: DeepEval defines a golden (a precursor to a test case holding an input plus the expected result, which becomes a full test case once your application produces the actual output). [Confident AI]

Reference comparison is the right tool when there is a defensible right answer: the invoice total, the classification label, the field that should have been extracted. It is a poor tool when many different outputs would all be acceptable, because a summary can be excellent and share very little wording with the one your expert wrote.

That constraint decides where a scorer can run, not just how good it is. Reference free scorers read only the input and the output, so they can also run on production samples where no expected result exists. [LangChain] Reference based scorers are confined to the golden dataset. We record which of the two each scorer is for exactly that reason, as set out in the EV-SCORE protocol.

Grounding scorers for retrieval

For features that fetch documents before answering, there is a specific scorer worth reading closely before you adopt it.

Ragas defines faithfulness as the share of claims in a response that can be inferred from the retrieved context, computed by identifying every claim and checking each against the context, expressed as supported claims divided by total claims[src] on a 0 to 1 scale. [Ragas]

Read what that measures. It scores whether the answer is grounded in whatever retrieval (the step that fetches documents or passages for the model to answer from) returned. It does not score whether that material was true. A confidently wrong source document produces a faithful answer. Where truth matters, you need an expected answer and a separate reference based scorer alongside it. More on that on the RAG feature page.

A judge, once it has been calibrated

A judge is the only one of the three that can read tone, completeness or whether an instruction was followed. It is also the only one that is itself a model behind an API, which means selecting it is not the same as trusting it.

Galtea recommends measuring a judge with accuracy, precision, recall, F1, Pearson and Spearman correlation and Cohen's kappa rather than one statistic, and writes that a judge with 0.9 accuracy and 0.1 Cohen's kappa[src] is essentially guessing. [Galtea] Hamel Husain's figure of about 100 examples per failure mode[src] sizes what it takes to check one. [Hamel Husain] The procedure is EV-JUDGE, and the reporting rules are in calibrating a judge against human verdicts.

For agents, no single scorer is enough

Scoring only the final answer inspects two points. Arize Phoenix writes that end to end evaluation only inspects the two endpoints, the input and the final output, and that the interesting failures happen in the middle where they cannot be seen, naming reasoning, tool selection and the decision path. [Arize AI]

Future AGI describes scoring an agent at four resolutions[src], end to end, trajectory, step and turn, and warns in both directions: a high end to end score can hide bad tool calls, and strong per step scores can still miss that the agent pursued the wrong goal. [Future AGI] Both warnings are why neither resolution gets dropped. See EV-AGENT for how a span (one recorded step inside a request, for example a single tool call) gets scored.

Write down what nothing covers

The last step is the one most often skipped. Any failure mode with no scorer against it goes on a written list of uncovered modes, and that list is repeated in every report.

A gap that is written down is a decision. A gap that is not written down reads as a pass, and a clean dashboard is the most expensive thing on this page. The full method is the methodology index, and what it does not cover is here.