Classification assigns a label: routing a ticket, tagging an intent, flagging content, picking a category. It is the feature type where evaluation is on the firmest ground, because the output comes from a fixed set and there is a right answer to compare against. The trap here is not the absence of metrics. It is reporting the wrong one.
What typically breaks in classification after a model change
The borderline cases flip
Clear cases stay clear. A billing question is still billing. What moves is the band between two classes: the ticket that is both a complaint and a cancellation, the message that is either urgent or merely annoyed. A new model version draws that line in a slightly different place, and the cases sitting on it change sides.
That is why a golden dataset (a fixed, versioned set of inputs with the verdict a domain expert gave each one) for a classifier is deliberately overweighted with borderline examples. A dataset of easy cases will pass every run and tell you nothing.
A rare class collapses while the headline number holds
If one class is 2 percent of your traffic and the model stops predicting it, aggregate accuracy moves by about those same two points, which is the arithmetic and not a measurement. The feature is broken for everyone in that class and the dashboard looks fine.
Galtea recommends measuring with accuracy, precision, recall, F1, Pearson correlation, Spearman correlation and Cohen's kappa rather than one statistic, and writes that 0.9 accuracy with 0.1 Cohen's kappa[src] is essentially guessing. [Galtea] Hamel Husain makes the same argument from the other end: raw agreement misleads when classes are imbalanced, so true positive rate and true negative rate should be reported separately against human labels. [Hamel Husain]
The output stops being a valid label
The model returns a class that is not in your enum, or the right class in different casing, or a short explanation wrapped around the label. Your mapping drops it into a fallback bucket and the fallback rate climbs. This is a formatting failure wearing a classification costume, and a code check catches it immediately if the fallback rate is a reported number rather than a log line.
Class drift in production makes an unchanged model look worse
The model has not changed and your traffic has. A new campaign, a new market, a seasonal peak, and the mix of classes arriving shifts. Aggregate numbers on the production sample move while the fixed dataset stays flat. This is drift (a measured change in a feature's scores between two runs) with a cause on your side rather than the provider's, and it is only separable because the two runs exist side by side. Langfuse says observation-level evaluators are the recommended target for live production data and lists sampling a percentage of traces as a way to manage the cost of judged evaluation. [Langfuse] The sampling method is EV-PROD.
The labels themselves stop matching the product
Six months in, your taxonomy has a class nobody uses and is missing two that support agents invented in a spreadsheet. The model is scored against a rubric (the written scoring standard applied to an output, in your expert's words) your product left behind. Future AGI recommends refreshing a golden dataset monthly by default, sampling 200-500 recent production traces[src] stratified by intent and persona, weekly during rapid change and immediately after model or corpus updates. [Future AGI]
Which scorers apply
LangSmith documents four evaluator types: human review, rule based code functions, LLM as judge (a model that scores another model's output against a written rubric) in reference free and reference based variants, and pairwise comparison of two application versions. [LangChain] Classification sits almost entirely in the second one. The label has a right answer, so the scorer (one automated check that turns an output into a number or a pass and fail verdict) set is conventional and cheap, and a judge is an unnecessary expense on most classifiers.
- Exact label match code check. Catches a wrong label against the expected label. Misses how wrong it was, since confusing two adjacent classes and confusing two unrelated ones score the same.
- Per class precision and recall. Catches a single class collapsing, which is the failure aggregates hide. Misses nothing about that class, and says nothing about the rest of the set, so it is always read alongside the whole table.
- Macro F1 across classes. Catches a drop weighted equally across classes rather than by traffic volume. Misses the business weighting, because your rare class may matter more or less than an equal share.
- Cohen's kappa against human labels. Catches agreement that is no better than chance given the class balance. Misses which class is at fault, so it is a warning light rather than a diagnosis.
- Fallback and unknown label rate. Catches outputs that are not valid labels at all. Misses valid labels that are wrong.
A judge belongs here only where the label is genuinely interpretive, for example a severity or a tone class your experts argue about. In that case it stops being a free measurement and picks up the whole calibration burden described in EV-SCORE.
What a baseline includes for this feature type
The eval baseline for a classifier is built around the confusion pairs, not around the easy cases.
- Real inputs per class, with the rare classes deliberately oversampled so each one has enough rows to move a number.
- Borderline cases collected on purpose, including the ones your own experts disagreed about, with the disagreement recorded.
- The label taxonomy itself versioned with the dataset, so a taxonomy change shows up as a dataset change rather than as a mysterious score drop.
- Written label rules from your domain expert, including the tie-break rule for cases that could sit in two classes.
- A report that leads with the per class table and shows the aggregate second, plus the fallback rate as its own line.
- threshold (the number agreed with you at which a measured drop counts as a finding on this dataset) values set per class where a class matters more, with blocking failures separated from regressions (a case that passed on the previous run and fails on this one) that gate on the change in fail rate. [Galtea] Construction is EV-DATA.
Which plan fits
One feature at $1,900 a month fits a classifier in production, after the baseline. The work is cheaper here than on any other feature type, because the scorers are code and there is no judge to recalibrate, so most of the monthly value is in the reruns themselves: every tracked update in the change log, every prompt change you flag, and the weekly schedule. The caveat is about what a number means. A per class score is a statement about the dataset it was measured on, not a prediction about next week's traffic, and a classifier that holds on the fixed set can still slip on production if your class mix moves. Both runs appear in the same report for that reason. The structured output side of this problem is on the extraction hub, and every figure here traces to the evidence list.