The MyCity chatbot, as reported

On 29 March 2024 The Markup reported that New York City's MyCity chatbot told users that landlords do not need to accept Section 8 vouchers and that an employer can take a cut of workers' tips, both of which are unlawful.[The Markup] That is journalism, not an incident report by the operator. No root cause was published alongside it. So this post reads the failure class, and stops where the public record stops.

What kind of source this is

Reporters put questions to a live public system and checked the answers against the law. That gives you two things: the outputs, and a verdict that they were wrong. It does not give you the system prompt, the retrieval corpus, the model id, the guardrails, or whether any evaluation ran at all.

We label this the way our evidence page labels it: reported, by a named publisher, on a stated date. It is not a confirmed engineering account, and we do not attribute a cause to a team that has not published one. If you see this incident cited as proof that some particular technique failed, check whether the citation goes past the reporting.

The failure class is fluent and wrong

The answers in the report were not incoherent. They read like answers. That is what makes this class hard: the output passes every check that looks at the shape of the text.

A judge (an LLM asked to score another model's output against a written rubric) pointed at helpfulness or tone would have had nothing to object to. The property that failed was correctness against an external authority. Housing law and labour law are the ground truth here, and neither one is inside the model. Somebody has to bring them in.

What an evaluation of this shape actually reads

An eval is a golden dataset (a fixed set of labelled inputs with the expected outputs agreed with you, versioned so two runs can be compared) plus a set of scorer (a function or a judge that turns one output into a number or a verdict). For a public-services assistant, the dataset is a list of the questions people actually ask, and the label on each one is the lawful answer.

That label does not come from a model. It comes from a person. Hamel Husain writes that in most organisations there are usually one or two key individuals whose judgment is crucial to the AI product, and that the judge is aligned iteratively against that person's own critiques.[Hamel Husain] For a housing question, that person is a housing lawyer, and they work for you, not for us. Our golden dataset protocol says who labels and what a label has to contain.

The starting size is smaller than people expect. Hamel Husain writes that his heuristic is to start with around 30 examples[src] and keep going until he sees no new failure modes.[Hamel Husain] Arize tells practitioners to begin by pulling 50 real traces[src] from their system.[Arize AI] DeepEval's documentation says even 10 to 20 inputs[src] is a great start.[Confident AI] Those are three separate published figures from three sources, and none of them says the items have to be failures. A first pass at a voucher question and a tips question is an afternoon of work, not a quarter.

The rule that keeps a dataset honest

Once the dataset exists, the temptation is to move the expected answer when the model disagrees with it. Galtea warns against updating ground truth to match model behaviour, and writes that additions should require code review while deletions are treated as production risk.[Galtea] On a question of law that rule is the whole exercise. If the correct answer is whatever the model said last, the eval measures nothing.

What an evaluation would not have read

Coverage is bounded by the dataset. A question that is not in it gets no verdict. A statute that changed last month does not update the labels by itself. A judge calibrated on housing questions says nothing about benefits questions. These are not edge cases in the method, they are the method's shape, and they are listed on what we cannot do.

We are not claiming that we would have caught this, or that any evaluation would have. The operator published no account of what it ran. We describe the class of check that reads this class of failure, and nothing beyond that.

What to take from it

Three things, in order of how cheap they are.

First, write down the twenty questions your assistant will be asked most, and get a person with the relevant authority to write the correct answer next to each one. That is a dataset. Our chat assistant hub lists the scorers that then apply.

Second, separate the two questions your scoring asks. "Is this answer well formed" and "is this answer correct" are different scorers with different ground truth, and merging them hides exactly this failure.

Third, decide what happens when the correct answer changes. A rule change is a dataset change, and a dataset change is a rerun. Other posts in the failures category follow the same pattern, and the eval baseline is where the first version of the list gets built.