Evaluating LLMs Beyond Accuracy
Accuracy scores tell you almost nothing about whether a language model is safe or useful to deploy. Helpfulness, harm, and hallucination need their own measurements.
Why one number is not enough
When people ask how good a language model is, they usually want a single number. Accuracy on some benchmark, a leaderboard rank, a percentage. That instinct comes from classical machine learning, where a model either predicts the correct class or it does not. Language generation does not work that way. A model can be fluent, confident, and completely wrong. It can be factually correct but useless because it dodges the question. It can be helpful in one context and dangerous in another. Squeezing all of that into one accuracy figure hides more than it reveals.
I think the most useful move is to stop treating evaluation as a single axis and instead track at least three separate ones: helpfulness, harm, and hallucination. They are not the same failure mode, they do not trade off in predictable ways, and they need different test sets and different metrics. A model that scores well on a multiple choice benchmark can still fail badly on all three.
Consider a customer support assistant. Accuracy on a held out set of factual questions might sit at ninety two percent. That sounds solid. But if the assistant refuses to answer a further fifteen percent of legitimate questions out of excessive caution, its helpfulness is much lower than the accuracy figure suggests. And if, on a separate set of adversarial prompts, it leaks internal policy text three times out of a hundred, that is a harm problem that the accuracy number never touched. Three different failure surfaces, three different measurements.
Helpfulness: measuring whether the answer actually helps
Helpfulness is not the same as correctness. A response can be technically accurate and still fail the user: too long, too hedged, missing the actual point of the question, or answering a slightly different question than the one asked. The way most labs measure this is through pairwise preference judgements, where a human or a calibrated judge model picks the better of two responses to the same prompt. This produces something closer to an Elo-style ranking than a percentage.
A concrete example: suppose you run five hundred prompts through two model versions and collect preference votes. Model A wins sixty percent of the time. That sixty percent win rate is far more informative than any accuracy score, because it directly encodes what users actually prefer, including tone, structure, and relevance, not just factual overlap with a reference answer.
The trap here is annotator drift and prompt selection. If your prompt set is dominated by easy factual questions, you are not measuring helpfulness on the messy, ambiguous requests that make up most real usage. I always push for a held out prompt set that is deliberately skewed towards ambiguity, multi step reasoning, and underspecified requests, because that is where helpfulness gaps actually show up. A model that looks identical on clean benchmark questions can diverge sharply once instructions get vague.
Harm: testing for the failure you are not looking for
Harm evaluation is different again, because you are not measuring average case performance, you are measuring worst case behaviour under adversarial pressure. This means the test set cannot be a random sample of ordinary prompts. It has to be built specifically to probe known risky categories: instructions for dangerous activities, private information extraction, biased or discriminatory completions, and prompt injection attempts that try to override the model's instructions.
A useful framing is attack success rate rather than accuracy. If you run two hundred adversarial prompts designed to extract a system prompt or bypass a safety instruction, and the model complies in fourteen of them, your attack success rate is seven percent. That number matters far more for deployment risk than a benchmark accuracy of ninety five percent on unrelated tasks. A model can be excellent at maths and still leak sensitive instructions under a cleverly worded jailbreak.
The practical difficulty is that harm categories keep expanding as people find new attack strategies, so a static test set goes stale quickly. Treat it the same way you would treat a security test suite: version it, keep adding newly discovered attack patterns, and rerun it every time you change the model, the system prompt, or the retrieval pipeline. A single clean run six months ago tells you nothing about current risk.
Hallucination: the failure that looks like confidence
Hallucination is the hardest of the three to catch because it wears the same voice as correct output. The model states something false with exactly the same fluency and certainty it uses for something true. Accuracy metrics built on multiple choice questions rarely catch this, because they only test whether the model can select a correct option among a few candidates, not whether it fabricates specific claims, citations, or numbers when generating open ended text.
A workable evaluation approach is claim level fact checking: break a generated answer into individual factual assertions, then verify each one against a trusted source or retrieval index. If a model writes a paragraph containing eight distinct factual claims and two of them are unsupported or contradicted by the source material, that gives you a hallucination rate of twenty five percent for that response, a number that a simple accuracy check on the final answer would never surface, because the paragraph as a whole might still read as broadly correct.
Leakage is a serious risk here too. If your fact checking source overlaps with data the model was trained or fine tuned on, you will systematically underestimate hallucination, because the model may reproduce memorised text that happens to be correct for reasons unrelated to reasoning. I always check the provenance of the verification corpus against the training data before trusting a hallucination score, the same way I would check for train test leakage in any other supervised setting. A hallucination rate that looks reassuringly low can simply mean your check set was not independent of training data.
