Chain-of-Thought Prompting: When It Helps, When It Hurts

Why the idea matters

Chain-of-thought prompting asks a language model to write out intermediate reasoning steps before giving a final answer, rather than jumping straight to a conclusion. The appeal is obvious: if a model shows its working, we assume it is more likely to catch its own mistakes, and we get a trace we can inspect. Early demonstrations on multi-step arithmetic and logic puzzles showed striking accuracy gains simply from adding a phrase like 'let us think step by step' or a few worked examples with explicit reasoning.

But treating chain-of-thought as a universal upgrade is a mistake I see repeated often, including in my own early experiments. The technique changes the distribution of tokens the model generates before it commits to an answer, and that shift is only helpful when the task actually benefits from decomposition. For tasks that are essentially pattern lookups, or where the reasoning trace itself introduces errors that then get propagated, forcing a chain of thought can make performance worse, slower, and harder to audit.

This matters practically because prompting decisions are rarely benchmarked properly before they ship. A team sees a promising anecdote, chain-of-thought fixes a tricky example, and the pattern becomes house style across an entire product. Without a leakage-aware, task-matched evaluation, you cannot tell whether you have found a genuine improvement or just a placebo that happens to look articulate.

A worked example: arithmetic versus retrieval

Consider a simple multi-step arithmetic problem: a shop sells 3 boxes of 12 pens each at 2 pounds per pen, then applies a 15 percent discount on the total. Compute the final price. Asked directly for a single number, a model has to hold four operations in its head implicitly and often slips on the order of operations or the discount arithmetic, producing something like 39.10 pounds when the correct answer is 30.60 pounds. Asked to reason step by step, the model can write: 3 boxes times 12 pens is 36 pens; 36 pens times 2 pounds is 72 pounds; 15 percent of 72 is 10.80; 72 minus 10.80 is 61.20. Even if a step is wrong along the way, the explicit structure makes it far easier to write correct arithmetic than to produce it in one leap, and this is where chain-of-thought earns its reputation: compositional tasks with a small number of clearly separable steps.

Now take a different kind of question: 'what is the capital of the country where the Amazon rainforest is mostly located?' This is a two-hop factual lookup, not a computation. A model that already has the fact 'Brazil contains most of the Amazon' and 'Brasilia is the capital of Brazil' stored can answer directly and correctly. Forcing a chain of thought here sometimes helps by making the intermediate fact explicit, but just as often it invites the model to hedge, second-guess a correct initial instinct, or drift into a plausible-sounding but wrong chain, for instance conflating population centres with capitals. I have seen direct-answer prompting outperform chain-of-thought on exactly this style of factual question, because the reasoning trace gives the model more surface area to talk itself out of a correct answer it already had.

The pattern that emerges is that chain-of-thought helps most when the task decomposes cleanly into steps the model can execute more reliably in isolation than jointly, and helps least, or actively hurts, when the task is a single retrieval, a classification with a short label space, or a judgement call where elaboration mainly adds opportunities for the model to rationalise an initial mistake.

Where reasoning traces backfire

There are three recurring failure modes worth naming explicitly. The first is snowballing: an early error in the trace becomes a premise for later steps, and the model, having committed to it in text, rarely backtracks. If the first line of arithmetic in my pens example were wrong, say 3 times 12 written as 32, everything downstream inherits that error while looking perfectly coherent. A single wrong number early in a long chain is more damaging than a wrong number produced in isolation, because the explanation lends it false authority.

The second is post-hoc rationalisation. On tasks involving sentiment, tone, or subjective judgement, models sometimes produce a chain of thought that reads as reasoning but is actually generated after the answer has effectively already been decided by the model's internal representations, then dressed up to justify it. This is hard to detect from the transcript alone, since the text looks like genuine deliberation. It matters for evaluation because a convincing-looking rationale is not evidence that the underlying decision process was sound, and it should not be mistaken for interpretability.

The third is cost and latency without benefit. Every additional token of reasoning costs compute and time, and in a production system serving many requests, an unnecessary reasoning trace on a task that did not need one is pure overhead. If a classifier task shows no measurable accuracy gain from chain-of-thought on a held-out, non-leaking test set, the extra tokens are a tax with no return, and that tax compounds at scale.

A practical way to decide

My rule of thumb is to test both prompting styles on a proper held-out split before choosing one, rather than trusting intuition or a handful of anecdotes. Split your evaluation set the same way you would for a supervised model, keep it separate from anything used to write or tune the prompt, and measure accuracy, latency, and token cost for direct-answer and chain-of-thought variants side by side. If the gain is within noise, or only appears on a handful of examples you already knew were hard, prefer the cheaper direct prompt and reserve reasoning traces for genuinely multi-step tasks.

It is also worth checking whether the reasoning trace is doing real work or just narrating a decision the model would have made anyway. One quick diagnostic is to compare the final answer distribution with and without the chain of thought on the same inputs; if they rarely disagree, the elaboration is not changing outcomes and mainly adds cost and a false sense of transparency. The broader lesson is that chain-of-thought is a tool for compositional problems, not a general-purpose accuracy switch, and treating it that way protects both your evaluation and your compute budget.