Test-Time Compute Scaling

For most of the last decade the recipe for a better language model was to train a bigger one on more tokens. The scaling relationships published by OpenAI and later refined in the Chinchilla work from DeepMind formalised that: quality improved predictably with parameters, data and training compute, and the practical question was how to split a training budget between them.

Test-time compute scaling is the observation that there is a second budget, and it behaves differently. A trained model can be run once, or it can be run many times, or it can be allowed to generate a much longer sequence of intermediate work before committing to an answer. Each of those spends more computation at inference and, on the right kind of task, produces better answers from identical weights.

The distinction that organises the whole subject is between two directions of spending.

Sequential scaling lets a single attempt get longer. The model reasons step by step, drafts, critiques its own draft, revises, and only then answers. Compute grows along the length of one trajectory, and later tokens can condition on the earlier work.

Parallel scaling runs many independent attempts. Compute grows across trajectories that cannot see each other, and something has to decide which attempt to keep. That selection step is not a detail; it is the whole problem, and most of the interesting engineering lives there.

The two compose. A practical system usually samples several reasoning trajectories, each of which is itself sequentially extended, then applies a selection rule. Understanding that a system has both a depth and a width, and that they cost differently and fail differently, is most of the conceptual work.

The reason this matters commercially is that inference compute is a knob you can turn per request, at runtime, without retraining. A larger model is a fixed cost paid on every request including the trivial ones. Inference budget can be spent where it is needed and withheld where it is not, which is a substantially better shape for a production system — provided you can tell the two kinds of request apart.

The simplest version is old and well documented. Chain-of-thought prompting, described by researchers at Google Research, showed that asking a model to work through intermediate steps before answering changes what it can solve, because the intermediate tokens act as a scratchpad that the final answer conditions on. A closely related line of work found that a bare instruction to reason step by step elicits much of the same behaviour without worked examples.

Everything after that is an elaboration on the same mechanism.

Self-refinement and its limits

The intuitive next move is to have the model criticise its own output and revise. Sometimes this works, and it is worth knowing exactly when. Work published by Google DeepMind argued that language models are not reliably able to self-correct reasoning errors using intrinsic feedback alone — asked to review its own answer with no new information, a model may change a correct answer to an incorrect one as readily as the reverse.

The practical reading is that revision needs an external signal. A compiler error, a failing test, a retrieved document, a unit-test result, a second model with a different vantage point: any of these give the revision step something to condition on that the first attempt did not have. Revision loops built on nothing but self-inspection tend to burn tokens and move sideways.

Structured search

Rather than one linear trace, the reasoning can branch. Tree-structured approaches maintain several partial solutions, score them, expand the promising ones and prune the rest; the tree-of-thoughts line of work formalised this, and Monte Carlo tree search variants push it further by using rollouts to estimate the value of a partial state. This is powerful on problems with clear intermediate state and a usable scoring function, and it is mostly wasted effort on problems where partial answers cannot be meaningfully scored.

Reasoning models

The recent shift is that long reasoning traces stopped being a prompting trick and became a trained behaviour. Models trained with reinforcement learning on verifiable outcomes learn to produce extended internal reasoning before answering. OpenAI introduced this style commercially with its reasoning model series; DeepSeek published open weights along with a description of training on verifiable rewards, which made the approach reproducible outside the largest labs.

For an engineer, the consequence is concrete: the amount of thinking is now a parameter you set rather than a behaviour you coax. Reasoning effort settings, thinking-token budgets and stop conditions are the levers, and they belong in your request configuration alongside temperature.

Sample the same prompt several times at non-zero temperature and you get several different answers. That is the easy half. Deciding which one to return is the hard half, and there are only a few real options.

Majority voting

Self-consistency, published by Google Research, samples multiple reasoning trajectories and returns the answer that appears most often, discarding the reasoning itself. It is remarkably effective and almost free to implement, with one hard constraint: it requires answers that can be compared for equality. A numeric result, a multiple-choice letter, a normalised string — fine. An essay, a design document or a piece of prose — the votes never collide, and the method degenerates.

Verifiers and best-of-n

Where answers cannot be compared to each other, they can sometimes be scored. Best-of-n generates candidates and returns the one a scoring model ranks highest.

Scorers come in two shapes. An outcome reward model looks at the final answer and judges it. A process reward model scores each step of the reasoning, an approach OpenAI described in work on step-by-step verification, and it gives a denser signal that supports pruning during generation rather than only ranking at the end. Process supervision is more expensive to build because it needs step-level labels.

The strongest verifier is not a model at all. If the task is code, run the tests. If it is SQL, execute the query. If it is arithmetic or symbolic manipulation, check it. Mechanical verification is exact, cheap and impossible to game, and it is the reason test-time compute scaling produces such visibly different results on verifiable tasks than on open-ended ones.

Coverage is not accuracy

This is the point people most often miss. As you draw more samples, the probability that at least one of them is correct rises — that is coverage, and it is what a pass-at-k measurement reports. Your system's accuracy is coverage multiplied by the quality of your selector. A perfect selector converts all of that coverage into accuracy. A weak selector converts very little of it, and a biased selector can convert less than a single sample would have delivered.

So the honest question when someone reports a gain from sampling is always the same: what selected the answer, and would that selector survive on data it has not seen. A reward model that is itself being optimised against will eventually be gamed, which is the inference-time echo of reward model overoptimisation in training.

Once both axes work, the engineering question becomes allocation. You have a compute budget. You can spend it on a larger model, on longer reasoning, on more samples, or on a verifier — and the right split is not constant.

A paper from researchers at UC Berkeley and Google DeepMind made the sharpest version of this argument: for some problem distributions, allocating additional compute at inference is a more effective use of a fixed budget than pretraining a larger model, and the best allocation strategy depends on the difficulty of the question. Easy questions are answered correctly on the first attempt and extra compute is wasted on them. Very hard questions are not solved by any budget you can afford, and compute spent there is also wasted. The gains concentrate in the middle band.

That immediately suggests the shape of a competent system: it should not spend the same budget on every request.

Difficulty routing is the practical expression of this. Estimate how hard a request is before committing compute, then choose a tier. Signals that can drive the estimate include a cheap first-pass attempt, the model's own uncertainty over the answer, disagreement among a small number of initial samples, task type, and retrieved context length. None of these is precise, and they do not need to be — the value comes from separating obviously-easy from probably-hard, not from a calibrated difficulty score.

Adaptive stopping is the other half. Rather than fixing the sample count in advance, draw samples until a consensus threshold is met or a budget cap is hit. On easy questions the loop exits after very few samples; on hard ones it uses the whole allowance. The same logic applies to sequential depth, where a stop condition on the reasoning trace prevents a model from continuing to think about a question it settled several paragraphs ago.

Serving mechanics matter more here than in ordinary inference, because the workload shape changes. Long reasoning traces enlarge the key-value cache and reduce how many sequences fit in a batch. Parallel sampling from a shared prompt is exactly the case that prefix caching is built for, since every sample shares the prefill. Continuous batching keeps the accelerator busy while trajectories of very different lengths finish at different times. Speculative decoding attacks the latency of long sequential traces specifically. These are not optimisations to add later; they determine whether a design is affordable at all.

Cost stops being a number and becomes a distribution. With fixed single-pass inference, cost per request is roughly predictable. With adaptive reasoning and sampling, the mean may be fine while the tail is many times larger. Budget on percentiles, cap hard, and alert on the tail rather than the average.

Tail latency gets worse than the average suggests. Parallel samples finish at different times and the request waits for the slowest. Long traces are unbounded in a way that ordinary completions are not. Any user-facing surface needs a deadline with a defined fallback answer, not an open-ended wait.

Models overthink easy questions. A reasoning model asked something trivial can still produce a long trace before answering. This is a real cost and a real latency problem, and it is the main argument for routing rather than applying maximum effort uniformly.

Longer is not monotonically better. Extended traces can wander, accumulate an early error and then rationalise it, or drift off the original instruction. More thinking is not free accuracy, and there is a point on most tasks past which additional length stops helping.

The visible reasoning may not be the actual computation. Research from Anthropic on chain-of-thought faithfulness has examined cases where a model's stated reasoning does not reflect the factors that actually determined its answer. Treat a reasoning trace as an artefact to inspect, not as an audit log of the model's internal process, and do not build a compliance story on it.

The attack surface grows with the trace. Longer reasoning over retrieved or tool-returned content gives injected instructions more opportunities to take effect, and intermediate steps are frequently not subject to the same output filtering as the final answer. Validate at every boundary where external content enters, not only at the end.

Caching gets harder. Exact-match response caching stops working when the system deliberately samples at temperature. Prefix caching still helps, and semantic caching of final answers can help, but the naive cache-hit assumptions from single-pass serving do not carry over.

Evaluation becomes ambiguous unless you declare the budget. A score is not a property of a model any more; it is a property of a model plus an inference configuration. Two teams reporting different results for the same weights are usually both right and reporting different budgets. Publish the sampling count, the reasoning effort, the selection method and the temperature alongside every number, internally as well as externally.

This is a short, sharply scoped topic, and it assumes some ground has already been covered. You are ready if you can explain what the key-value cache is, why batching improves throughput, and what temperature does to a distribution over tokens. If any of those are unfamiliar, spend a day on inference mechanics first; the allocation arguments here are meaningless without a feel for what generation actually costs.

The most useful way to work through the material is against a task of your own that has a checkable answer. Almost every technique here is easier to evaluate — and far easier to believe — when correctness is mechanical rather than a judgement call. A set of SQL queries against a schema, a batch of unit-tested coding problems, or a collection of arithmetic word problems all work well.

A sensible order of experiments:

1. Establish a single-pass baseline with a fixed prompt, fixed temperature and a scored evaluation set. Record the cost and latency alongside the score, because every later comparison is a three-way trade.
2. Add reasoning depth alone. Chain-of-thought or a reasoning effort setting, nothing else. Note where it helps and where it does nothing.
3. Add width alone. Sample several times, apply majority voting where the answer space allows it. Measure coverage separately from accuracy so you can see how much your selector is throwing away.
4. Replace the selector with a real verifier. Execute the code, run the query, check the result. This is usually the single largest improvement in the sequence.
5. Make it adaptive. Route by estimated difficulty, stop early on consensus, cap the budget. This is where the cost curve becomes acceptable.

From there the natural continuations are serving optimisation, since the throughput characteristics of this workload are unusual, and evaluation design, since a system with an adjustable budget cannot be compared honestly without one. If you are running open-weights models locally, the same techniques apply directly and are arguably more valuable there, because a verifier plus sampling is a way to buy quality with time on hardware that cannot hold a larger model.