Pick the Right Model for Your Machine

"Which local model is best" is not a question with an answer, and treating it as one is the root of most wasted time in this hobby. It becomes answerable only once three things are fixed: the hardware you will run it on, the job you will give it, and the quality bar that job has to clear.

Aggregate leaderboard scores compress dozens of capabilities into a single number. The capabilities you care about are almost always a narrow subset of that. A model that sits mid-table overall may be the strongest available option for German summarization, or for producing valid SQL against your schema, or for calling functions with correctly typed arguments — and the aggregate ranking will never tell you, because those skills are a rounding error in the average.

Why a small model often wins

The argument that a smaller model can beat a much larger one is not consolation for people with modest hardware. It falls out of how these systems get used.

A model that fits entirely in accelerator memory and responds quickly changes what you can afford to do around it. You can supply a longer few-shot prompt. You can retrieve more context and still leave room. You can sample several times and take the consistent answer, or run a second pass that checks the first. You can iterate on the prompt twenty times in an afternoon rather than three. Each of those is an end-to-end quality gain, and collectively they routinely outweigh the raw capability gap between model sizes on a constrained task.

The larger model, meanwhile, is paying for generality you are not using. If the job is extracting six fields from an invoice, world knowledge and multi-step reasoning are not what is being tested. Instruction-following and format compliance are, and those correlate poorly with size beyond a point.

The model is one component

The last piece of framing the course insists on: swapping models is rarely the biggest lever available. Retrieval quality, prompt structure, output constraints and the verification step around the model typically move end-to-end accuracy more than any model change does. People reach for a new download because it is the easiest thing to change, not because it is the most effective. Establishing a measurement first tells you whether the model was ever the bottleneck.

Before quality enters the conversation, three quantities determine whether a model is even a candidate on your machine.

One: the weights

Parameter count multiplied by bytes per parameter. Four-bit quantization stores roughly half a byte per weight, eight-bit roughly one byte, sixteen-bit floating point two. That single multiplication rules out most of the internet's recommendations for most people's hardware in about five seconds.

Mixture-of-experts architectures complicate this in a way worth understanding, because they are now common. Such a model has a large total parameter count and a much smaller number of parameters active for any given token. The total governs the memory footprint, since the experts have to be somewhere. The active count governs how much has to be read per token. The practical consequence is a model that is unusually fast for its size if you can hold it, and unusable if you cannot — there is no graceful middle.

Two: the key-value cache

Every token in context stores a key and a value vector for every layer. The cache size scales with context length, layer count, the number of key-value heads, head dimension and the precision it is stored in. Grouped-query attention reduces the head count dramatically, which is why two models with the same parameter count can have wildly different context costs.

The important habit is treating context length as a purchase. Configuring a long context reserves that memory before you have written a word, and doing so can push a model that comfortably fit into a partial CPU offload, at which point you have traded a large amount of speed for context you may never fill. Cache quantization buys some of it back, at a quality cost that is small for most tasks and not zero.

Three: memory bandwidth

Token generation is memory-bound rather than compute-bound. Producing each new token requires streaming the active weights out of memory, so the ceiling on generation rate is set by how many bytes per second the memory subsystem can deliver, divided by how many bytes each token requires. This is why quantization makes a model faster as well as smaller: fewer bytes read per token.

Prompt processing behaves differently. It handles many tokens in parallel and is compute-bound, which is why time to first token grows with prompt length while the steady-state generation rate does not. Two systems with identical memory capacity and different bandwidth are not equivalent, and a model split between accelerator memory and system RAM is governed by the slower tier plus the cost of crossing the bus.

Get these three numbers right and the shortlist writes itself. Get them wrong and you spend the evening downloading things that were never going to work.

A quantization format is not only a compression scheme. It is a commitment to a runtime, and choosing weights before choosing a runtime is a common and annoying ordering mistake.

• GGUF belongs to the llama.cpp family — llama.cpp itself, Ollama, LM Studio, KoboldCpp and others. Its distinguishing capability is graceful splitting of layers between accelerator and CPU, which no other format handles as well. That flexibility, more than any quality advantage, is why it dominates consumer use. Within GGUF, the k-quant family mixes precision across tensor types rather than applying one width uniformly, and the newer importance-weighted variants use a calibration pass to decide where precision is worth spending.
• AWQ is activation-aware quantization: it identifies weight channels that matter most for output and protects them, targeting four-bit GPU inference with fast kernels. It is aimed at serving stacks such as vLLM rather than at desktop use.
• GPTQ is an earlier post-training method using second-order information to minimize the error introduced by rounding. Plenty of weights still ship in it; it has gradually been displaced for new releases.
• EXL2 allows a variable bit rate across layers, which lets you target a memory budget directly rather than picking from fixed widths. It is tied to the ExLlamaV2 runtime.
• MLX is Apple's array framework with its own quantization scheme, native to unified memory and generally the fastest path on Apple hardware — at the cost of a smaller ecosystem than the CUDA world.
• NF4 through bitsandbytes exists mainly to make fine-tuning fit on modest hardware. It is a training format that also runs, not a serving format.

How much quality does quantization cost

The honest answer is that degradation is gradual down to roughly the four-bit region and steeper below it, that the exact knee depends on the model and the task, and that anyone quoting a universal figure is extrapolating from one experiment. Several patterns are consistent enough to plan around:

Smaller models suffer more than larger ones at the same bit width, because they carry less redundancy to lose. Structured output — JSON, function call arguments, code — degrades noticeably earlier than conversational prose, since a single wrong token breaks a parse while a slightly worse adjective does not. And perplexity is a poor proxy for any of this: a shift too small to register in perplexity can be the difference between valid and invalid output on every request.

The rule the course settles on is conditional rather than absolute. Prefer a larger model at a moderate quantization over a smaller model at higher precision only while the larger one still fits entirely in accelerator memory. The moment it spills into system RAM, the ordering inverts, and it inverts hard.

An advertised context length is a statement about the maximum positional range the implementation will accept. It is not a claim that the model uses that range well, and the two are frequently far apart.

Many long-context figures come from scaling the position encoding beyond the length the model was trained on — RoPE scaling, YaRN and related techniques. These extend the addressable range. They do not manufacture training signal at those lengths. A model stretched to a headline number will accept the tokens and may attend to them poorly.

The published evidence is worth knowing rather than guessing at. Stanford researchers documented, in work titled "Lost in the Middle", that models retrieve information placed near the start and end of a long context considerably more reliably than material buried in the middle. NVIDIA built the RULER benchmark specifically to separate claimed context from effective context, and reported that effective lengths fall short of advertised ones across the models it tested. Neither result is controversial; both are routinely ignored by people pasting an entire codebase into a prompt and wondering why the answer missed the relevant file.

What follows in practice

• Retrieval beats stuffing. Selecting the right few thousand tokens usually outperforms supplying a hundred thousand and hoping attention finds the needle. It is also cheaper in cache memory.
• Position is a design variable. If material must be long, put the critical instruction where the model attends reliably. Instructions placed after a long document frequently outperform the identical instruction placed before it.
• Confirm what the runtime actually applied. Runtimes impose their own context defaults, often smaller than the model supports, and they truncate silently. The number in the model card is not the number in effect.
• Long context has a running cost. The cache grows with the window you configure, and that memory competes directly with the weights. A long context that forces a CPU offload has made the system worse in exchange for capacity you are not using.

The chapter that covers this ends with a small test you can run yourself: place a distinctive fact at several depths in a long document and ask for it back. It takes fifteen minutes and it will change how you structure prompts.

Two configuration issues account for a large share of "this model is not very good" conclusions, and neither has anything to do with the weights.

Sampling settings

The model produces a probability distribution over the next token; sampling settings decide how a token is drawn from it. Temperature flattens or sharpens the distribution. Top-k truncates to a fixed number of candidates. Top-p truncates to a cumulative probability mass. Min-p truncates relative to the most likely token, which adapts better across steps where the model is confident and steps where it is not. Repetition and frequency penalties discourage reuse of recent tokens.

The defaults shipped by most front ends are tuned for open-ended chat, which is the wrong setting for a lot of work. Extraction, classification, code and anything with a required output shape want near-greedy decoding. Prose, brainstorming and dialogue want more entropy. Repetition penalties in particular are actively harmful for structured output, where the repeated tokens being penalized are the closing braces and recurring field names you need.

If valid JSON is a requirement, constrain the decode with a grammar or schema rather than asking politely and parsing hopefully. Every runtime worth using supports some form of this, and it converts a probabilistic problem into a deterministic one.

The chat template

Each model family expects conversation turns wrapped in a particular structure of role markers and special tokens. A client that formats them incorrectly produces output that looks like weak capability: instructions half-followed, the model answering a question you did not ask, responses that trail into invented dialogue. This is the most under-diagnosed cause of disappointing local output, and it is entirely fixable. Before concluding a model is poor, verify the template being applied.

Reporting speed without fooling yourself

Time to first token and steady-state generation rate are different measurements and answer different questions. Report both, and report the conditions with them: prompt length, configured context, quantization, whether the model was already resident, and whether anything else was competing for the accelerator.

A cold-start measurement is mostly a measurement of your storage device. A ten-token prompt hides the prefill cost that will dominate your real workload. A benchmark run once is a sample rather than a result. None of this requires elaborate tooling — it requires stating conditions, which is exactly what the numbers circulating in forums almost never do, and why they cannot be compared to each other.

Public evaluations are useful for one job and misused for another. They are good at narrowing a field of hundreds to a shortlist of five. They are bad at picking the winner among those five for your task, and that is the decision people actually try to make with them.

Human preference arenas, of which the Chatbot Arena work originating at UC Berkeley is the best known, collect pairwise votes on open-ended prompts. What they measure is genuine and genuinely useful: what people prefer when shown two answers. What they also measure, unavoidably, is formatting, length, confidence and tone. A model that presents well scores well. If your application post-processes output into a form nobody reads directly, the thing being ranked is not the thing you need.

Static academic suites — multiple-choice knowledge tests, word-problem sets, coding pass rates — are reproducible and comparable, which is their whole point. Their weaknesses are that they are narrow, that top models have saturated several of them to the point where differences are noise, and that their test items are public text that ends up inside training corpora. Hugging Face rebuilt its open leaderboard partly for these reasons, which is itself a useful signal about how much weight the original scores deserved.

Vendor-published numbers are not usually falsified. They are optimized. The vendor chooses the prompt, the scaffold, the sampling settings, the number of attempts and the comparison points, and every one of those choices is made by someone who wants a particular result. Two vendors' self-reported figures for the same benchmark are two different experiments, and placing them in one table implies a comparability that does not exist.

Contamination is the failure underneath all of it. When evaluation items leak into pretraining data, scores rise without capability rising. It is difficult to prove for any specific model, it is known to happen, and it is the strongest argument for the position this course takes: a small private evaluation built from work nobody has published is worth more to you than any public table, because it is the one thing that cannot have been trained on.

Use the leaderboards to generate candidates. Then stop reading them.

Twenty prompts drawn from real work will tell you more about a model than every benchmark in existence, because they test the distribution you actually operate in.

Building the set

Collect genuine inputs rather than imagined ones — real documents, real questions, real code from the repository you work in. Include the cases that have already failed, since those are where models differ most. Mix routine examples with adversarial ones: ambiguous instructions, missing information, inputs that should produce a refusal or a clarifying question rather than an answer.

Write expected properties, not expected strings. "Returns valid JSON with these five keys", "cites only files that exist", "declines when the document does not contain the answer" are checkable. "Gives a good summary" is not. Properties give you pass and fail; scores out of ten drift with mood and with whichever model you graded first.

Running it without fooling yourself

Blind and shuffle. Label outputs A and B, randomize the order, and judge without knowing which model produced which. Knowing the name anchors judgment more strongly than most people believe about themselves.

Run each prompt more than once, because sampling is stochastic and a single generation is a sample rather than a measurement. Hold a few prompts back so you can detect the point at which you have tuned your prompts to one specific model rather than improved them generally. And version the evaluation alongside the prompt template: change either and prior results are no longer comparable, which is worth writing down at the top of the file.

Licences before you ship

Open weights and open source are not the same category, and the difference has consequences. Some families ship under permissive licenses that impose almost nothing. Others ship under bespoke community licenses carrying acceptable-use policies, attribution or naming requirements, and conditions that activate at commercial scale. Some releases are research-only and are not licensed for production at all.

Two details catch people out. The license attaches to derivatives, so a model you fine-tune inherits the terms of what you fine-tuned. And the license on a quantized or re-uploaded copy is whatever the original publisher granted, regardless of what the re-uploader wrote in the repository description. Check the source, once, before the model is embedded in a product.

Keeping the shortlist alive

The list rots. New releases arrive constantly, and the honest way to handle that is to treat the shortlist as a maintained document: what you tested, on which hardware, at which quantization, on which version of the evaluation, and when. When something new appears, run the evaluation rather than reading the announcement — it takes less time than the argument you would otherwise have with yourself. Retire entries that no longer have a job. A shortlist of four models with dates beside them is worth more than a folder of forty downloads and a vague memory of which one felt good.