Dataset Engineering: Build the Data That Makes Models Great

Dataset engineering is the practice of treating a training corpus as a built artefact with a specification, a build process, tests and a version history — rather than as a folder of files someone assembled once.

It exists as a distinct discipline because the leverage moved. When architectures were changing every few months, architecture work dominated. Now that most teams are fine-tuning or adapting an existing base model, the architecture is fixed, the compute budget is fixed, and almost the only lever left is what the model sees. Two teams with identical hardware, identical base weights and identical hyperparameters will produce very different models, and the difference will be the corpus.

The work is not data science and it is not analytics. A data scientist asks what the data says. A dataset engineer asks what the data will teach, which is a different and more adversarial question, because a model will learn the artefacts as readily as the signal. Boilerplate that appears on every page becomes a stylistic tic. Template text in a scraped forum becomes a phrasing the model reaches for unprompted. Duplicated passages become memorised verbatim. None of these show up in a summary statistic; they show up in generations.

A useful mental model is that a dataset has an anatomy. There is the raw layer, which you never delete because you will want to reprocess it under new rules. There is the processed layer, produced by a documented and re-runnable transformation. There is the training view, which is a specific mixture of processed sources at specific weights. And there are the evaluation sets, which must be constructed to be disjoint from all of the above and are the only thing standing between you and self-deception.

The habits that distinguish an engineered corpus from a collected one are unglamorous: every record carries provenance, every filter is a versioned piece of code rather than a manual pass, every rejection is sampled and reviewed rather than discarded silently, and every mixture is reproducible from a manifest. That discipline is what makes it possible to answer the only question that matters when a model regresses, which is what changed in the data.

Every corpus starts with an acquisition decision, and acquisition decisions are the ones most likely to be irreversible.

Sources fall into a few recognisable groups. There are open crawls and open corpora, where the collection work has been done and the remaining task is filtering — the Pile from EleutherAI, RefinedWeb from the Technology Innovation Institute, Dolma from the Allen Institute for AI and FineWeb from Hugging Face all published their construction methodology, which makes them worth reading as engineering documents rather than only as downloads. There is targeted crawling, where you go after a specific domain. There is licensed and purchased data. There is internal data, which is usually the highest-value and highest-risk category. And there is synthetic generation, covered separately below.

Three concerns need to be settled before a single byte is stored, not after.

Licensing and terms. Permission to read a page is not permission to train on it. Licences differ per source, sometimes per file within a source, and a permissive licence on a repository does not extend to vendored dependencies inside it. Record the licence per record, not per dataset, because you will eventually need to produce a subset that excludes a licence class.

Personal data. Crawled text contains email addresses, phone numbers, keys and occasionally credentials. Detection at ingestion is cheaper than remediation after training, and remediation after training may mean retraining. Regulations such as the GDPR contemplate rights over personal data that are extremely awkward to honour once that data is inside model weights.

Provenance. Every record should carry where it came from, when it was fetched, under what licence, and which pipeline version produced it. This feels like overhead on day one. It is the only thing that lets you answer a takedown request, reproduce a build, exclude a source retrospectively, or explain to a customer what the model was trained on.

Then there is the sampling question, which is where judgement enters. A corpus is not a sample of the internet, it is a curriculum. Mixture weights across sources are a design choice with visible downstream effects: over-weight code and prose style shifts, over-weight one forum and its register leaks everywhere. Decide the mixture deliberately, write it down, and treat changing it as a change that requires re-evaluation.

Raw collected text is mostly unusable. The filtering stage is where most of the volume disappears and most of the quality appears.

Filtering

Filters stack in rough order of cost. Cheap structural heuristics come first: language identification, length bounds, character-class ratios, symbol-to-word ratios, repeated-line detection, and boilerplate stripping. These are crude and they remove an enormous amount of genuine garbage for almost no compute.

Above them sit statistical filters. Perplexity scoring against a small reference model flags text that is anomalous relative to a domain, which catches machine-translated sludge and encoding damage. Above those sit learned quality classifiers, often trained to distinguish curated reference text from raw crawl, which is a technique the open corpus projects describe in their papers.

The discipline that matters more than the specific filter is calibration. Every filter has a threshold, every threshold discards real content, and nobody can guess the right value. Sample the rejects. Read them. A filter that is removing exactly what you intended looks completely different from one that is quietly deleting an entire language, dialect or subject area, and only manual inspection of the rejected pile tells them apart.

Deduplication

Duplication is the failure that costs the most and shows the least. Duplicated text inflates the effective weight of whatever is duplicated, increases verbatim memorisation, and corrupts evaluation when a duplicated document straddles the train and test split. Google Research published work specifically arguing that deduplicating training data produces better language models, and the finding has held up across subsequent corpus projects.

There are three levels, and they cost very different amounts.

• Exact duplicates. Hash the normalised content and drop repeats. Trivial, and worth doing first because it shrinks everything downstream.
• Near duplicates at document level. MinHash combined with locality-sensitive hashing is the standard approach: shingle the document, build a compact signature, and bucket signatures so that only plausible pairs are compared. SimHash is the common alternative. Both trade recall for the ability to finish.
• Substring and passage level. Suffix-array approaches find long repeated spans that document-level methods miss entirely — the same licence header, the same scraped boilerplate paragraph, the same recycled article body inside otherwise different pages.

Two practical warnings. Deduplication is quadratic if you write it naively, and the naive version will appear to work on your ten-thousand-document sample and then never finish on the real corpus. And the normalisation you apply before hashing silently defines what counts as a duplicate; case folding, whitespace collapsing and punctuation stripping each change the answer, so pin them and document them.

Once the base corpus exists, the datasets that shape behaviour are mostly constructed rather than collected.

Synthetic generation

Generating training data with a stronger model is now routine, and it works best when generation is paired with verification. Unverified generation compounds the teacher's errors and its stylistic habits into the student. Verified generation does not.

The SQL case in this course is the clean illustration of why. You can generate a natural-language question and a candidate query, then execute the query against a real schema and discard anything that errors, returns nothing, or fails a semantic check. The verifier turns a plausible-looking pair into a known-correct one. The same pattern generalises anywhere execution is possible: code with tests, mathematics with a checkable result, structured extraction with a schema validator. Where no verifier exists, expect the ceiling to be lower and the diversity to collapse towards whatever the generator finds easy.

Diversity is the second failure mode. Sampling repeatedly from one prompt template produces a dataset with the surface variety of a mail merge. Seeding generation from real artefacts — actual schemas, actual documents, actual tickets — and varying the task framing, difficulty and persona is what keeps a synthetic set from teaching a single narrow pattern.

Instruction data

Instruction tuning datasets are triples of instruction, optional input and response, and their quality is dominated by the response. Common defects are easy to name and hard to resist: responses that begin with a restatement of the question, responses that hedge instead of answering, uniform length regardless of question complexity, and a house style so consistent that the model learns the style rather than the task. Coverage matters too — if every example is a successful request, the model has never seen what refusing, asking for clarification, or admitting ignorance looks like.

Preference data

Preference datasets are pairs of responses with a judgement about which is better, and they feed the preference optimisation methods that shape model behaviour. The construction details determine whether they work. Preferences collected over responses from a different model than the one you are training are less useful than on-policy comparisons. Pairs that are too easy to separate teach nothing. Annotator agreement should be measured rather than assumed.

LLM-as-labeller

Using a model to label or judge data scales annotation dramatically and imports a specific set of biases that are documented in the literature from LMSYS and others: position bias, where the option presented first is favoured; verbosity bias, where longer answers score higher regardless of quality; and self-preference, where a model rates its own outputs generously. The mitigations are mechanical — randomise presentation order, force a rubric with explicit criteria, calibrate against a human-labelled subset, and hold out a human-only evaluation set that no model has ever scored.

A dataset's quality is not a property you can read off a summary table. It is measured by what a model trained on it does, and everything else is a proxy.

That said, the proxies are worth having. Intrinsic checks — length distributions, vocabulary coverage, duplicate rate, language mix, source balance, per-field null rates — catch gross defects early and cheaply. Extrinsic checks train a small model on candidate corpora and compare. The ablation is the only real evidence, and the discipline is to change one thing at a time: same base model, same hyperparameters, same evaluation, one dataset variable. Teams that change the filter and the mixture and the generation prompt together learn nothing from the result.

Contamination is the failure that makes every other measurement meaningless. If evaluation examples have leaked into training data, your benchmark scores measure recall of the answer key. It happens constantly and mostly by accident, because benchmark items are published on the open web and open web crawls collect them.

Defences worth building in:

• Overlap detection. Search training data for n-gram overlap against every evaluation set before training, not after a suspicious result. Substring matching over a suffix array is the thorough version; hashed n-gram intersection is the practical one.
• Canary strings. Some benchmark suites, including BIG-bench, embed a unique identifier specifically so corpus builders can detect and exclude the benchmark. Grep for them.
• Held-out sets constructed by time or source, not by random slice. A random split of a corpus that contains near-duplicates will place near-identical records on both sides, and the resulting evaluation will be optimistic in a way no amount of statistical care can repair. Deduplicate before splitting, and prefer splits along a natural boundary such as document source or publication date.
• A private evaluation set that never leaves your control. The only benchmark guaranteed to be uncontaminated is one nobody has published.

Related and quieter is feature leakage in the ordinary supervised sense: a field that encodes the answer, a timestamp that reveals the label, an identifier that correlates with the outcome. The symptom is a validation score that is too good, and the correct response to a validation score that is too good is suspicion rather than celebration.

A dataset that cannot be rebuilt is a liability, however good it is. The build has to be reproducible, and reproducibility here means something stricter than it does in application code, because the inputs are enormous and the outputs are expensive.

The pattern that works is content addressing plus manifests. Store immutable data blobs keyed by a hash of their contents. Describe a dataset version as a manifest listing those hashes plus the pipeline version and configuration that produced them. A dataset version then becomes a small text file you can commit, diff and review, rather than a directory nobody dares to touch. Tools such as DVC and lakeFS implement variants of this idea; the principle matters more than the tool.

Lineage extends the same idea per record. If you can trace an individual training example back through every transformation to its original source, you can answer questions that are otherwise unanswerable: which upstream source produced this bad behaviour, which records must be removed for a takedown, which subset needs reprocessing after a filter bug.

Pipeline design has a few properties worth insisting on.

Idempotent and resumable. A corpus job that fails eighty per cent of the way through and has to start over will eventually consume more human patience than compute. Checkpoint by shard.

Sharded and streaming. Nothing should require the whole corpus in memory. Shard everything, process shards independently, and make the shard boundary the unit of retry and parallelism.

Incremental. When one filter changes, only the affected stage should re-run. Caching intermediate stages by input hash and configuration hash is what makes iteration on a large corpus feel possible rather than heroic.

Observable. Emit counts at every stage: in, out, rejected, and rejected by reason. A stage that silently drops most of its input is the most common serious pipeline bug, and the only thing that catches it is a number that was supposed to be small suddenly not being small.

Documented in the artefact. A datasheet describing motivation, composition, collection process, preprocessing, known limitations and intended uses should ship with the dataset. The datasheets-for-datasets convention from the research community is a reasonable template. It costs an afternoon and it answers the questions your future colleagues, your auditors and your customers will ask.

A short diagnostic. This material fits you if you have already trained or fine-tuned something, been disappointed by the result, and found that hyperparameter changes did not fix it. It does not fit you yet if you have never run a training job at all, because the value of every decision described here is measured in a training outcome you have not yet learned to read.

There is also a scoping question worth answering honestly before investing. Not every problem is a data problem. If the model needs facts it was never trained on, retrieval is cheaper and updates instantly. If the model has the knowledge but the wrong format, prompting and structured output constraints are cheaper still. Dataset work earns its cost when you need a behaviour, a domain register or a capability that the base model does not have and cannot be prompted into.

If it does apply, the productive order is roughly:

1. Build the smallest honest evaluation set first. A few hundred examples you constructed yourself, disjoint from anything you will train on. Everything downstream is measured against it, and building it first prevents the temptation to define success after seeing results.
2. Get one clean pipeline working end to end on a small corpus. Collection, filtering, dedup, split, manifest. Small enough to iterate in minutes.
3. Scale the stages that break. Deduplication is usually the first to break, then storage layout, then the cost of any per-record model call.
4. Add generation only once verification exists. Synthetic data without a verifier is a way to industrialise your teacher model's mistakes.
5. Run ablations. One variable at a time, same everything else. This is where the discipline actually pays.

From here, the natural continuations are supervised fine-tuning and preference optimisation, which consume what you have built, and evaluation engineering, which is the other half of the same job. The code that accompanies this course is published in a public repository, so the pipeline stages can be read and adapted rather than reimplemented from a description.