Natural Language Processing

Natural language processing is the discipline of turning unstructured text into something a program can act on, and turning a program's intent back into text a person will accept. Large language models have absorbed a great deal of that surface area, and for plenty of tasks the correct first move genuinely is to write a prompt and measure what comes back. The trouble begins when a prompt is the only instrument available, because most of the interesting problems in the field are not "produce a plausible paragraph".

They look more like this. Which of these forty thousand support tickets concern billing, and can you demonstrate that the classifier has not quietly changed its behavior since the last model update? Which span of this contract is the termination clause, with character offsets, so a reviewer can see it highlighted in place? Which documents in a large archive should be surfaced for a three-word query, fast enough that a search box does not feel broken, on hardware that is not allowed to send data anywhere?

Those are four different problem shapes and they want four different things. Classification wants calibrated scores and a decision threshold you can move when the cost of a false positive changes. Extraction wants exact spans and an agreement on how partial overlaps are scored, because "close enough" is a policy decision, not a technical one. Retrieval wants an index, usually a reranking stage, and a definition of relevance that survives contact with real users. Generation wants a decoding strategy, a length policy, and some way of noticing when the output is fluent and wrong.

Collapsing all four into "ask the model nicely" is how teams end up with a system that demonstrates beautifully and cannot be diagnosed. When quality drops, there is nothing to inspect: no confusion matrix, no per-class recall, no held-out set, no record of what the labels were supposed to mean. The prompt gets edited, the demo looks fine again, and nobody can say whether the change helped.

The through-line of serious NLP work is that text is a representation problem before it is a modeling problem. How you split characters into tokens, how you turn tokens into vectors, what you decide counts as a document, and what a human annotator was told to mark all constrain the ceiling of everything downstream. A better model rarely rescues a task whose definition was never pinned down. This course is organized around that ordering: representation, task formulation, model, evaluation, and only then serving.

Every text system begins by chopping a string into units. Word-level splitting was abandoned for good reasons: vocabularies explode, unseen words become a hole in the model, and morphologically rich languages punish you immediately. Subword schemes replaced it. Byte-pair encoding, introduced for machine translation by researchers at the University of Edinburgh, merges frequent character pairs until it reaches a target vocabulary size. WordPiece, used by Google's BERT, optimizes a related objective. SentencePiece, also from Google, works directly on raw text including whitespace, which makes it usable for languages that do not separate words with spaces.

The practical consequences are larger than they look. A tokenizer trained mostly on English prose will fragment other scripts far more aggressively, so the same sentence costs more tokens in one language than another. That inflates latency, inflates cost, and eats into whatever context budget you have. Source code, chemical names, long identifiers, emoji and unusual Unicode all fragment badly for the same reason. If your product handles Japanese, Hindi or code alongside English, tokenization is not a detail, it is a capacity planning input.

Then there are the failure modes that produce silent wrongness rather than errors. Truncation is the classic one: a document longer than the model's window is cut, usually from the end, and the pipeline reports success while quietly discarding the section that mattered. Tokenizer and model version drift is another. Load a checkpoint with a tokenizer it was not trained with and you do not get an exception, you get degraded output that looks like the model got worse for no reason.

Embeddings sit immediately on top. The early static approaches, word2vec from Google and GloVe from Stanford, assign one vector per word type, which means a single vector has to serve every sense of "bank" or "charge". Contextual models produce a different vector for each occurrence, which resolves the ambiguity but changes what you can cache and precompute. Sentence and passage embedding models are trained specifically so that vector similarity approximates semantic relatedness, which is not the same objective as language modeling and not something you should assume any given model was optimized for.

The standard mistakes here are worth naming. Cosine similarity between embeddings from two different models is meaningless. Similarity scores are not calibrated across queries, so a fixed cutoff that works for one query behaves badly for another. Asymmetric retrieval models expect queries and documents to be encoded differently, and ignoring that instruction quietly degrades every result. And embedding a whole page of mixed content into one vector averages away the specific passage you were hoping to find.

For most text problems there are at least four viable implementations, and the interesting engineering question is which one earns its cost.

Rules and patterns. Regular expressions, gazetteers, dictionaries, hand-written normalizers. Unfashionable, instant, free to run, completely inspectable, and genuinely the right answer for well-formed identifiers such as invoice numbers, dates, postcodes and ticker symbols. They fail on variation and they rot when the input format changes, so they need ownership. A large share of production pipelines still start with a normalization layer built exactly this way.

Classical machine learning on sparse features. TF-IDF vectors into a linear classifier or gradient-boosted trees. This approach trains on a laptop without special hardware, serves fast enough that inference latency stops being a design constraint, and produces feature weights a domain expert can read and argue with. For topic-style classification on a decent volume of labeled examples it is a serious baseline, and skipping it means you never find out whether the expensive option is actually better.

Fine-tuned encoder models. A pretrained transformer encoder with a task head, trained on your labels. This is still the workhorse for token classification, span extraction, and high-volume classification where latency and unit cost matter. The models are small enough to serve cheaply on modest hardware, they produce probabilities you can threshold and calibrate, and their behavior is frozen at deployment, which is exactly what you want when an auditor asks what the system did last quarter.

Generative language models. Prompting, optionally with retrieval, few-shot examples, or constrained decoding into a schema. Unbeatable when the label space is open-ended, when you have almost no labeled data, or when the output genuinely needs to be prose. The costs are real: higher and more variable latency, per-token pricing, sensitivity to prompt phrasing, and outputs that can drift when the underlying model is updated beneath you.

The choice is rarely about accuracy alone. Ask how many labeled examples exist, how much variance in latency the product tolerates, whether the output must be explainable to someone outside engineering, and how often the input distribution changes. A common and sensible production shape is a hybrid: rules normalize, a cheap classifier routes, an encoder handles the high-volume path, and a generative model handles the residual it was routed. Also worth remembering that a generative model can label training data for the cheap model, which converts a per-request cost into a one-time cost.

Modeling in NLP has become substantially easier. Evaluation has not, and it is where most projects actually go wrong.

Start with the labels. Text annotation is a judgment task, and two competent annotators reading the same guideline will disagree on real examples. If nobody measured inter-annotator agreement, you do not know whether the task is well defined, and you have no idea what score would represent a ceiling. A model that appears to plateau below human performance may simply have hit the noise floor of the labels it was trained on. The fix is unglamorous: write the guideline, have several people label the same sample independently, measure the disagreement, resolve it, and revise the guideline before touching a model.

Then pick metrics that match the decision. Accuracy is close to useless on imbalanced data, which describes most real classification. Precision and recall need to be reported separately because they trade against each other and the business usually cares far more about one. For span extraction, decide explicitly whether a partial overlap counts, because exact-match and overlap-based scoring can rank two systems in opposite orders. For retrieval, rank-aware measures matter more than set-based ones, since users read from the top.

Generation is the genuinely hard case. Reference-overlap metrics such as BLEU, from IBM Research, and ROUGE, developed at the University of Southern California's Information Sciences Institute, compare output against reference texts and were designed for translation and summarization respectively. They reward surface similarity, which means a correct answer phrased differently scores badly and a fluent, wrong answer that reuses vocabulary can score well. Using a language model as a judge is now common and helps, but it carries its own biases. The MT-Bench work from the LMSYS group at UC Berkeley described several of them explicitly, including sensitivity to the order in which candidates are presented and a preference for longer answers. If you use a model judge, validate it against human ratings on a sample before trusting it as your primary signal.

The structural failure that outranks all of these is a contaminated split. Near-duplicate documents across train and test, splitting a set of ticket threads at message level instead of thread level, or holding out random rows from time-ordered data all inflate scores in ways that look like progress. Split by the unit that will actually be unseen in production: the customer, the document, the time period. When an offline number looks surprisingly good, contamination is the first hypothesis, not the last.

Text in a notebook is clean. Text in production arrives from OCR of a scanned fax, from an email client that mangles quoting, from a form where someone pasted a spreadsheet cell, and from users who type in a language nobody planned for.

Length. Real documents overflow model windows. Naive truncation drops content silently. Chunking introduces its own problems: split mid-sentence and you break meaning, split too large and retrieval returns a wall of mostly irrelevant text. Chunking on structural boundaries such as headings, clauses or speaker turns generally beats fixed character counts, and keeping a little overlap prevents an answer that straddles a boundary from disappearing.

Encoding and normalization. Mojibake, smart quotes, non-breaking spaces, zero-width joiners, mixed line endings and multiple Unicode normalization forms all cause matching failures that are invisible when you print the string. Normalize deliberately and early, and log a sample of raw input so you can see what actually arrived.

Latency and throughput. Per-request encoding is usually not the bottleneck; batching, tokenization overhead, cold model loading and network round trips often are. If you have a retrieval stage plus a reranker plus a generation call, the budget is spent three times. Measure the tail, not the mean, because the tail is what users describe as "it hangs".

Drift. The vocabulary of any live system moves. New product names, new abbreviations, a new customer segment, a change in the upstream form that alters formatting. A classifier trained on last year's tickets degrades gradually and rarely triggers an alert, because the pipeline is still returning confident answers. Monitoring input distribution, prediction distribution and the rate of low-confidence outputs catches this well before anyone files a complaint.

Privacy and retention. Text carries personal data by default. Names, addresses, health details and account identifiers arrive whether or not you asked for them. Detection and redaction of personal information is itself an NLP task with its own error rates, so treat it as a system with a measured recall rather than a checkbox. Where the text cannot leave the building at all, local inference stops being a preference and becomes a requirement.

Multilingual reality. Models trained predominantly on English degrade unevenly across languages, and the degradation is not visible in an English test set. If you serve multiple languages, you need evaluation data in each of them. Machine-translating your English test set to make one is a tempting shortcut that produces translationese and flatters the model.

A reasonable self-check before starting.

You should be able to write a Python script that reads a folder of files, applies a function to each, and writes results to disk, without looking up how to open a file. You should know why a model that scores perfectly on data it was trained on has told you nothing. You should be able to read code that uses arrays and be untroubled by an operation over an axis.

You do not need calculus, and you do not need to have trained a neural network before. The material builds the sequence modeling ideas from a starting point of ordinary programming.

Signals that you are ready and would benefit soon: you have a text problem in front of you right now, you have tried the obvious prompt and it works most of the time, and you cannot explain the cases where it does not. Or you inherited a search feature and want to know whether the answer is a better embedding model, a reranker, or better chunking, rather than trying all three at random.

Signals that you should wait: you have never used git, or you are hoping the course will produce a finished product without any data work of your own. Every serious NLP project involves looking at your own text closely enough to be irritated by it, and there is no version of the discipline that skips that step.

The other honest prerequisite is a source of data. The techniques are transferable, but you learn them by applying them to a corpus you care about and can judge. Bring one.

NLP is the entry point to several adjacent specializations, and it is worth knowing which door you are standing in front of.

If your interest is retrieval and knowledge systems, the natural continuation is retrieval-augmented generation: chunking strategy, hybrid sparse and dense search, reranking with cross-encoders, and evaluation of retrieval quality separately from answer quality. The embedding and evaluation material here is the prerequisite for doing that well rather than assembling a pipeline and hoping.

If your interest is adapting model behavior, the continuation is fine-tuning and preference optimization, where the dataset work covered here becomes the main event. The quality of instruction data determines the outcome far more reliably than the choice of training algorithm.

If your interest is systems, the continuation is serving: quantization, batching, KV cache behavior, and running inference on hardware you control. That is a distinct skill set with its own failure modes, and it is where a working prototype becomes something that survives real traffic.

And if your interest is breadth, the same representation-then-task-then-evaluation discipline transfers directly to speech and vision. The tokenizer becomes a feature extractor, the embedding becomes a patch or frame representation, and the evaluation problems rhyme almost exactly.