AI: Beginning to Advanced

Strip away the vocabulary and there are four moving parts. A parameterized function that maps inputs to outputs. A loss that scores how wrong the output was. A way to compute the derivative of that loss with respect to every parameter. An update rule that moves the parameters slightly downhill. That is the whole of it. Convolutional networks, transformers, diffusion models and reinforcement learning agents are all variations on which function you chose and what you scored.

This is worth stating plainly because the field's presentation obscures it. Architecture diagrams imply that the interesting content is in the boxes. It usually is not. The interesting content is in what the loss rewards, what inductive bias the architecture encodes, and whether the optimization can actually find a good solution in the shape of landscape you created.

A single artificial neuron makes the point. It takes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinearity. Without the nonlinearity, stacking layers is pointless: a composition of linear maps is another linear map, so a hundred layers would have exactly the representational power of one. That single observation is why activation functions exist, and it also explains why the choice of activation is not cosmetic. Saturating functions squash large inputs toward a flat region where the derivative approaches zero, which starves the layers beneath them of gradient signal. The move toward rectified units and their variants was driven by gradient flow, not by expressiveness.

The perceptron history is instructive rather than decorative. Rosenblatt's perceptron could only separate linearly separable data, a limitation made famous by Minsky and Papert's critique, and the field's response — stack the units and find a way to train the stack — is precisely the problem that backpropagation solved. Understanding what could not be represented, and then what could not be trained, tells you which of today's limitations are representational and which are optimization artifacts. Those get confused constantly.

Once the core is genuinely internalized, the rest of the material stops being a list of techniques to memorize and starts being a set of design choices with visible reasons.

Backpropagation is bookkeeping. You have a computation graph; you know the derivative of each primitive operation; the chain rule tells you how to compose them; reverse-mode traversal computes every parameter gradient in roughly the cost of one forward pass. There is no magic and no approximation. What there is, is a great deal of opportunity for the numbers to go bad.

Gradients that vanish or explode

Because the chain rule multiplies terms along the path from loss to parameter, a deep path multiplies many numbers together. If those numbers are consistently below one, the product collapses toward zero and early layers stop learning. If they are consistently above one, the product blows up and the update throws the parameters somewhere useless. Nearly every structural innovation in deep networks is a response to this: residual connections give the gradient a short path back, normalization layers keep activations in a range where derivatives are well behaved, careful initialization schemes set the initial scale so the forward pass neither shrinks nor amplifies signal layer over layer, and gradient clipping is the blunt instrument for recurrent models where the path length is the sequence length.

The failures that do not raise errors

A wrong gradient does not crash. It trains, slowly and badly, and looks like a hyperparameter problem. The habits worth building early are gradient checking against numerical differences on a tiny example, overfitting a single batch deliberately to prove the model can memorize before asking it to generalize, and watching gradient norms per layer rather than only the loss curve. A model that cannot drive the loss to nearly zero on ten examples has a bug, not a data problem.

Optimization is not a solved detail

Plain gradient descent, momentum, and adaptive methods behave differently on the same problem. Adaptive optimizers converge fast and can generalize worse; learning rate schedules matter more than most people expect; batch size interacts with learning rate in ways that make results non-transferable between machines. Reproducing someone's reported result while changing the batch size to fit your GPU is a standard way to conclude that a technique does not work when in fact you changed the optimization problem.

There is a strong argument for implementing a miniature tensor library with reverse-mode autodiff before leaning on PyTorch. Not because you will use it, but because PyTorch's API stops being arbitrary once you have written the thing it is a polished version of.

Building it forces several ideas into the open. A tensor is a buffer plus a shape plus a stride, which is why some reshape operations are free and others copy. Broadcasting is a rule for aligning shapes without materializing data, which is why a shape mismatch error is usually a design mistake three lines earlier. Autograd requires each operation to record enough context to compute its own local derivative, which is why intermediate activations dominate training memory and why gradient checkpointing — recomputing activations instead of storing them — trades compute for memory rather than being free.

It also makes the leaky parts of the abstraction visible. Detaching a tensor severs the graph, which is the correct move in some places and a silent bug in others. In-place operations can invalidate a value some backward function still needs. Moving a tensor between devices is a copy with real cost. Eager execution is easy to debug and leaves optimization opportunities on the table; graph capture and compilation reclaim some of them at the price of much worse error messages and surprises around dynamic control flow.

Once your own version trains a small network correctly, comparing it against the production framework becomes a genuinely useful exercise. The numerical differences you find are almost always about numerical precision, default initializations, or how reductions are ordered — and those three things account for a surprising share of the reproducibility complaints in the field.

The point is not craftsmanship for its own sake. It is that debugging a training run requires a mental model of what the framework is doing, and reading the documentation does not produce that model. Writing the smallest version that works does.

At some point the constraint stops being your understanding and becomes your hardware. Treating the accelerator as a black box that makes things faster is an expensive habit, and it is the one that most reliably caps how far an experiment can be pushed.

A GPU is a throughput device with a memory hierarchy. It has a large pool of high-bandwidth memory, a much smaller and much faster shared memory and register file close to the compute units, and thousands of threads scheduled in groups. Performance follows from arithmetic intensity: how many operations you perform per byte moved. A large matrix multiplication has high arithmetic intensity and saturates the compute units. An elementwise activation over a large tensor does almost no arithmetic per byte and is limited purely by memory bandwidth. This is why fusing elementwise operations into a single kernel is such a common optimization and why a model that looks compute-heavy on paper can spend most of its time waiting on memory.

Occupancy, kernel launch overhead and synchronization are the other recurring themes. A network built from many tiny operations pays launch overhead repeatedly and leaves the device idle between them. Calling anything that forces a synchronization inside the training loop — printing a loss value, converting a tensor to a Python float — serializes the pipeline. Data loading that cannot keep up leaves the accelerator starved, which shows up as low utilization and gets misdiagnosed as a slow model.

Memory is the other hard wall. Parameters, gradients, optimizer state and stored activations all occupy it simultaneously, and activation memory scales with batch size while parameter memory does not. Mixed precision reduces both memory and bandwidth pressure by holding most tensors in a reduced floating-point format while keeping a higher-precision copy of the weights for the update, with loss scaling to stop small gradients underflowing. Gradient accumulation simulates a large batch on a small card at the cost of wall-clock time.

The habit worth building is profiling before optimizing. Intuition about where the time goes is wrong often enough that any serious speedup work should start with a trace, not a guess.

Architectures are not a ranked list. Each family encodes an assumption about the data, and the assumption is the whole value.

Convolutional networks assume that useful features are local and that a feature is worth detecting regardless of where it appears. Weight sharing across positions gives translation equivariance and drastically fewer parameters than a dense layer over the same input. That assumption is excellent for images and poor for data with no spatial structure.

Recurrent networks assume sequential dependence and carry a hidden state forward. Gated variants such as LSTM and GRU exist because the plain version could not hold information across long gaps, for exactly the gradient-multiplication reason described earlier. Their unavoidable weakness is that the recurrence is sequential, so training cannot be parallelized across time steps — which is precisely the constraint attention removed.

Transformers replace recurrence with attention over all positions, which parallelizes beautifully and costs quadratically in sequence length. That quadratic term drives an entire research area: sparse and windowed attention, memory-efficient kernels that avoid materializing the attention matrix, and state-space models such as the Mamba family that revisit recurrence with a formulation that can be trained in parallel.

The generative families differ in what they optimize. Autoencoders and variational autoencoders learn a compressed latent representation with an explicit reconstruction objective, which makes them stable to train and prone to blurry outputs. Generative adversarial networks replace the reconstruction objective with a discriminator, which produces sharp samples and a training process that is genuinely difficult — mode collapse, where the generator finds a narrow set of outputs that fool the discriminator, is the canonical failure. Diffusion models learn to reverse a gradual noising process, which turns generation into many small, individually easy prediction steps. The result is stable training and excellent sample quality, paid for with expensive multi-step sampling, which is why so much diffusion work is about reducing the number of steps.

Reinforcement learning is a different problem class altogether: no fixed dataset, a reward that arrives late and sparsely, and a data distribution that shifts as the policy improves. Its instability is not a sign of poor implementation. It is intrinsic to optimizing against a moving target.

Knowing which assumption a family encodes is what lets you choose one for a new problem instead of reaching for whatever was in the last paper you read.

A trained model is an artifact, not a product. The distance between the two is where most machine learning effort actually goes, and it divides into three problems.

Making training fit. When a model no longer fits one device, parallelism strategies diverge. Data parallelism replicates the model and splits the batch, which is simplest and hits a communication wall as gradients must be synchronized every step. Tensor parallelism splits individual layers across devices, trading heavy inter-device traffic for the ability to hold a larger layer. Pipeline parallelism splits by depth and introduces bubbles that scheduling tries to fill. Sharding optimizer state and gradients across devices removes a large chunk of redundant memory. These compose, and the right combination depends on your interconnect far more than on your model.

Making inference affordable. Post-training quantization reduces numeric precision after the fact and is fast to apply; quantization-aware training bakes the reduced precision into the training process and generally holds quality better at aggressive settings. Pruning removes weights or whole structures, with structured pruning giving real speedups and unstructured pruning giving compression that hardware often cannot exploit. Knowledge distillation trains a smaller student to match a larger teacher's outputs, which frequently transfers more than training the small model from scratch would. On edge hardware, all of this is bounded by what the target runtime actually supports, and that support is narrower than the research literature implies.

Making it stay correct. Data drifts, upstream schemas change, and a feature computed one way in training and another way in serving — training-serving skew — is a failure that produces no error and no alert, only a model that performs worse in production than it did in evaluation. Versioning the data and the code together, tracking experiments so a result can be reproduced months later, monitoring input distributions rather than only output metrics, and shadow-deploying before switching traffic are the practices that separate a model that survives from one that quietly degrades until somebody notices a business number moving.

None of this is glamorous, and all of it is the difference between research code and something an organization can depend on.

This course is deliberately steeper than a general introduction, and starting it underprepared wastes more time than spending a fortnight on the gaps first.

A reasonable self-test: can you multiply two matrices by hand and say what the shapes must be for the operation to be defined? Can you take a partial derivative of a composed function and explain which term came from where? Can you write a Python class with methods that mutate state, and read a stack trace to the line that actually caused the failure? Can you sit with a bug for an hour without concluding the library is broken? If three of those four are yes, you will be fine; the fourth tends to develop under pressure.

What you do not need is prior deep learning experience, familiarity with any specific framework, or a background in research. The material builds the neuron before it builds the network and the network before it builds the framework.

There is a genuine choice to make about ordering. If your goal is to ship an application that calls a model, this is not the shortest path and an application-layer course will get you there faster. Come here when you want to know why the model behaves as it does, when you need to modify rather than call, or when you are being asked derivation questions in interviews and finding that API familiarity does not answer them.

The honest expectation to set: the early chapters feel slow because they are building foundations you cannot skip, the middle chapters are where most people either commit or drift away, and the systems material at the end is what makes the whole thing employable rather than merely interesting. Finishing the architecture chapters and stopping before the production chapters is a reliable way to end up with knowledge that does not convert into work.