Local Image Generation: Stable Diffusion, FLUX & ComfyUI

A diffusion model is trained by destroying images and learning to undo the damage. The forward process adds Gaussian noise in small steps until the picture is indistinguishable from static. The network learns to look at a noisy image and predict what noise was added. Generation runs that in reverse: start from pure noise, ask the model what noise to remove, remove part of it, ask again. Nothing is retrieved from a database. What you get is the model's best guess at a clean image consistent with the conditioning you supplied.

The reason any of this fits on a desktop is latent diffusion. The paper that introduced it — Rombach and colleagues at CompVis and Stability AI, presented at CVPR 2022 — moved the denoising loop out of pixel space and into a compressed latent space produced by a variational autoencoder. Denoising a small latent tensor is dramatically cheaper than denoising a full-resolution image, and that single architectural decision is what put image generation on consumer hardware. It also explains a whole class of bugs. The VAE is a separate component that encodes into the latent space at the start and decodes out of it at the end, so if you load a VAE that does not match your checkpoint, every step in between is fine and the final decode hands you something washed out, over-saturated or blotchy.

Samplers, steps and guidance

The sampler is the numerical solver that walks the reverse process, and the scheduler decides how much noise is targeted at each step. This distinction matters more than it looks. The DPM-Solver++ work by Lu and colleagues was motivated precisely by reaching a converged sample in fewer solver steps than the first-order methods it replaced, which is why the DPM++ family paired with a Karras noise schedule became the common default over plain Euler. Ancestral samplers — the ones with an "a" in the name — inject fresh noise each step and never fully settle, which is why adding steps keeps changing the image instead of refining it. Step count has a knee: below it the image is undercooked, above it you are burning time for differences you cannot see.

Classifier-free guidance is the knob usually labeled CFG. Mechanically it extrapolates away from the model's unconditioned prediction toward its conditioned one, amplifying the influence of your prompt. Raise it and adherence improves until contrast blows out and the image looks scorched. The useful range differs sharply between model families, and guidance-distilled models have the guidance behavior baked in rather than applied at sampling time. That is why a negative prompt that reliably removes an artifact in SDXL does nothing at all in FLUX schnell: there is no unconditional branch for it to push against.

Seeds are what make this a craft instead of a slot machine

Fix the seed, sampler, scheduler, step count, guidance value, resolution and model, and you get the same image every time on the same hardware. Change exactly one and you can see what that variable does. This is the difference between people who improve at image generation and people who reroll for an hour. Almost every genuinely useful technique in the course — comparing checkpoints, tuning a LoRA weight, deciding whether a ControlNet is helping — depends on holding the seed still and moving one thing.

Prompting is real but it is the smaller half. A prompt is one conditioning signal among several, and it is the least precise one available. Once you can supply a depth map, a pose skeleton, a reference image, a mask and a trained style adapter, the text prompt stops carrying the whole burden and becomes what it should be: a description of subject and mood, not a wish list of technical corrections.

Two open model families dominate serious local work, and they fail in different directions.

SDXL, described in Stability AI's technical report by Podell and colleagues, uses two text encoders together — CLIP ViT-L and OpenCLIP ViT-bigG — in front of a UNet the report puts at roughly 2.6 billion parameters, trained natively at 1024 pixels with aspect-ratio bucketing and shipped alongside an optional refiner. The practical consequences follow from that description. It responds well to compact, keyword-leaning prompts, partly because CLIP was trained on caption-like text and partly because most community fine-tunes were trained on tag-heavy datasets. It is small enough that ordinary people can fine-tune it, which is why the ecosystem of checkpoints, LoRAs and ControlNets around SDXL is enormous and mature. And it degrades badly when you generate far from its trained resolutions, producing the duplicated heads and spare limbs that everybody recognizes.

FLUX.1 from Black Forest Labs took a different route: a rectified flow transformer rather than a UNet, conditioned by a T5 encoder alongside CLIP. The T5 encoder is why FLUX follows long natural-language descriptions so much better, and why it can place short spans of legible text inside an image. It is also why the memory footprint is awkward. The open-weight dev checkpoint is a 12 billion parameter model, which at two bytes per parameter is about 24 GB of weights before the text encoder is loaded at all. Almost nobody runs it that way on consumer hardware; the normal setup is an fp8 or GGUF build with the text encoder offloaded to system RAM.

Three FLUX.1 variants matter and they are not interchangeable. The pro model is API-only. The dev model has open weights under a non-commercial license and is guidance-distilled. The schnell model is Apache 2.0, distilled down to very few steps, fastest of the three and the least steerable. FLUX.2 continued the line with a larger open-weight dev release that adds multi-reference conditioning and, being larger again, effectively demands quantization on a consumer card.

How to compare a new checkpoint

SD 1.5 is still around and still has a use: it is tiny, extremely fast, has a vast back catalog of adapters, and produces visibly lower fidelity. It remains a reasonable choice for iterating on composition cheaply before switching to a heavier model for the final render. A stream of newer open releases keeps arriving, and the point of learning this material is not to memorize a ranking that will be stale in a quarter. It is to be able to pick up an unfamiliar checkpoint, read its model card, work out what prompt style and guidance range it wants, and find out within an hour whether it is better than what you already run.

The gap between someone who generates pretty images and someone who can deliver a specific image on request is entirely about conditioning the model with something other than words.

ControlNet

ControlNet, introduced by Zhang, Rao and Agrawala at ICCV 2023, freezes the base model and trains a copy of its encoder connected through zero-initialized convolutions, so training begins as a no-op and cannot damage the original. In use it takes a structural hint — Canny edges, a depth map, an OpenPose skeleton, a scribble, a normal map, a segmentation mask, lineart, or a tile — and holds the generation to it.

Two practical points get missed. First, the preprocessor matters as much as the ControlNet itself: a depth estimator that flattens your subject into the background gives the model a bad map to obey, and the output will faithfully reflect the bad map. Look at the preprocessed image before you blame the generation. Second, control strength and the start and end percentages are more useful than most people realize. Composition is decided in the early steps and detail in the late ones, so applying control for the first portion of the schedule and releasing it afterwards usually gives you the structure you wanted without the stiffness that full-strength control produces. Stacking several ControlNets is possible, but they compete for the same latent, and weights that work alone will fight each other.

LoRA, and why the dataset is the whole job

Low-rank adaptation, introduced by Hu and colleagues at Microsoft for language models, injects small trainable matrices into attention layers instead of updating all the weights. Applied to diffusion, it is how you get a consistent style, a recurring character or a specific product into a model that has never seen them.

Training one is mostly dataset work. Rank, alpha and learning rate matter, but a small set of clean, varied, consistently captioned images is generally a better starting point than a large scraped one, and adding images that repeat what the set already contains tends to make things worse rather than better. Overfitting has a recognizable signature: the LoRA starts dragging training-set backgrounds, watermarks or one particular pose into every generation, and the trigger word stops behaving like a modifier and starts behaving like an entire prompt. When that happens the answer is almost never more training. It is fewer near-duplicate images, more variety in framing and lighting, and captions that describe what varies rather than what is constant.

Adapters are architecture-specific. An SDXL LoRA will not load onto FLUX or SD 1.5, because the tensor shapes do not match. This is the single most common "my LoRA does nothing" cause, ahead of forgetting the trigger word.

Reference-image adapters in the IP-Adapter family solve a related problem from the other side: they condition on an image rather than text, which gets you style transfer and rough character consistency without training anything at all. For one-off work that is often the right tool, and training a LoRA is the answer only when you need the same thing repeatedly.

Editing: inpainting, outpainting and the seam problem

Inpainting masks a region and denoises only inside it. Whether it works comes down to four settings. Denoise strength decides how much of the original structure survives — low values retouch, high values replace. Whether the model processes the whole image or only a crop around the mask changes both quality and memory use, and cropping usually wins because the masked region gets more effective resolution. Mask blur hides the boundary. Context padding decides how much surrounding image the model can see, which determines whether the new pixels match the existing lighting. The classic failure is a patch that is entirely correct in isolation and obviously pasted in, and it is nearly always a hard-edged mask with too little context.

Outpainting is the same operation on a padded canvas, with the extra difficulty that the model has context on one side and nothing on the other, so it drifts. Expanding in several modest passes beats one large expansion.

Upscaling is two different things

Pixel-space super-resolution, the Real-ESRGAN family and its many community fine-tunes, enlarges what is there. It is fast, faithful and invents nothing, which means a soft source stays soft and skin can go waxy. Diffusion-based upscaling — a high-resolution second pass, or tiled re-diffusion in the Ultimate SD Upscale mould — re-runs the sampler at a larger size with low denoise and adds detail that was never in the original. That is exactly what you want on fabric, foliage, hair and stone, and exactly what you do not want on text, logos and background faces, which it will cheerfully rewrite into nonsense. Tiling brings its own artifacts: visible seams when overlap is too small, and repeated motifs when the prompt is applied to every tile and each one tries to render the main subject.

During a run the card is holding model weights, the text encoder, the latent, the activations for the current step, and finally the VAE decode. That last item is a distinct memory spike proportional to output resolution, which produces one of the more confusing beginner failures: a workflow that samples happily at high resolution and then crashes on the final step. Tiled VAE decode exists precisely for this, and switching it on costs a little time and solves the problem outright.

The arithmetic worth internalizing is parameters multiplied by bytes per parameter. Half precision is two bytes, fp8 is one, four-bit quantized formats are roughly half again. That single relationship is why quantization is the largest lever available on a small card, and why FLUX went from impractical to routine on mid-range hardware once fp8 and GGUF builds appeared. Quantization is not free. It generally shows up as slightly softer fine detail and worse text rendering, the loss is uneven across models, and the only way to know whether it matters for your work is to render the same seed both ways and look.

Beyond precision, the levers are offloading and attention. Text-encoder offload to CPU is close to free, because a large encoder like T5 runs once per prompt rather than once per step, so the transfer cost is amortized over the whole generation. Sequential model offload trades a lot of speed for the ability to run something that otherwise will not load. Attention backend choice — PyTorch scaled dot-product attention, xFormers, Flash Attention or one of the newer sparse variants — affects both memory and throughput, and is the most common source of "it worked yesterday" breakage after an update.

Choosing hardware without being sold to

Order the criteria: VRAM first, memory bandwidth second, raw compute third. A card with more memory and less compute will run models that a faster card physically cannot load, and no amount of speed compensates for not fitting. Compute determines how long a generation takes; memory determines whether it happens at all. That asymmetry is why buying advice for image generation looks so different from buying advice for gaming.

Apple Silicon is a genuine option with a genuine catch. Unified memory means models fit that would never fit on a similarly priced discrete GPU. The catch is that MPS support across the diffusion stack lags CUDA, some custom nodes silently fall back to CPU, and throughput is lower than the memory capacity suggests. AMD through ROCm on Linux is workable and improving steadily; on Windows it remains the harder path, and a meaningful fraction of community tooling assumes CUDA without saying so.

Most of the time lost to local image generation is not spent generating. It is spent on a short list of recurring failures, and knowing the shape of each one turns an evening into ten minutes.

Out of memory on the last step rather than the first. That is the VAE decode spike. Enable tiled decoding, or generate smaller and upscale in a second pass.

Washed-out, gray or over-saturated output. Almost always a VAE mismatch. Either the checkpoint has a VAE baked in and you loaded a second one over it, or you loaded one built for a different model family.

A LoRA that appears to do nothing. In order of likelihood: it was trained for a different base architecture, the weight is too low to see, or the trigger word from the model card was never used. The console usually says which, if you read it.

Custom node installs that fail. ComfyUI's extension ecosystem is Python packages with pinned dependencies, and two popular node packs wanting incompatible versions of the same library is routine. Install nodes deliberately rather than in bulk, keep the application in its own virtual environment, and read the terminal rather than the Manager dialog, because the actual pip error is in the terminal.

A downloaded workflow that loads broken. Workflows are JSON that references node classes by name. If the author used a custom pack you do not have, the graph opens with red boxes. This is normal, expected, and fixed by installing the pack, not by editing the JSON.

The model reloading on every single run. That is system RAM pressure evicting cached weights. More system memory, or keeping fewer models resident at once.

Duplicated subjects, stretched torsos, extra limbs. Generating far outside the resolution range the model was trained on. Generate near the native size and enlarge afterwards.

Scorched, over-contrasted images. Guidance set too high for that model, or a distilled model being driven as though it supported classifier-free guidance in the first place.

Results that will not reproduce. A different sampler, a silently updated custom node, a different scheduler, or different hardware. Record the whole configuration, not just the prompt, and record the model file hash rather than the filename.

Faces and hands. Still the weak point of every model in this class, and still solved the same way: detect the region, crop it, regenerate the crop so that the face occupies the model's full working resolution, and composite it back. A detection-and-refine pass is a standard part of a serious workflow rather than an admission of failure.

Three separate questions get collapsed into one, and they have different answers: may you use the weights, may you use the outputs commercially, and does anyone own the result.

The weights. SDXL ships under the CreativeML Open RAIL++-M license, which permits commercial use and attaches a list of use-based restrictions that travel with the model to anyone you pass it to. FLUX.1 schnell is Apache 2.0, about as permissive as open weights get. FLUX.1 dev is under a non-commercial license from Black Forest Labs, with a separate paid commercial license available. That last one catches people constantly, because dev is the variant most tutorials demonstrate and the one most workflows on the internet assume. Community fine-tunes inherit whatever the base model imposed and occasionally add terms of their own, so a model card is a license document rather than decoration.

Copyright in the output. The United States Copyright Office has taken the position that material generated by AI without sufficient human authorship is not protected by copyright, while works containing meaningful human authorship can be registered for the human-authored elements. Other jurisdictions have reached different conclusions and the position is still moving. The practical implication is worth sitting with: you may be perfectly entitled to sell an image and completely unable to stop somebody else copying it.

Everyone else's rights. Trademark, character copyright and personality or likeness rights apply to a generated image exactly as they apply to a drawing. A model license says nothing about whether you may sell a picture of a recognizable person or a protected character, and a LoRA trained on a living artist's portfolio or a specific individual's face raises questions the license file does not answer for you.

Platform policy is a fourth layer. Stock libraries, print-on-demand services, game asset marketplaces and app stores each set their own rules on AI-generated content. Some require disclosure, some restrict categories, some prohibit it outright, and all of them change the rules periodically. Anyone building income around generated images should read the policy of the specific platform they intend to sell on before building the pipeline, not after.

The honest readiness test is temperament rather than skill. If you can install a Python application, follow a stack trace to the line that failed, and tolerate a tool that shows you its wiring instead of hiding it, you have enough. If your only experience is a hosted prompt box, expect the first session to feel like a step backwards, because a node graph forces you to specify a dozen things that a hosted tool quietly decided for you. That is the cost, and the control is what you get for paying it.

A sequence that works: get one image out of a stock SDXL workflow before changing anything at all. Then hold a seed fixed and learn the sampler, steps and guidance triangle by moving one variable at a time, because everything later assumes you can read what a change did. Then ControlNet, which is the point where the process stops being a gamble. Then using other people's LoRAs, well before training your own. Inpainting and upscaling are finishing skills and belong after you can reliably produce a good base image, since neither will rescue a weak one. Training a LoRA comes last, because it is the first point where dataset discipline matters more than any setting in the interface.

Two adjacent directions are worth knowing about. Local video generation reuses the same diffusion machinery and multiplies the memory problem by the frame count, so it is a natural next step for anyone who has already solved their VRAM constraints. And if you also want to run language models locally, the hardware conversation converges on the same card with a different bottleneck: image generation at high resolution tends to be compute-bound, while text generation is dominated by memory bandwidth. Understanding why the same GPU behaves differently for the two workloads is most of what you need to buy hardware sensibly for either.