AI Engineering: From Prompt to Production

Adding a language model to an application introduces a component unlike anything else in a normal stack, and most of the discipline follows from its three unusual properties.

It is nondeterministic. The same input can produce different output, so equality assertions do not work as tests and reproducing a bug report may be impossible without the original trace. It is priced per token and slower than anything else in your request path, which turns prompt design into a performance and budget decision rather than a purely qualitative one. And its failure mode is not an exception — it is a well-formed, confident, wrong answer that flows downstream and looks exactly like a correct one.

That last property is the one that reshapes system design. Everywhere else in engineering, the contract is enforced: a type checker rejects the wrong shape, a database rejects the invalid row. Here the contract is prose, prose is not enforceable, and a component that fails silently must be wrapped in something that checks it. A large fraction of the code in a mature AI feature exists to constrain, validate, retry, fall back and record — not to call the model.

What the codebase actually looks like

People new to this expect the model call to be the center of gravity. In a system that has survived contact with users, it is a small function surrounded by much larger concerns: ingesting and indexing content, assembling context under a budget, validating output against a schema, handling partial failures, storing conversations and their provenance, metering cost per tenant, capturing feedback, and replaying traffic against candidate configurations. The model is a dependency, in the same sense that a payment provider is a dependency, and the engineering is in everything around it.

Where the human goes

Before writing any of it there is a product decision that determines the entire risk profile: is the model deciding, drafting, or suggesting. A model that drafts something a human approves can be wrong often at low cost. A model that acts autonomously must be right nearly always, or be bounded so that being wrong is cheap and reversible. Teams that skip this decision tend to build the autonomous version by accident, because the demo was impressive, and then discover the review burden they created for someone else. Deciding it explicitly — and designing the interface so the human can actually see what they are approving — is cheaper than retrofitting oversight later.

The first real engineering problem in almost every project is the same: downstream code needs a predictable shape, and the model produces text. The gap between those two facts is where early prototypes die.

There is a progression of techniques, each stricter than the last.

Ask and parse. Request JSON in the prompt and parse the reply. Works surprisingly often and fails in tedious ways — a preamble sentence before the object, a trailing explanation, markdown fences, a trailing comma, an unescaped quote in a string field.

Provider JSON modes. Many APIs guarantee syntactically valid JSON. This removes the parse errors and none of the semantic ones.

Tool and function calling with a declared schema. The model is given a typed signature and asked to fill it. This is usually the right default for hosted models, and the schema becomes documentation for both the model and your code.

Constrained decoding. A grammar or state machine restricts which tokens can be sampled next, so invalid output is not merely unlikely but impossible. Local runtimes support this well, and it is the strongest guarantee available.

What none of them give you

Every technique above guarantees structure. None guarantees truth. A schema-valid object can contain an invented identifier, a date that does not exist, or a category chosen because the enum offered no honest option. Structural validity is a precondition for correctness, never a substitute, and treating a parsed object as trustworthy because it parsed is a recurring source of production incidents.

Two design habits fix most of the remaining problems.

The first is schema design as a quality lever. Flat structures outperform deeply nested ones. Enumerated values outperform free strings when the set is known. Optional fields with an explicit null, or an enum member meaning "not stated in the source", give the model somewhere honest to go — omit it and it will fabricate a plausible value, because your schema told it a value was required.

The second is separating reasoning from extraction. Forcing rigid structure from the first token constrains the model exactly when it needs room to work. Either provide a free-text field for working that your code ignores, or run two passes: reason in prose, then extract into the schema in a second cheap call. The second pass is often small enough to run on a much smaller model.

Then validate properly. Parse with a real validator, apply business rules the schema cannot express, and on failure return the validator error to the model once. A bounded repair attempt is good engineering; an unbounded retry loop is a cost incident waiting for an unlucky input.

Grounding a model in your own content is the most common requirement in this field and the most commonly underestimated. The demo — embed some documents, search, stuff the results into a prompt — takes an afternoon. The version that answers real questions about a real corpus does not.

When a grounded system gives a bad answer, the cause is almost never the model. It is that the passage containing the answer never made it into the context. Diagnosis therefore starts by evaluating retrieval on its own: take a set of real questions, label which documents genuinely contain the answers, and measure how often those documents appear in the top results. If they do not, no prompt change will help, and every hour spent rewriting instructions is wasted.

The levers that move retrieval quality

Chunking that respects document structure. Fixed-size splits cut tables in half and orphan headings from the text beneath them. Splitting on structural boundaries and carrying section headers into each chunk is a small change with a large effect. Retrieving a small precise chunk but sending its larger parent section to the model gives you precision in search and context in generation.

Hybrid search. Dense embeddings capture meaning and miss exact tokens; lexical search does the reverse. Product codes, error identifiers, function names and internal acronyms are precisely where embeddings underperform, and they are precisely what people search for in a corporate corpus. Fusing both result sets is close to free and reliably better than either.

Reranking. A cross-encoder that scores each candidate against the query jointly is more accurate than the vector similarity that produced the candidates. Retrieve generously, rerank, keep few. Of all the changes available, this is usually the one with the best quality-per-unit-of-effort.

Query rewriting. Users do not write queries that look like documents. They write fragments, and in a conversation they write pronouns — "what about the second one" carries no retrievable content at all. Rewriting the query against conversation history before searching fixes a whole class of baffling failures.

The parts nobody demos

Ingestion is the majority of the work and none of the fun. PDFs with multi-column layouts, scanned documents needing OCR, tables that lose all meaning when flattened to text, and the same content duplicated across three systems in slightly different versions. Then the lifecycle: incremental re-indexing when a document changes, propagating deletions so retracted content stops being retrievable, and detecting staleness so the system does not confidently cite a superseded policy.

Permissions deserve their own emphasis. If different users may see different documents, access control must be applied as a filter during retrieval, not by removing results afterwards and certainly not by asking the model not to mention them. A retrieval layer that ignores authorization is a data leak with a chat interface, and it is the fastest way to turn an internal pilot into a security review.

Evaluation is usually the thing a team builds second, after something reached users and misbehaved. The ones that ship confidently build it first, because it is the only thing that lets you change a prompt, swap a model or restructure retrieval without guessing.

Useful evaluation has three layers, and they answer different questions.

Deterministic assertions. Cheap, fast, and more useful than they get credit for. Did the output parse. Does it satisfy the schema. Does every claimed citation correspond to a document that was actually retrieved. Does it avoid emitting content that matches a secret pattern. These run in CI on every change and catch a real share of regressions without any model in the loop.

Reference comparison. Where a correct answer exists — extraction, classification, routing, structured transformation — you can score against it directly. Most business tasks contain more of this category than people assume, and it is worth carving out because the metrics are unambiguous.

Judged comparison. For open-ended output, a model judge comparing two candidates is the practical instrument. Pairwise questions are far more stable than absolute scores. The biases here are documented rather than folklore — the MT-Bench work from the LMSYS group at UC Berkeley characterized position bias, a preference for longer answers, and a tendency for a judge to rate its own outputs generously. Design around all three: swap the order of the candidates and require both passes to agree, report output length beside win rate so a verbosity win is not read as a quality win, and check the judge against human labels on a sample before letting it gate anything. A judge prompt is production code and deserves versioning and review like any other.

Building the set

Golden sets assembled from imagination are pleasant and useless. Build from real traffic: sample production inputs, stratify so rare and difficult categories are represented rather than drowned out by the easy majority, and grow the set every time a user reports something wrong. That last habit is the flywheel of the whole discipline — each production failure becomes a permanent test case, so the same defect cannot return quietly.

Gating in CI needs adapting to nondeterminism. Equality assertions do not survive contact with sampling. What works is fixing what you can, running each case several times, and comparing aggregate pass rates against a threshold with an allowance for noise. Treat a drop as a signal to investigate, not as an automatic block, or the pipeline will be disabled within a month.

Offline scores are a proxy. The measures that decide whether the feature is working live in the product: how often users edit the output, how often they retry, how often they abandon, how often a conversation escalates to a human. Instrument these from day one, because they are also the labels you will wish you had later.

The underrated skill is error analysis. Read a sample of failures and sort them into causes. Teams consistently find that one category dominates — retrieval missing a document type, a schema field the model cannot fill honestly, a prompt ambiguity — and fixing that single cause moves the aggregate more than any amount of general tuning.

Model calls are the slowest and most expensive thing in a typical request path, and both properties are largely under your control once you understand what drives them.

Latency decomposes into retrieval time, prompt assembly, time to first token, and generation time proportional to output length. Output length dominates. A prompt that produces a thorough explanation when a one-line answer would do costs more and feels slower on every single call, forever. Instructing for brevity and capping maximum tokens are unglamorous changes with immediate effect.

Perceived latency is a different quantity from actual latency, and users respond to the former. Streaming the response transforms the experience without making anything faster. So does showing intermediate state — what is being searched, what was found — during a multi-step operation. If the total is long enough that no amount of streaming helps, the work belongs in a background job with a status endpoint rather than in a request that will time out at a proxy you do not control.

Caching, properly

Three distinct mechanisms get called caching and they behave differently.

Exact-match response caching is trivial and effective for repeated identical inputs, which are more common than you would think in agent loops and internal tools. Semantic caching, serving a stored answer for a similar-but-not-identical query, is tempting and hazardous — the failure mode is confidently returning an answer to a question the user did not ask, and the similarity threshold that avoids that is usually strict enough to eliminate most of the benefit. Provider-side prompt caching rewards a stable prefix, which means system instructions and reference material go at the front and anything volatile goes at the end. Reordering a prompt for cache friendliness is one of the highest-return changes available in a high-volume system.

Routing is the other large lever. Send routine inputs to a small fast model and difficult ones to a large one, with a classifier, a confidence signal or a verifier deciding. Many workloads are dominated by easy cases, and the saving compounds.

Reliability

Treat the model provider as an unreliable network dependency, because it is one. Set aggressive timeouts, retry with jitter on transient errors only, and use idempotency so a retry cannot double-charge or double-post. Configure a fallback model and test the path, because an untested fallback is a decoration. Design degradation that is useful rather than blank: returning the retrieved passages without a generated summary is a far better outcome than an error page, and users accept it.

Rate limits deserve architectural attention rather than a try-except. Shared quota across features means one batch job can starve the interactive path, so separate the quotas, queue the batch work, and shed load deliberately instead of letting it fail randomly.

LLM features introduce vulnerability classes that traditional application security did not have to handle, and OWASP now publishes a dedicated Top 10 for LLM applications precisely because the existing lists did not cover them.

The root issue is that a model has no separation between instructions and data. Everything arrives as one stream of text. If any part of that stream comes from somewhere untrusted — a user message, a retrieved document, a web page, an email body, a tool result — then whoever controls that source can attempt to redirect the model. This is prompt injection, and it is not a bug with a patch. It is a structural property that has to be mitigated architecturally.

The mitigations that actually help are unglamorous and layered. Give tools the narrowest possible permissions, scoped to the current user rather than to the application. Assume any text the model reads may be adversarial and never grant it capability that a compromised instruction could abuse. Keep secrets out of the context entirely; a key the model never sees cannot be leaked by it. Require human confirmation for consequential and irreversible actions. And treat the output of the model as untrusted input to whatever consumes it next.

That final point causes more real incidents than the injection itself. Model output rendered as HTML enables script injection. Model output interpolated into SQL enables the oldest vulnerability in the book. Model output passed to a shell, an eval, or a file path is the same category. A particularly quiet variant: rendered markdown containing an image URL with data appended to it exfiltrates conversation content the moment the client fetches the image. Encode on output, allowlist link and image destinations, and apply the same escaping rules you would to any user-submitted content — because functionally that is what it is.

Data governance and content safety

Decide deliberately what leaves your boundary. Which fields go to a third-party provider, what that provider retains, whether personal data is redacted before the call, and whether your own tracing captures things it should not. Traces are extremely useful and are a common place for secrets and personal data to accumulate unnoticed, so redaction and retention policy belong in the logging layer from the start.

Content filtering on input and output has a real cost in latency and false positives. A guard model that blocks legitimate requests is a product defect visible to users, so calibrate against real traffic and make the threshold a tunable rather than a constant buried in code. And measure the guardrails themselves: an unmeasured filter is an assumption, and assumptions in a security control are how systems fail quietly.

The gap between a working prototype and a maintained feature is mostly operational discipline that has no AI-specific magic to it, applied to a component that behaves unusually.

Prompts are code. They change behavior, they cause incidents, and they need version control, review, and a rollout path with the same seriousness as a database migration. Storing them in a config service is fine; storing them in a service that lets anyone edit production behavior without review is not. Whichever you choose, the prompt version must be recorded on every request so a complaint from last week can be tied to what was actually running.

Model versions move underneath you. Providers update models, sometimes with notice and sometimes with a pointer alias that silently changes what you are calling. Pin explicit versions where the provider allows it, and re-run the evaluation suite whenever anything changes — the model, the prompt, the retrieval configuration, the chunking, or the schema. This is the payoff for having built evaluation early, and it is what makes a model upgrade a routine change rather than a leap of faith.

Tracing is not optional

Capture the whole request: user input, retrieved context with document identifiers, the assembled prompt, model and sampling parameters, the raw output, validation results, latency broken out by stage, token counts, cost, and any downstream feedback. Without this, a report that "the assistant gave a wrong answer yesterday" is undebuggable, because you cannot reconstruct what it saw. With it, you usually find the cause in minutes and it is usually retrieval.

Traces also enable the technique that makes changes safe: replay. Take a set of captured production requests, run them against a candidate configuration offline, and compare. Shadow the new configuration against live traffic without serving its output. Canary to a small share of users with the product metrics watched rather than only the error rate. None of this is novel engineering practice — it is the standard playbook, applied to a component where a regression does not throw an exception and therefore will not page anyone.

Cost and the flywheel

Meter spend per feature, per tenant and per user, and alert on the derivative rather than the total. Runaway agent loops, a retry path that never terminates, and a prompt change that quietly tripled output length all show up as a slope before they show up as an invoice.

Then close the loop. Every production failure becomes an evaluation case. Every evaluation case that starts passing stays in the suite. Over time the suite encodes everything the system has ever got wrong, and the feature becomes progressively harder to break — which is the only durable form of quality in a system built on a component you do not control.