AI Prompt Engineer Interview Questions: What the System Design Round Actually Tests

Zero-shot gives the model an instruction and nothing else, no examples, relying entirely on what it learned during pretraining and fine-tuning. Few-shot adds two to five worked examples in the prompt itself, anchoring the model to a specific input-output pattern before it sees the real query. Chain-of-thought asks the model to reason step by step before giving a final answer, which surfaces intermediate logic you can inspect, and often improves accuracy on multi-step problems, at the cost of more output tokens and slower responses.

Most chat-tuned models are trained to treat system-role messages as higher-priority instructions than user-role messages, roughly the way an admin config outranks a runtime request, though the exact weighting is model-specific and not something labs publish in detail. In practice, the system prompt sets persistent behavior, tone, format, constraints, refusal rules, that should hold across an entire conversation, while the user prompt carries the actual query that changes turn to turn. The split matters operationally too: the system prompt is usually where you'd put anything you don't want a user to override through clever phrasing in their own message.

At each step, the model takes the sequence of tokens generated so far and outputs a probability distribution over every token in its vocabulary, then samples one token from that distribution according to the decoding strategy in use, greedy, top-k, top-p, or some mix. It repeats this one token at a time, feeding its own output back in as new input, until it produces a stop token or hits a length limit. Everything the model appears to "know" is encoded in the weights that produce that distribution, there's no separate lookup or reasoning module running underneath, which is exactly why chain-of-thought helps: it gives the model more of its own generated tokens to condition on before it has to commit to a final answer.

Temperature rescales the probability distribution before sampling, low temperature sharpens it toward the highest-probability tokens, high temperature flattens it out and gives lower-probability tokens more of a chance. Top-p, nucleus sampling, instead caps the sampling pool to the smallest set of tokens whose cumulative probability crosses a threshold, so it adapts to how confident the distribution is at each step rather than applying a fixed reshaping. For factual extraction, temperature near zero and a tight top-p, you want the model's best guess, not a creative one. For marketing copy or brainstorming, a higher temperature (0.7 to 1.0) and a looser top-p actually help, since some of the value is in getting outputs you wouldn't have written yourself.

Public benchmarks test broad, general knowledge and reasoning across a wide spread of topics, which correlates loosely with real capability but says very little about your specific task's distribution of inputs, edge cases, and failure modes. A model can score well on MMLU and still fail consistently on your domain's jargon, your specific output format, or the particular way your users phrase requests. Benchmark contamination is a real, ongoing concern too, some portion of public benchmark data likely leaked into pretraining sets, which inflates scores in ways that don't transfer to a genuinely novel task.

The model is fine-tuned or prompted to recognize when a task needs an external capability it doesn't have, real-time data, a calculation, a database lookup, and, instead of generating a prose answer, output a structured call, a function name and a set of arguments matching a schema you provided. Your application code parses that structured output, actually executes the function, and feeds the result back into the model's context as a new message so it can generate a final answer grounded in the real result. The model never runs any code itself, it only ever decides what to call and interprets what comes back.

That you can't. Current-generation models, LLM-as-judge, RAG, and every mitigation covered on this page reduce hallucination rate, none of them reduce it to zero. The candidates who lose points here aren't the ones who admit that, they're the ones who confidently describe a "solution" that an interviewer with real production experience immediately recognizes doesn't hold up past a demo. Saying "here's how I'd reduce it, and here's how I'd measure whether the reduction is actually working" is a stronger answer than any claim of elimination.

A base model is trained purely to predict the next token from raw text, books, web pages, code. If you hand it "The capital of France is," it will likely finish the sentence correctly, but if you hand it a question like "What's the capital of France?" it might just continue with more questions, because that's a statistically plausible continuation of text that looks like a quiz. It has no notion that it's supposed to answer you specifically.

An instruct-tuned or chat model goes through an extra stage, usually supervised fine-tuning on instruction-response pairs followed by RLHF or a similar preference-based step, so it learns the pattern "when given an instruction, produce a helpful response to that instruction" rather than just "continue this text." That's why chat models respond well to direct commands like "summarize this" or "write a function that does X," while a raw base model needs to be prompted more like a text-completion puzzle, often with few-shot examples showing the exact pattern you want continued. If you're prompting a base model in production, which still happens for cost or licensing reasons, you generally need more few-shot scaffolding and fewer bare instructions.

The context window is the maximum number of tokens, input and output combined, that a model can attend to in a single call. It's not a soft limit, it's architectural. Every token in the window gets a position and takes part in the attention computation, and the model simply has no representation for anything outside that window.

What happens when you go over it depends on the API. Most providers will just reject the call with an error before generation even starts, telling you the token count exceeded the limit. Some frameworks silently truncate the oldest messages to make room, which is worse in practice because you don't get an error, you just get a model that's missing earlier context and starts contradicting things the user said three turns ago. In a real chat product you have to manage this yourself: track token counts as the conversation grows, and once you're near the ceiling, either summarize older turns into a compact system note, drop the least relevant history, or move older context into a retrieval step instead of keeping it in the raw prompt.

Delimiters solve a boundary problem. When you paste user text, a document, or retrieved content directly into a prompt without marking where it starts and ends, the model has to guess which parts are your instructions and which parts are data to operate on. That ambiguity is exactly where prompt injection and confused instruction-following come from, if a pasted document happens to contain a sentence like "ignore the above and do this instead," an unmarked prompt makes it much easier for the model to treat that as a real instruction.

Wrapping content in something explicit, triple quotes, XML-style tags like <document>...</document>, or markdown headers, gives the model a clean signal: everything inside this boundary is content to read or transform, not commands to obey. It also just improves parsing reliability. If you ask a model to "summarize the text between the triple backticks," it's far less likely to accidentally summarize your instructions along with the text. In practice, XML-style tags tend to work slightly better than quotes for longer or nested content because you can label multiple sections distinctly, like <context>, <question>, and <examples>, and the model has clearly seen that pattern heavily in training data from documentation and code.

It's a mixed bag, and the honest answer is it depends on what you're actually testing. Persona framing can shift tone and register in a genuinely useful way, "act as a terse senior engineer doing code review" reliably produces shorter, more direct feedback than a neutral prompt, because the model is pattern-matching to what that kind of writing looks like in its training data. It's real style conditioning, not nothing.

Where it falls apart is the claim that saying "act as a world-class expert" makes the model more factually accurate or better at reasoning. Several independent tests have shown persona framing doesn't reliably improve correctness on math, logic, or factual QA tasks, and can occasionally make things worse if the persona pulls the model toward a more confident, less hedged tone on things it's actually unsure about. The practical takeaway is to use persona prompting for tone, format, and audience calibration, and use other techniques, few-shot examples, explicit reasoning steps, retrieval grounding, when what you actually need is correctness.

Negative instructions ask the model to suppress a concept, and suppression is a much harder generation problem than it sounds. To correctly follow "don't mention pricing," the model still has to represent the concept of pricing internally to know what to avoid, and in a long or complex response it can lose track of that constraint the same way a person might forget a rule they were told at the start of a long conversation. The failure gets worse when the instruction competes with something else in the prompt, like a user directly asking "how much does this cost," where the pull toward answering the question can override the suppression rule.

The more reliable fix is to reframe the constraint as a positive instruction plus a fallback: instead of "don't mention pricing," say "if the user asks about pricing, respond with 'let me connect you with our sales team' and don't provide numbers." That gives the model a concrete action to take rather than an abstract thing to avoid. For anything safety- or compliance-critical, don't rely on prompt wording alone, add a post-generation check, a regex or classifier that scans the output for banned terms before it reaches the user, so the constraint is enforced by code, not just by hoping the model complies.

max_tokens is a hard ceiling on how many tokens the model is allowed to generate in that call, regardless of content. Once it hits that count, generation stops immediately, even mid-sentence, mid-word, or mid-JSON-object. It's a safety valve against runaway or unexpectedly long output, and it directly controls your cost per call since you're billed per token.

A stop sequence is different, it's a specific string that, if the model generates it, tells the API to stop before including it in the output. You'd use one when you know the exact boundary of what you want, for example stopping generation at "nnHuman:" in a raw completion-style chat loop, or at a closing delimiter like "</answer>" when you've asked the model to wrap its response in tags. You need both together because stop sequences only help if the model actually produces the stop string, and models sometimes don't, due to a bug in your prompt, an edge case, or the model deciding to keep going past where you expected. max_tokens is the backstop that guarantees the call terminates and your cost stays bounded even if the stop sequence never fires.

An embedding is a list of numbers, a vector, usually a few hundred to a few thousand dimensions, that represents a piece of text in a way that captures its meaning rather than its exact wording. Two sentences that mean roughly the same thing, "the store closes at 9pm" and "closing time is 9 in the evening," end up as vectors that sit close together in that space, even though they don't share many words. That's the whole trick behind semantic search, you're comparing meaning, not keywords.

A prompt engineer needs this because it's the mechanism underneath every RAG pipeline you'll build. When a user asks a question, you embed that question, compare it against embeddings of your document chunks using cosine similarity or a similar distance metric, and pull back the closest matches to stuff into the prompt as context. If retrieval is returning irrelevant chunks, the bug is often not in your prompt at all, it's in the embedding model being a poor fit for your domain, your chunks being too long or too short to embed a coherent single idea, or a mismatch between how you embedded the documents versus how you're embedding the query. Understanding that the retrieval step and the prompt-writing step are two separate failure surfaces is what separates someone who can debug a broken RAG system from someone who just keeps rewriting the prompt and wondering why nothing improves.

Few-shot examples work well when the task has a clear, repeatable pattern, classifying support tickets into five categories, say, and the risk is that the model anchors too hard to your examples and gets brittle on edge cases that don't resemble any of them. Chain-of-thought earns its keep on tasks with real multi-step logic: math, multi-hop questions, anything where you need to see where reasoning went wrong to fix the prompt. It costs more tokens and more latency, and on tasks that don't actually require multiple reasoning steps, it can introduce more room for the model to talk itself into a wrong answer instead of less.

The part interviewers actually care about: you need a way to evaluate which one wins for your specific task, not a general opinion. That means a test set, a scoring function, and a repeatable eval pipeline, not five examples you tried by hand and liked the look of.

Pick examples that cover the edges of your input distribution, not just the common case. If you're classifying support tickets, include one ambiguous ticket that could plausibly fit two categories, not five clean examples that all look alike. Rotate or randomize example order between calls where the model supports it, since some models show a mild recency bias toward the last example in the list. And test the prompt against inputs that deliberately don't resemble any of your examples, if accuracy craters on those, the model memorized your examples' surface pattern instead of the underlying task.

Run the same chain-of-thought prompt multiple times at a non-zero temperature, then take the majority answer across the runs instead of trusting a single generation. It works because wrong reasoning paths tend to diverge from each other while correct ones tend to converge on the same answer, so voting filters out a chunk of the noise. It's worth paying for N generations instead of one on high-stakes, low-volume tasks, a compliance check that runs a few hundred times a day, not a chat feature serving ten thousand requests a minute, where the extra latency and cost are hard to justify.

They're related but not identical. A jailbreak targets the model's own safety training, tricking it into ignoring its guidelines through role-play framing, hypothetical scenarios, or encoding tricks. A prompt injection targets your application's instructions specifically, smuggling new commands into content the model reads as part of its context, a document, a webpage, a user message, so the model does something the developer never intended. Some defenses overlap, output filtering catches symptoms of both, but a jailbreak-resistant base model won't stop an injection hidden inside a retrieved PDF, and input sanitization won't stop a jailbreak framed as an innocent creative-writing request.

Constrained decoding, grammar-based sampling that only allows tokens matching a schema, is the most reliable option where the model provider supports it, since it makes invalid JSON structurally impossible rather than just unlikely. Where that's not available, a strict schema in the prompt plus a validation step on the output, retrying with the validation error fed back into a follow-up call, gets you most of the way there. Budget for the retry path in your latency numbers, not just the happy path, because it fires more often than a demo led you to believe.

import json
from jsonschema import validate, ValidationError

def get_structured_output(prompt, schema, model_call, max_retries=2):
  for attempt in range(max_retries + 1):
    raw = model_call(prompt)
    try:
      parsed = json.loads(raw)
      validate(parsed, schema)
      return parsed
    except (json.JSONDecodeError, ValidationError) as e:
      prompt = f"{prompt}nnYour last output failed validation: {e}. Return valid JSON matching the schema only."
  raise ValueError("Model failed to produce valid output after retries")

Prompt compression shrinks a long prompt, a big retrieved context, a long conversation history, down to fewer tokens while trying to preserve the information that matters, either through learned compression models, extractive summarization of less-relevant chunks, or simply dropping older conversation turns past a certain age. You need it when cost or latency scales with input tokens faster than your budget allows, long-running agent conversations and RAG pipelines pulling in large documents are the usual suspects. You don't need it for most single-turn tasks with a modest context window, where the engineering cost of compression outweighs the savings.

Tokenizers split text into subword units, not characters or whole words, so the same string can cost a very different number of tokens depending on language, formatting, and vocabulary rarity. A rare technical term or a name might get split into three or four tokens where a common word costs one, which matters for cost estimates on anything user-generated. It also explains a specific, recurring gotcha: models are worse at character-level tasks, counting letters in a word, reversing a string, than their fluency would suggest, because the model never sees individual characters, it sees tokens that already group several characters together.

Temperature zero should, in theory, always pick the single highest-probability token, greedy decoding, giving deterministic output. In practice, floating-point non-associativity in how GPU kernels batch and parallelize computation means the exact probability values can shift by tiny amounts depending on what else is running on the same hardware at the same time, and a tie or near-tie between two tokens can flip which one wins. Some providers also don't guarantee true greedy decoding at "temperature zero", they may apply a very low but nonzero temperature internally. If a system needs strict reproducibility, seed parameters help where the API exposes one, but I wouldn't promise anyone that a hosted LLM API is fully deterministic. I've been surprised by this more than once.

RAG changes what the model knows by retrieving relevant information at query time and injecting it into the context; the model's weights never change. Fine-tuning changes how the model behaves by updating its weights on task-specific examples. Reach for RAG when the knowledge needs to stay current or auditable, you can point to the exact document a fact came from, and updating it means re-indexing, not retraining. Reach for fine-tuning when you're changing behavior rather than knowledge: a specific output format, a tone, a domain-specific way of reasoning that few-shot examples in the prompt can't reliably instill.

Fine-tuning is expensive and slow to iterate on, every change means a new training run and a new eval pass. RAG is cheaper to update but brittle at the retrieval layer, garbage in, garbage out, no matter how good the generation step is. Most production systems described in the loops candidates report back to us end up combining both, RAG for factual grounding, a lightweight fine-tune, often LoRA, for task-specific behavior on top. That's a qualitative read, not a controlled study, so take the ratio with a grain of salt.

LoRA, low-rank adaptation, freezes the base model's weights and trains a much smaller pair of low-rank matrices that get added into specific layers, usually the attention projections, instead of updating every parameter in the model. The result behaves close to a fully fine-tuned model on the target task at a fraction of the training compute and, importantly for production, a fraction of the storage: you can keep dozens of small adapters around for different tasks and swap them at inference time without hosting dozens of full model copies. The trade-off is capacity, a LoRA adapter with a small rank has less room to shift the model's behavior than full fine-tuning does, so it works best when the target task is a variation on something the base model already does reasonably well.

Fine-tuning on a narrow dataset can degrade performance on tasks the base model used to handle fine, because gradient updates aimed at the new task can overwrite weights that were doing useful work elsewhere. A model fine-tuned hard on customer support tickets might get noticeably worse at general reasoning, or at a slightly different support task it wasn't fine-tuned on. The practical guard is running your full eval suite, not just the fine-tuning task's eval set, before and after a fine-tune, and watching for regressions on tasks you didn't intend to touch. LoRA and lower learning rates both reduce the risk somewhat, since they change less of the model, but neither eliminates it.

Cost per token, latency, and context window matter as much as raw capability, and capability itself should be measured on your specific task, not a general leaderboard. A model that scores well on MMLU can still be the wrong pick for a coding autocomplete feature that needs sub-100ms responses and predictable token-based billing at high volume. Smaller, quantized models frequently beat larger general-purpose ones on narrow, well-defined tasks, classification, extraction, a fixed-format rewrite, once you've measured both against your eval set instead of assuming bigger means better. The interviewer here is usually checking whether you've shipped a real system under a real budget, not whether you can recite a benchmark leaderboard from memory.

Hallucination usually refers to a model generating content that's fluent and confident-sounding but not grounded in any real source, a fabricated citation, a nonexistent API method, a plausible but invented fact. Being wrong more broadly includes hallucination but also covers genuine reasoning errors, outdated training data, or misinterpreting an ambiguous instruction. It matters operationally because the mitigations differ: hallucination is addressed by grounding, RAG, citation requirements, confidence calibration, while a reasoning error might need a different prompting strategy or a bigger model entirely. Lumping both under one "the model was wrong" bucket in your monitoring makes it much harder to tell which fix actually worked.

LLM-as-judge uses a, usually stronger, model to score or compare outputs against a rubric instead of relying purely on human raters, which scales far better than humans reading every output by hand. The classic failure mode is self-preference bias: a judge model tends to rate outputs from its own model family, or outputs written in a style similar to its own, more favorably, independent of actual quality. Position bias is the other common one, in pairwise comparisons, judges show a measurable tendency to favor whichever answer they see first or second, depending on the model. Mitigations include swapping answer order and averaging, using a judge from a different model family than the one being evaluated, and spot-checking judge scores against a small human-labeled sample often enough to catch drift.

Have a meaningful fraction of examples double-scored by two different raters from the start, not as an afterthought, and track agreement with a real statistic, Cohen's kappa, not just raw percent agreement, which overstates agreement when one label is much more common than the others. If agreement is low on a specific category of example, that's a rubric problem, not a rater problem, most of the time, tighten the rubric's language for that category rather than retraining the rater on vibes. I'd be skeptical of any eval pipeline that reports a single aggregate score without ever having measured how consistent its own raters were with each other.

Chunk your documents, embed each chunk into a vector, store the vectors in a vector database, embed the incoming query the same way, retrieve the top-k most similar chunks, inject them into the model's context, then generate an answer grounded in that context. Retrieval is where it breaks. A 2025 arXiv survey of RAG architectures put it plainly: generation quality is tightly coupled to retrieval quality, and when relevant documents aren't retrieved, or irrelevant context gets pulled in alongside them, the generator produces incorrect or misleading output almost regardless of how good the underlying model is (arXiv:2506.00054). Candidates who've only read about RAG tend to focus their prep on the generation step. The ones who've actually run one in production talk about chunk size, embedding model choice, and reranking instead.

Too small and you lose surrounding context, a chunk might contain a pronoun with no antecedent, or a number with no unit, because the sentence that explained it got split into a different chunk. Too large and you dilute the embedding, a chunk covering five unrelated subtopics produces a vector that's a blurry average of all five, and it becomes harder for a query about one specific subtopic to retrieve it with high similarity. There's no universal right answer, it depends on your document structure and query patterns, but 200 to 500 tokens with some overlap between adjacent chunks is a reasonable starting point for prose-heavy documents, and you tune from there against real retrieval failures, not a rule of thumb.

Vector similarity search is fast but approximate, it's optimized for retrieving a broad candidate set quickly, not for perfectly ranking that set by relevance. A cross-encoder reranker looks at the query and each candidate chunk together, rather than comparing pre-computed vectors independently, and produces a more accurate relevance score, at the cost of running a second, heavier model over a smaller candidate set. It earns its keep when your top-k retrieval quality is inconsistent, the right chunk shows up somewhere in the top 20 but not reliably in the top 3, since reranking that top 20 down to the top 3 fixes exactly that gap. It's harder to justify on a latency-sensitive feature where the initial retrieval is already good enough that reranking mostly reorders chunks that were fine either way.

First, check whether it's actually a contradiction or just two documents describing different scopes or time periods that look contradictory out of context, a pricing page from 2024 and one from 2026 aren't wrong, they're both right for their respective dates. If it's a genuine contradiction, prefer the more recent or more authoritative source using metadata, a timestamp, a document tier ranking, rather than leaving the model to guess which one to trust, since models don't reliably resolve contradictions on their own and tend to blend both into an answer that's confidently wrong. The more durable fix is upstream: flag contradicting documents during ingestion so a human resolves the source-of-truth question before it ever reaches a live query.

BM25 is classic keyword-based search, it's very good at exact matches, product SKUs, error codes, specific proper nouns, the kind of query where semantic similarity actually works against you because a vector search might rank a topically similar chunk above the one with the exact code you searched for. Vector similarity is good at semantic matches, a query phrased completely differently from the source text but meaning the same thing. Hybrid search runs both and merges the results, often with a reciprocal rank fusion step, because production queries are a mix of both patterns and picking only one leaves a predictable category of query failing consistently.

A RAG pipeline follows a fixed sequence, retrieve, then generate, every time, regardless of the query. An agent decides its own sequence of steps at runtime, it might retrieve, then decide it needs to call a tool based on what it retrieved, then retrieve again with a refined query based on the tool's result, looping until it decides it has enough to answer. The control flow lives inside the model's own reasoning rather than in a fixed pipeline your code defines up front. That flexibility is also exactly why agents fail in less predictable ways than RAG pipelines do, a fixed pipeline fails the same way every time it fails, an agent can wander into a failure mode you never anticipated because it chose a sequence of steps you didn't design.

Match model size to the reasoning load at each step, not the pipeline as a whole. The top-level planning step, deciding what sequence of tools to call and why, usually benefits from a stronger model since a bad plan compounds into every step underneath it. Individual tool-calling steps, deciding which specific function to invoke given a clear sub-goal, are often simple enough that a smaller, faster, cheaper model handles them just as reliably, and running ten of those steps on an expensive model when a cheap one would do is a common way agent pipelines quietly blow their cost budget. Measuring where accuracy actually degrades on cheaper models, rather than assuming, is worth the afternoon it takes to run the comparison.

Response latency, and its tail, p95 and p99 matter more than the average for anything user-facing, confidence distribution over time, a shift toward lower-confidence outputs often precedes a quality problem before users start complaining, safety filter trigger rate, task completion rate, and cost per query. The hard part isn't picking these metrics, it's that automated output quality evaluation is often impossible for open-ended tasks, so you also need a human review pipeline and scheduled eval-set runs feeding into the same dashboard, not just the metrics you can compute automatically. Alert thresholds need real thought too, set them too sensitive and they fire constantly on normal variance until everyone ignores them, which is functionally the same as having no alerts at all.

This is fundamentally a systems design question wearing an LLM costume. The strongest answers blend LLM-specific signals with standard observability practice rather than treating it as an entirely new discipline.

Traffic splitting is the obvious part, everyone gets that right. The harder parts are defining a success metric that actually reflects quality, task completion, not just engagement, since engagement can go up on a worse feature simply because it's different, picking a test duration long enough to wash out the initial spike of user curiosity that any change gets regardless of whether it's actually better, and having a rollback plan ready before you launch, not improvised after the canary slice starts regressing. Novelty effects specifically mean your first few days of data almost always look better than the change deserves, so a test that only runs for three days is measuring curiosity more than quality.

Grounding through RAG so answers are tied to actual source documents rather than the model's parametric memory, calibrated confidence signals so the system knows when to hedge, output validation that checks generated claims against the source material before showing them to a user, human escalation for anything below a confidence threshold, and failure logging with a real audit trail so you can go back and see what the model actually said when a complaint comes in. Stack Overflow's Developer Survey has tracked developer sentiment toward AI tools for a few years running now, and answers that sound right but are close, not quite right, consistently shows up as one of the most common frustrations developers report, which tracks with what shows up in support-chatbot postmortems too.

The answer an interviewer actually wants to hear is that you can't fully eliminate hallucinations, and credibility comes from demonstrating you understand that, not from claiming you've solved it.

Assign someone, ideally not the person who wrote the prompt, fresh eyes catch different things, to actively try to break it: adversarial phrasing, edge-case inputs, attempts to extract the system prompt itself, requests dressed up as hypotheticals or role-play to bypass refusal behavior. Keep a running library of these attempts and their outcomes so the next prompt in the same product doesn't have to rediscover the same failures from scratch. Automated red-teaming tools exist and help cover more ground faster, but I'd treat them as a supplement to a human doing this deliberately, not a replacement, since the creative, lateral-thinking attacks that actually find new failure modes still tend to come from a person, not a script running a fixed attack library.

Route a small percentage of real traffic, often single digits, to the new prompt or model version while the rest keeps running the current one, and compare metrics between the two slices before deciding to roll out further. Rollback triggers should be specific and pre-agreed before launch, not decided in the moment: a measurable drop in task completion rate, a spike in safety filter triggers, a latency regression past an agreed threshold. Waiting for someone to notice something feels off is too slow and too subjective, by the time it's obvious without a metric, it's usually been live for longer than it should have been.

A kill switch that can disable the feature or fall back to a safe default response faster than a full deploy cycle, since waiting for a normal release process during an active incident is how a small problem becomes a big one. After the immediate mitigation, the actual debugging looks a lot like any other production incident: what changed recently, a model version, a prompt edit, an upstream data source feeding RAG, and can you reproduce it against a small, isolated set of inputs before touching anything live. The instinct to immediately rewrite the prompt at 2am under pressure is usually the wrong one, that's how you trade one failure mode for a different, less-tested one.

Cap context length aggressively where the task allows it, truncating or summarizing conversation history past a certain point instead of letting it grow unbounded turn after turn. Route by task complexity, a cheap, fast model for simple classification or routing decisions, a more expensive model reserved for the subset of requests that actually need it, rather than sending every request through your most capable, and most expensive, model by default. Cache aggressively for repeated or near-duplicate queries where an exact-match or embedding-similarity cache hit saves a full generation. None of this is exotic, it's mostly applying normal cost-engineering discipline to a system where the per-request cost is a lot less predictable than a typical API call.

Temperature goes near zero first, you want the least amount of sampling variance possible when the cost of a wrong answer is high. Output format gets locked to a schema, JSON with explicit field types, validated on the way out, not left to prose you then have to parse with regex. Build in explicit refusal cases: the prompt should tell the model exactly what to do when it's not confident, return a null field or a flagged status, not force a guess into a required field. And add a second-pass validation step that checks the extracted output against the source document, since asking a model to grade its own confident-sounding first answer without a second look catches surprisingly little.

The prompt you'd write for a demo of this and the prompt you'd actually ship are different documents. The demo prompt optimizes for looking good on three examples. The production prompt optimizes for what happens on the four hundredth edge case nobody tested.

You don't, fully, and saying so out loud is a better answer than pretending you've solved it. What you can do is treat prompts like versioned artifacts, not strings baked into application code: a prompt registry that tracks which prompt version paired with which model version produced which eval score. Every model update triggers a regression run against your full eval set before the update rolls out to production traffic, not after. Canary deployments, routing a small slice of real traffic to the new model version and comparing its outputs against the old one before a full rollout, catch regressions a static eval set sometimes misses, because real traffic has a longer tail of odd inputs than any eval set built ahead of time.

No complete solution exists, and an interviewer who hears you claim one should immediately distrust the rest of your answer. What works is defense in depth: sanitizing and flagging suspicious patterns in user input before it reaches the model, isolating untrusted content, user-submitted text, retrieved documents, from trusted instructions using clear delimiters or structural separation so the model has a better shot at telling them apart, and filtering output before it reaches the user or triggers a downstream action. Constitutional AI-style training in the base model reduces the attack surface further but doesn't close it. Most interviewers are listening for at least two of these layers named specifically, not a single "we sanitize inputs" answer.

Research on long-context models has repeatedly found that models are noticeably better at using information placed at the very start or very end of a long context than information buried in the middle, even when the stated context window is large enough to hold everything. The practical fix isn't a clever prompt trick, it's structural: put the most important instructions and the most relevant retrieved passages near the start or end of the prompt, not the middle, and don't assume a bigger context window means every position in it gets used equally. I'd treat "we have a 200K token context window so we just stuff everything in" as a yellow flag in an interview answer, not a green one.

This is the question most candidates skip preparing for, and it's the one that separates people who've dealt with AI product quality at scale from people who've only read about it. Start with a consistent rubric, specific, checkable criteria a judge, human or model-based, can apply the same way every time, rather than an open "rate this 1 to 5" prompt that different judges interpret differently. Run inter-rater reliability checks early: have two judges score the same sample and measure agreement, because a rubric that different graders interpret differently is worse than no rubric, it gives you false confidence in a number that isn't measuring anything consistent.

Build a "bad output" library from real failures as you find them, not hypothetical ones you imagine in advance, and use it to stress-test the rubric itself. Revisit the eval periodically too, because the definition of a good output drifts as the task and your users' expectations drift, an eval frozen from six months ago is quietly measuring the wrong thing without telling you.

Static indexing, embed everything once and never touch it, works for a demo and fails quietly in production, which is exactly the gap that tripped up the mock interview candidate from the start of this page. You need incremental indexing that handles both additions and deletions, since a document that's been removed or superseded but never gets pulled from the vector store will keep getting retrieved as if it's still current. Metadata timestamps on chunks let you filter or downweight stale content at retrieval time, and a periodic full reindex catches embedding drift and structural changes a purely incremental pipeline misses over time.

How aggressively you need to solve this depends entirely on the domain. A knowledge base of internal engineering docs that changes weekly needs a tighter loop than a knowledge base of historical product documentation that rarely changes at all. I don't think there's a single right refresh cadence, it's a cost-of-staleness calculation specific to what a wrong answer actually costs you.

Getting stuck in a loop, calling the same tool repeatedly with slightly different arguments because it's not making progress toward the goal but also hasn't recognized that it isn't. A close second: compounding errors, an early step in the chain produces a subtly wrong result, and every subsequent step builds on that wrong result with high confidence because nothing in the loop is checking the earlier work against ground truth. Both point to the same fix: a step budget, a hard cap on iterations before forcing a stop or human handoff, and a way to detect when the agent's own confidence isn't backed by actual progress, not just trusting the model to know when to quit on its own.

Store memory with metadata, when it was created, what it was derived from, how confident the source was, rather than as flat, undated facts that all look equally trustworthy to a later retrieval. Decay or expire memories that haven't been reconfirmed in a while, especially anything describing a fact likely to change, a user's stated preference, a project's current status, and weight retrieval toward recency for that category specifically. I don't think there's a clean, generally agreed-on solution to this yet, most production agent memory systems I've seen described are closer to a pragmatic patchwork of timestamps and heuristics than a principled architecture, and I'd be honest about that in an interview rather than pretending there's a solved playbook.

Detect and redact PII before it reaches the model where the task doesn't actually need it, a support ticket classifier probably doesn't need someone's actual card number in the prompt to classify the ticket correctly. Where the task genuinely requires PII, identity verification, account-specific support, make sure it's not logged in plaintext anywhere downstream, prompts, model outputs, and eval samples all need the same handling as any other system touching sensitive data, not a lighter standard just because an LLM is involved. This is also a place where "the model provider says they don't train on our data" isn't the whole answer, your own logging, eval pipeline, and third-party monitoring tools all need the same scrutiny.