OpenAI Interview Questions (2026): What They Actually Ask

"OpenAI is at the frontier of AI research" is not an answer. OpenAI lists alignment with its mission and values as one of the four things it looks for, and its guide sends candidates to the Charter and recent blog posts before the first call. That tells you what a good answer is built from: a specific position on how OpenAI goes about building powerful AI, and why that approach is the one you want to work inside.

Name something concrete. It might be a recent release on the team you applied to, a research post you disagreed with, or a trade-off you think the company handled well or badly. Then explain how it differs from what Anthropic, Google DeepMind or Meta are doing, in your own words. You don't have to agree with every OpenAI decision. You do have to show you've thought about them. Specific beats enthusiastic every time.

The distinction the question draws: ownership means you tracked whether the thing you shipped was working, not just whether your code was merged. Strong answers describe someone who watched metrics after launch, caught a problem nobody had assigned to them, and either fixed it or escalated it. The question is about behavior after the code reaches production, not during the build.

Expect a follow-up on the outcome and what you'd do differently. "It all went well" is a weak answer. The most convincing versions have you catching something significant and acting on it even when it wasn't strictly your job.

OpenAI doesn't publish an end-to-end figure, but it does publish the gap at each step. Résumé review typically takes one week. After the introductory call you hear within a week whether you move to the skills-based assessment, and after the assessment you hear within a week whether you advance to final interviews. The finals themselves run over one or two days, and OpenAI says to expect a decision within one week of them.

Add those up and a smooth process takes a few weeks plus however long scheduling takes on both sides. Two things can stretch it: roles that ask for more than one assessment, and reference checks, which the recruiter may request at the decision stage.

Mostly not. Hello Interview's November 2025 roundup describes OpenAI as having "made a deliberate shift away from standard LeetCode-style questions" toward production-oriented problems: a key-value store with serialization, a resumable iterator, an in-memory database, a spreadsheet with cell dependencies, a multithreaded web crawler. Candidates it quotes say they spent the whole hour typing. OpenAI's own guide points the same way: for engineering interviews it looks for well-designed solutions, high-quality code, optimal performance and good test coverage.

You still need standard patterns (binary search, heaps, BFS and DFS, hash maps, cycle detection) because they sit inside the solutions. What changes is the scoring. Hello Interview also reports candidates hearing that OpenAI won't pass a weak coding score even if everything else goes well, so treat the coding bar as fixed.

The skills-based assessment is the stage between the introductory call and the final interviews. OpenAI says formats vary by team and may include pair coding interviews, take-home projects and technical tests, and that some roles ask for more than one assessment. So a take-home is possible but not guaranteed, and the recruiting team tells you which format you'll get and sends prep material for it.

If you do get a take-home, write it the way the guide says engineering work is judged: a clear design, readable code, sensible performance and real tests. A smaller feature set that is tested and easy to follow is a safer bet than a larger one that isn't. Be ready to explain what you left out and why.

It depends on the interview, and OpenAI says so directly. Its guide states that some formats intentionally allow AI and other tools, while others are designed to assess your independent problem-solving without them. The rules for your specific interview are in your preparation materials, and if you're unsure the guide asks you to check with your recruiter before the interview.

Two practical consequences. First, read the prep materials for each round separately, because the rule can differ between rounds in the same process. Second, practise both modes: problems solved on your own, and problems where an assistant is allowed and you're judged on how well you direct it, review its output and test the result.

Treat it as scored. OpenAI's guide names four things it cares about in candidates: collaboration, effective communication, openness to feedback, and alignment with its mission and values. It doesn't publish how those are weighted against technical results, so nobody outside the company can tell you the exact weight. What you can see is that mission alignment sits on that short list, and that the introductory call already asks about your motivations and goals.

Rehearsed value statements are easy to spot. Read the Charter, pick one or two points you actually have a view on, and be ready to discuss where you'd draw lines on safety, deployment and speed, including the places you're unsure.

It depends on the role. OpenAI says final interviews focus on your area of expertise, so an applied or infrastructure engineer is mainly tested on engineering, while research engineering and ML roles go deep on models and training. The job posting usually makes the track clear; if not, ask the recruiter.

Some ML literacy helps everywhere, because the products you'd work on are built around models. The guide's recommended technical reading is the Deep Learning Book and Spinning Up in Deep RL. For an applied role, knowing tokens, sampling parameters, context limits and evaluation basics (the next two questions) is a sensible floor.

A token is the unit the model actually works with, not a word. Tokenization breaks text into subword pieces using a byte-pair-encoding vocabulary; OpenAI's open-source tiktoken library implements the encodings, such as cl100k_base and o200k_base, depending on model generation. As a rough rule, one token is about four characters or three quarters of a word in English. Email addresses, URLs and unusual technical terms split into more tokens because they don't map onto common subword chunks.

It matters for two concrete reasons. Cost: input tokens (the prompt) and output tokens (the completion) are billed separately and usually priced differently, so a long system prompt or a large retrieved context adds up before the model writes anything. Context window: every model caps the total tokens it can handle, and if a long conversation plus a large completion runs past it you get truncation or an error. The usual fix is to count tokens with tiktoken before the call and decide in advance what to trim or summarize.

Both control how random the next-token choice is, but they work differently. temperature rescales the probability distribution before sampling: near 0 the model almost always picks the most likely token, so output is repetitive and close to deterministic, while higher values flatten the distribution and let less likely tokens through, which adds variety and raises the odds of drifting off-topic. top_p, or nucleus sampling, samples only from the smallest set of tokens whose cumulative probability reaches p, so it adapts to how confident the model is at each step. On an obvious next word it may consider two candidates, on an ambiguous one fifty.

OpenAI's API reference recommends changing one of the two, not both, and that's a good default to state because changing both makes behavior hard to debug. Low temperature suits extraction and code generation, where you want the most likely answer every time. Higher values suit brainstorming. The common mistake is assuming temperature 0 means byte-for-byte identical output. It gets close, but batching and floating-point effects on the serving side can still produce small differences for the same request.

Store a list of (timestamp, value) pairs per key in a hash map. For get(), binary search that list for the largest timestamp that does not exceed the query. For set(), append; if timestamps arrive in increasing order per key, the list stays sorted with no extra work. If they can arrive out of order, say so and insert with bisect instead.

Listed in Hello Interview's November 2025 roundup of OpenAI coding questions, which frames it as tracking values that change over time, such as model versions or configuration. A linear scan works but misses the point. Good follow-ups to practise: concurrent writers (a lock per key or a concurrent map), memory limits on old versions, and persisting the store to disk.

from collections import defaultdict

class TimeMap:
  def __init__(self):
    self.store = defaultdict(list) # key -> [(timestamp, value)]

  def set(self, key: str, value: str, timestamp: int) -> None:
    self.store[key].append((timestamp, value))

  def get(self, key: str, timestamp: int) -> str:
    pairs = self.store.get(key)
    if not pairs:
      return ""
    # binary search for rightmost timestamp <= query
    lo, hi = 0, len(pairs) - 1
    result = ""
    while lo <= hi:
      mid = (lo + hi) // 2
      if pairs[mid][0] <= timestamp:
        result = pairs[mid][1]
        lo = mid + 1
      else:
        hi = mid - 1
    return result

Length-prefix encoding. For each string, write its length, a separator, then the string itself. On decode, read digits up to the separator, then read exactly that many characters and repeat. Because you never search for the separator inside the payload, the content can contain anything, including the separator.

Hello Interview lists this as the "KV Store Serialize/Deserialize" problem and notes that most candidates land on length-prefix encoding, the same idea the Redis protocol uses. Joining with a rare character such as "#" breaks the moment a value contains it, and that edge case is the first thing to test. The version below encodes a list of strings; for a key-value store, encode keys and values alternately.

def encode(strs: list[str]) -> str:
  return "".join(f"{len(s)}#{s}" for s in strs)

def decode(s: str) -> list[str]:
  result = []
  i = 0
  while i < len(s):
    j = s.index("#", i) # the length prefix never contains '#'
    length = int(s[i:j])
    result.append(s[j + 1 : j + 1 + length])
    i = j + 1 + length
  return result

Wrap the underlying data in a class that stores the current position. next() advances and returns the element, get_state() returns a snapshot of the position, and set_state() restores it. For lazy sources such as generators you can't index directly, so either materialize the data or record how many elements were consumed and replay on restore.

Hello Interview lists it among OpenAI's reported questions and says candidates had to maintain state between calls, handle nested structures and support operations like skip and reset. Ask early whether the state has to survive a process restart (serialize it) or only live in memory, because that changes the design. Nested structures usually mean keeping a stack of positions rather than one index.

Sort meetings by start time and keep a min-heap of end times, one entry per room in use. For each meeting, if the earliest-ending room is free (heap top is at or before the current start), pop it and reuse the room. Push the current meeting's end time either way. The heap size at the end is the answer, in O(n log n).

Hello Interview lists meeting rooms next to a SQL executor as OpenAI problems that test overlap checks and resource allocation. A natural extension is returning which room each meeting gets, which means storing room IDs in the heap alongside end times and keeping a pool of freed IDs.

Embedding drift happens when the embeddings in your index no longer match what the query encoder produces, often because the embedding model changed without a re-index or because the data shifted. Detection: track the similarity between recent queries and their top results over a rolling window, and track a retrieval success signal such as clicks or "this helped" feedback. A sustained drop in either is the alert.

Then find the cause: a model update without re-indexing, a shift in query vocabulary, or a stale corpus. Fixes range from re-indexing everything with the current model (clean, expensive), to a learned adapter that maps old vectors into the new space (fast, approximate), to running old and new indexes side by side and blending results during a migration.

Standard attention writes the full N by N score matrix to GPU high-bandwidth memory (HBM), which costs quadratic memory and a lot of slow reads and writes. FlashAttention computes exact attention in tiles that fit in on-chip SRAM and never writes the full matrix to HBM. The trick is an online softmax: it keeps a running maximum and running normalizer per row, so tiles can be processed one at a time and rescaled as new maxima appear, and the result matches standard attention exactly.

The FlashAttention paper frames it as an IO-aware algorithm and quotes about 19 TB/s of SRAM bandwidth against 1.5 to 2.0 TB/s of HBM on an A100, which is why cutting HBM traffic speeds things up. Extra memory becomes linear in sequence length. If you can explain why the softmax denominator can be accumulated across tiles without seeing every input first, you're ahead of the answer "it avoids materializing the attention matrix."

Mixed precision runs most of the forward and backward pass in FP16 or BF16, which halves activation memory and uses the tensor cores on modern GPUs, while keeping FP32 weights for the optimizer update so small updates aren't lost to rounding. FP16 has a narrow exponent range, so small gradients can underflow to zero and silently stop some parameters from learning. Loss scaling fixes that: multiply the loss before backpropagation so gradients land in FP16's range, then unscale before the optimizer step, skipping steps where the scaled gradients overflowed.

BF16 has the same exponent range as FP32, so it usually doesn't need loss scaling, which is a big reason large-model training moved to BF16 once hardware supported it. Saying why BF16 avoids the problem is what separates a strong answer from a mechanical description. The snippet uses the current torch.amp API; the older torch.cuda.amp spellings are deprecated in the PyTorch AMP docs.

import torch

scaler = torch.amp.GradScaler("cuda")
for batch in dataloader:
  optimizer.zero_grad()
  with torch.amp.autocast("cuda", dtype=torch.float16):
    loss = model(batch)
  scaler.scale(loss).backward()
  scaler.step(optimizer) # skips the step if gradients overflowed
  scaler.update()

Hello Interview's OpenAI list shows this as a senior-to-staff behavioral question, reported by 3 users and last asked in August 2026. It tests judgment, not a memorized framework. A good answer separates capability evaluation (what the model can do, including in misuse), safety evaluation (red-teaming results, known failure modes, how they compare with the model being replaced), readiness to respond (monitoring, rate limits, the ability to roll back or restrict access), and the benefit to users if you ship versus if you wait.

Then say who decides and what would make you stop. Naming a concrete threshold, such as a specific category of dangerous capability that blocks release regardless of commercial pressure, lands better than "it's a balance." Expect the interviewer to push on a scenario where the criteria conflict.

This checks whether you have held a real ethical position in engineering work, rather than whether you can recite values back. Describe the specific decision, the concrete risk to users or others (not "it didn't feel right"), what you said and to whom, and what happened. The outcome doesn't have to be a win. "I raised it, the team shipped anyway, and here's what I saw afterward" is credible.

Two weak versions to avoid: having no example at all, and presenting a technical disagreement as an ethical one. A dispute about a data schema is not this question. A dispute about collecting data users didn't expect you to keep is.

This tests openness to feedback, one of the four traits OpenAI's guide lists. The answer is only as good as the original belief: a strong story has you holding a confident technical or product position, meeting evidence against it, and updating. Weak versions change a minor preference, or have you changing your mind because someone senior said so without engaging with the argument.

Say what the evidence was, why it convinced you, and what you did differently afterward. If the new view later turned out to be partly wrong too, say that. It shows you keep updating.

Hello Interview's OpenAI list includes this for mid-level to senior and manager candidates, reported by 5 users and last asked in March 2026. The trap is a sanitized failure where every choice you made was right and the outcome was bad luck. Interviewers want to see where your judgment was wrong.

Give the real causal chain: the decision you made, why it looked right with what you knew then, the signals you missed, the actual impact, and what you changed afterward. Prepare for "What would you change now?" and "Where have you used that lesson since?" If you can't answer both, the story isn't deep enough yet. Keep it structured; telling a complicated story clearly is part of what's being assessed.

Hello Interview lists this as a senior-level project retrospective at OpenAI (1 report, May 2026): explain the technical challenges, your approach and what you learned. OpenAI's guide also tells candidates to be ready to discuss their work at the introductory call, so have this story ready from day one.

Pick a project where you made the key decisions. Cover the constraint that made it hard, two options you rejected and why, the part that went wrong, and a measurable result. Interviewers usually drill into one component until you reach the edge of what you know. Choose a project you can defend at that depth, and be honest about which parts other people built.

The interviewer is listening for how you scoped the decision, not whether you got lucky. Strong answers name the information you didn't have, what you did to reduce uncertainty in the time available (a quick experiment, a call with someone closer to the problem, a fallback plan), and the risk you knowingly accepted. The weak version describes a decision that, looking back, wasn't risky at all.

This overlaps with the failure question, so prepare a story that is genuinely different rather than the same anecdote told again.

OpenAI ships fast and its guide says final interviews are meant to stretch you, so expect some version of this. A strong answer separates decisions you're comfortable revisiting quickly, which covers most technical choices where being wrong is cheap, from decisions you'd want more certainty on first, like a core data model or a public API that is expensive to change. Naming that split shows you've thought about it instead of saying "I'm adaptable."

If you come from a slower, process-heavy company, say honestly what you'd need to change about how you work. That's more convincing than claiming you already work that way.

This tests whether you can hold a technical position under social pressure without becoming unreasonable. Strong answers describe the specific disagreement, the evidence or reasoning you brought, and how it actually ended, including times you turned out to be wrong and updated. Stories where you were right all along and the other person simply gave in are less convincing than ones with real back-and-forth.

Keep your stories consistent across the loop. If you tell a "changed my mind" story in one round and a "held my ground" story in another, both should show the same habit: you follow the evidence.

Effective communication is on OpenAI's short list of what it looks for, and engineers there work with policy, legal and research partners. The interviewer wants evidence you can turn a technical risk into something a non-engineer can act on, without oversimplifying to the point of being misleading or hiding the risk in jargon.

Name the risk in plain language, what you recommended, and what the stakeholder decided, including cases where they didn't take your advice and what happened next. If your first explanation didn't land and you had to adjust, include it. That's a stronger signal than claiming you got it right the first time.

A hypothetical like this checks your instinct under real stakes rather than a rehearsed story. A good answer goes in order: assess current impact (who is affected right now, and is it still happening), fix or mitigate before investigating root cause, tell whoever owns the relationship with affected users before the postmortem is written, then run a blameless root-cause analysis. Jumping straight to "I'd write a detailed postmortem" skips the live harm.

A hard follow-up to prepare: what if the fix means taking the feature down for users who aren't affected? There's no clean answer. The point is reasoning about the trade-off out loud instead of freezing.

Model each table as a column list plus a list of rows. INSERT appends a row. SELECT with WHERE filters rows by evaluating the condition against each row. For compound conditions (AND/OR), parse the expression into a small tree. Start with a single equality filter and let the interviewer add scope.

Hello Interview lists it as an OpenAI question and describes the work as parsing commands, managing data structures and handling edge cases when operations interact, with "maybe joins" as a possible extension, not query optimization. Scoping is the real test: a full SQL parser up front burns the hour, while a minimal core with clean extension points leaves room for what comes next, such as ORDER BY, an index on one column, or a join.

class InMemoryDB:
  def __init__(self):
    self.tables = {} # table_name -> {"cols": [...], "rows": [[...]]}

  def create_table(self, name: str, cols: list[str]) -> None:
    self.tables[name] = {"cols": cols, "rows": []}

  def insert(self, table: str, values: list) -> None:
    self.tables[table]["rows"].append(values)

  def select(self, table: str, col_filter: str = None, val_filter=None) -> list:
    t = self.tables[table]
    col_idx = t["cols"].index(col_filter) if col_filter else None
    return [row for row in t["rows"]
        if col_idx is None or row[col_idx] == val_filter]

Treat cells as nodes in a dependency graph. getCell evaluates a formula by recursively resolving the cells it references, memoizing results so repeated reads are cheap. Cycle detection is a DFS with a "currently evaluating" set: if you reach a cell that is already in the set, you have a cycle and should return an error instead of recursing forever. You can also run the same DFS inside setCell to reject a cycle before accepting it.

Hello Interview lists it among OpenAI's reported questions and calls it a graph problem dressed up as a spreadsheet. Two bugs to avoid in the version below. Substituting references with plain string replace turns A1 into garbage inside A10, so substitute whole matches with a regex. And clearing the whole cache on every write is correct but slow; the follow-up is to keep a reverse-dependency map and invalidate only the cells downstream of the change.

import re

REF = re.compile(r'[A-Z]+[0-9]+')

class Spreadsheet:
  def __init__(self):
    self.formulas = {} # cell -> formula string
    self.cache = {}  # cell -> computed value

  def setCell(self, cell: str, formula: str) -> None:
    self.formulas[cell] = formula
    self.cache = {} # simple, correct invalidation

  def getCell(self, cell: str, visiting: set = None) -> float:
    if visiting is None:
      visiting = set()
    if cell in self.cache:
      return self.cache[cell]
    if cell in visiting:
      raise ValueError(f"Circular dependency at {cell}")
    visiting.add(cell)
    formula = self.formulas.get(cell, "0")
    # replace whole references only, so A1 never matches inside A10
    expr = REF.sub(lambda m: str(self.getCell(m.group(0), visiting)), formula)
    result = float(eval(expr, {"__builtins__": {}})) # use a real parser outside an interview
    visiting.discard(cell)
    self.cache[cell] = result
    return result

Use a fixed-size thread pool as the concurrency limit, a shared visited set protected by a lock, and a queue of URLs to fetch. Workers pull a URL, fetch the page, extract links, and enqueue the ones they successfully add to the visited set. Check-and-add must happen under the same lock, or two threads can claim the same URL. Handle failures with bounded retries, and decide how the crawl ends: usually when the queue is empty and no worker is busy.

Hello Interview lists it as an OpenAI question and notes you need to coordinate threads, avoid crawling the same URL twice, handle failures and "probably implement some kind of rate limiting." Per-host politeness (a delay or token bucket per domain) is a likely extension, as is staying within one hostname.

Store the graph as a map from user to a set of followed users. snapshot() can deep-copy the current state and return an ID, which is fine at interview scale; the stronger version records every follow and unfollow with a version number, so a snapshot is just a version and a historical query replays or binary-searches each edge's history. recommend(user) walks the user's follows, collects who they follow, removes the user and anyone already followed, and ranks candidates by how many paths lead to them.

Hello Interview's OpenAI question list shows this as a coding question for mid-level to principal candidates, reported by 23 users and last asked in early September 2026. Ask whether recommend() should run against the live graph or against a snapshot, and whether ties need a stable order. Both change the implementation.

Split the target path on "/" and process it against a stack that starts at the current directory, or at root if the path is absolute. Skip empty parts and ".", pop on "..", push anything else. With symbolic links, each time you push a component, check whether the partial path is a link; if it is, replace the stack with the resolved target (absolute or relative to the link's parent) and continue with the remaining components.

Hello Interview lists this as an OpenAI question and points out the hard part: resolving links correctly and detecting cycles. Guard against a link loop with a visited set of link paths, or a maximum resolution depth like real systems use. Clarify early whether ".." after a link should go to the physical parent or the logical one, because shells differ.

Hello Interview's OpenAI list has "Design Online Chess" as its most reported OpenAI system design question: 94 reports, last asked in early September 2026, for junior to senior manager candidates. The prompt asks for something like Chess.com across devices, with move validation, matchmaking, leaderboards and move undo.

Keep each game's state on one authoritative game server, with clients connected over WebSockets, so moves are validated and ordered in one place and there's no race between two devices. Store every game as an append-only move list; undo is then an agreed request that removes the last move, and replays come for free. Matchmaking is a queue per time control, bucketed by rating and widened the longer a player waits. Leaderboards can be a sorted set updated after each rated game. Discuss reconnection (clock handling while a player is offline), cheat detection as an offline pipeline, and what happens when a game server dies mid-game.

Prompt injection is when untrusted input, from a user or from retrieved content, gets the model to ignore its instructions or act on an attacker's. Defend in layers. Input: classify likely injection before it reaches the model, and mark untrusted content clearly as data. Privileges: give the model only the tools and data the task needs, and require confirmation for anything irreversible, so a successful injection can't do much. Output: check responses for leaked system instructions, unexpected tool calls or links to unknown hosts. Monitoring: flag sessions where behavior leaves the expected task.

No single layer is enough, and it's worth saying that the goal is to raise the cost of an attack and limit the damage, not to make injection impossible. Bring one concrete injection example, such as instructions hidden in a web page the model is summarizing, and walk through which layer stops it.

Start with tiering. Customers with a latency SLA need reserved capacity that free or best-effort traffic can't consume, otherwise a free-tier burst slows a paying customer. Inside a tier, use weighted fair queuing rather than FIFO so one high-volume customer can't starve the rest. Autoscale on queue depth and time-in-queue per tier, not raw GPU utilization, which can look healthy while one tier's queue backs up.

Batching is where tenants meet. Continuous batching lets requests from different customers share GPU steps, but the scheduler has to enforce per-tenant token limits and preempt or cap very long generations so they don't hold short requests hostage. Mention admission control (reject early with a clear error instead of timing out) and how you'd measure fairness: per-tenant p99 latency against its SLA.

RLHF fine-tunes a language model against a reward signal learned from human preferences. The classic pipeline: supervised fine-tuning on demonstrations, training a reward model on human comparisons of two outputs for the same prompt, then optimizing the policy against that reward model with RL (typically PPO) and a KL penalty that keeps it close to the starting model. Where it goes wrong: reward hacking, where the policy finds outputs that score well but aren't actually better (verbosity is the well-known case); distribution shift, where the policy drifts to outputs the reward model never saw and scores unreliably; and a reward model that learns labeler quirks.

Follow-ups often ask about alternatives such as DPO, which skips the explicit reward model, or AI feedback in place of human labels, and when you'd choose each. Characterizing the failure modes matters more than deriving the PPO update.

They attack different bottlenecks, so they compose more than they compete. Quantization (INT8, FP8 or lower for weights, sometimes activations) cuts memory and bandwidth for a small, measurable accuracy cost; validate on your own eval set, because some tasks suffer more than others. Speculative decoding has a small draft model propose several tokens that the large model verifies in one pass. The original paper reported 2 to 3 times faster generation with identical output distribution, because the large model still decides every token. The cost is running the draft model and a speedup that depends on how often its guesses are accepted.

LoRA doesn't cut per-token cost for a single model. It makes fine-tuning cheap and lets one server hold many small adapters on a shared base model, which matters for multi-tenant serving. A strong answer says which bottleneck each one addresses, memory and bandwidth, sequential latency, or per-customer model count, and when you'd stack them.

Data parallelism copies the whole model to every worker, splits each batch across them, and averages gradients with an all-reduce after each step. It stops working once the model no longer fits on one GPU. Tensor parallelism splits individual layers (attention heads, matrix multiplications) across GPUs, which needs a fast interconnect like NVLink because activations move inside every forward and backward pass. Pipeline parallelism gives each GPU a contiguous block of layers and passes activations down the line; it needs micro-batching to keep every stage busy and shrink the idle "bubble."

Large runs combine all three: tensor parallelism inside a node, pipeline parallelism across nodes, data parallelism across the cluster. A likely follow-up is what breaks first as you scale from 8 to 512 GPUs. Usually it's communication, all-reduce bandwidth or pipeline bubbles, before raw compute.

import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

dist.init_process_group(backend="nccl")
local_rank = int(os.environ["LOCAL_RANK"]) # set by torchrun
torch.cuda.set_device(local_rank)
model = MyLLM().to(local_rank)
model = DDP(model, device_ids=[local_rank])
# gradients are all-reduced automatically during loss.backward()

In plain data parallelism every GPU holds a full copy of the parameters, gradients and optimizer states, which wastes memory; Adam alone keeps two extra FP32 values per parameter. ZeRO removes the duplication in stages. Stage 1 shards optimizer states across data-parallel workers. Stage 2 also shards gradients. Stage 3 also shards the parameters, so each GPU gathers a layer's full weights only while computing that layer and then frees them.

The price is communication, since stage 3 gathers parameters before every layer, so it wants a fast interconnect and pairs well with activation checkpointing. A common follow-up is how ZeRO-3 compares with tensor parallelism for a 70B-plus model. It isn't either-or: ZeRO attacks memory redundancy, tensor parallelism splits compute, and large runs often use both.

A straggler is a worker whose step time is consistently slower than its peers, often because of a flaky network card, thermal throttling or a GPU that is failing slowly. Synchronous training waits for the slowest worker at every all-reduce, so one straggler slows the whole job. Detect it by logging per-worker step time and flagging any worker whose rolling p95 sits well above the fleet median.

Mitigation: automatically cordon and replace the node once health checks confirm the hardware problem, then resume from the latest checkpoint on a fresh allocation. Some setups relax synchronization to limit the damage from one slow worker, at a cost to reproducibility. The strong answer covers the full loop, detection, confirmation, replacement and resume, with no human needed at 2 a.m.

Store candidate strings in a trie where each node tracks how often completions pass through it. As the user types, walk to the node for the current prefix, which costs time proportional to the prefix, not the dataset. From there, either precompute and cache the top suggestions at each trie node, which makes reads fast but costs memory and slows updates, or run a bounded search for the best completions on demand. For read-heavy traffic, precomputing wins.

Framing it around prompt suggestions rather than a search box ties a classic problem to the product. The follow-up to prepare: updating frequencies as new queries arrive without rebuilding the trie, which means incremental updates along the path and refreshing cached top lists only where a count crossed a threshold.

Each user has a balance and every API call deducts the tokens it used. The hard constraint is atomicity: two simultaneous requests must not let a user overspend. Options: lock the balance row per deduction (simple, becomes a hotspot at high call rates), optimistic concurrency with a version check and retry, or a reserve-then-settle pattern that reserves an estimated amount when the request starts and settles the real count when generation finishes.

Reserve-then-settle fits streaming generation best, because the final token count isn't known up front. Prepare the abandoned-request case, a client disconnecting mid-stream: bill what was generated, and let the unused reservation expire on a TTL so credits return automatically. For scale, keep hot balances in a fast store with periodic reconciliation against an append-only usage ledger.

Every run needs an immutable record tied to one run ID: dataset version, hyperparameters, base checkpoint, code commit and resulting eval metrics. Without it, a team running hundreds of jobs a week soon can't answer "what changed between the run that worked and the one that didn't." Metadata goes in a queryable database. Large artifacts (checkpoints, datasets) go in object storage addressed by content hash, so identical datasets aren't stored twice.

Versioning needs a promotion workflow: a run produces a candidate checkpoint, an eval suite runs automatically on a held-out benchmark, and only checkpoints that clear the bar get tagged as deployable. Add lineage so any served model can be traced back to its exact data and code.

Clarify scope first: the UI, the API layer, session storage, model routing, or all of them. The UI streams tokens as they arrive (server-sent events or WebSockets) and exposes a model picker, system prompt editor and parameter controls. The API layer authenticates, enforces rate limits and quotas, forwards the request to model serving (usually abstracted by the interviewer) and streams the response back while counting tokens. Storage keeps conversations as ordered messages per session, plus saved presets and usage records.

It's a product you can use before the interview, so use it. Candidates who design a generic chat app without streaming, token counting, parameter handling or cancellation of an in-flight request show they haven't. Good extensions: comparing two models side by side, and sharing a prompt safely without leaking an API key.

Pin down the target first: 200ms P99 for time to first token is realistic, for a full long completion it is not. The request path runs from an API gateway (auth, rate limits) to a load balancer and GPU replicas running a serving engine such as vLLM or Triton. In FP16 the weights of a 20B model take about 40GB, so one 80GB GPU fits a replica with room left for the KV cache; tensor parallelism across two GPUs is a latency choice, not a requirement.

Use continuous batching so new requests join in-flight batches instead of waiting, and paged KV-cache management to avoid fragmentation. Cap batch size or tokens per step to protect tail latency, and autoscale on queue depth and time-to-first-token, not CPU. Quantizing weights to INT8 or FP8 frees memory and bandwidth for bigger batches, at an accuracy cost you validate offline before rollout.

Jobs are submitted with a type, payload, run time and retry policy. Schedulers find due jobs and claim them with an atomic compare-and-swap on a status field (pending to claimed), so two schedulers can't take the same job. Workers run claimed jobs, retries use exponential backoff up to a limit, and jobs that exhaust retries move to a dead-letter table. The database is the source of truth, and schedulers and workers stay stateless. Partition the jobs table by time bucket so the due-jobs query scans a narrow range.

The key follow-up: a worker crashes after claiming a job. Answer with a lease: the worker updates a heartbeat, and a job whose heartbeat is older than N seconds goes back to pending. That means jobs can run more than once, so ask for idempotent handlers or de-duplicate by job ID.

Split it into ingestion and query paths. Ingestion chunks documents (help articles, resolved tickets) into passages of a few hundred tokens with some overlap, embeds them and writes them to a vector index with approximate nearest-neighbor search. At query time, embed the question with the same model, fetch the top-k passages, and put them in the prompt with the question and an instruction to answer only from them and cite which passage it used.

Design around two failures: irrelevant retrieval, fixed with a reranker that re-scores the top candidates, and stale content, fixed with re-ingestion on document change. The follow-up to expect is evaluation: measure retrieval separately from answer quality, with a labeled set of questions and their correct passages, so you know which half broke.

Moderation runs on both sides of the model: the prompt before or alongside generation, and the completion before or while it streams to the user. A classifier much smaller and faster than the generation model scores each policy category with a confidence value. Latency drives the design, so the input check usually runs in parallel with generation and the output check scans the stream in chunks instead of waiting for the full response. Borderline scores go to human review rather than an automatic block.

Version your categories and thresholds, because policy changes and months later you'll need to know which version flagged a given item. The interviewer will push on precision versus recall: tuned for recall, it blocks legitimate use often enough that users notice; tuned for precision, real violations get through. Name the trade-off and how you'd measure both in production.

Define the target first: claims that are false, or claims not supported by the provided context. Then layer the evaluation. Automatic checks: entailment models that flag claims the reference text doesn't support. Model-graded checks: a stronger model scores factual consistency against a rubric, calibrated against human labels. Human review: a stratified sample weighted toward entity-heavy and numerical answers, where errors cluster.

To reduce hallucination, ground answers in retrieved sources, let the model say it doesn't know and reward that in training and evals, and use domain fine-tuning where it applies. Surface-overlap metrics like BLEU say little about factuality, so don't lead with them. The interesting part is the human loop: who labels, how you settle disagreements, and how you check that the evaluation itself stays reliable over time.

The goal is to bound how much compute a failure costs, and at this scale failures are routine. Checkpoint frequency is a trade-off: too often and I/O steals GPU time, too rarely and a failure wipes out hours. Asynchronous checkpointing helps, copying state off the GPUs quickly and writing it in the background while training continues. Shard checkpoints so each rank writes its own slice and no single node becomes a bottleneck, and store them somewhere redundant enough that a storage failure doesn't take the recovery point with it.

The detail interviewers care about is completeness. A checkpoint that wrote 990 of 1,000 shards is useless unless restart logic detects it. Write a manifest last, only after every shard is confirmed, and have the orchestrator resume automatically from the newest checkpoint that has one.