Nvidia Interview Questions (2026): What They Actually Ask

Sort the intervals by start time. Iterate through, and for each interval, check if it overlaps with the last merged interval (i.e., its start is less than or equal to the last merged end). If yes, extend the end. If no, add it as a new interval.

A standard medium-leaning warm-up that fits a HackerRank-style exercise. Expect edge cases: single-interval input, entirely overlapping intervals, and intervals that touch but do not overlap (e.g., [1,3] and [3,5], most interviewers want these merged).

Floyd's tortoise and hare. Move a slow pointer one node at a time and a fast pointer two nodes at a time. If the list has a cycle, they meet inside it. If the fast pointer hits null first, there's no cycle.

The part candidates fumble is finding the cycle's entry point. Once slow and fast meet, reset one pointer to the head and advance both one step at a time, they'll meet exactly at the entry node. To remove the cycle, walk from that entry point until you find the node whose next pointer points back to it, then set that pointer to null.

def detect_and_remove_cycle(head):
  slow = fast = head
  has_cycle = False

  while fast and fast.next:
    slow = slow.next
    fast = fast.next.next
    if slow == fast:
      has_cycle = True
      break

  if not has_cycle:
    return head

  # find the entry point of the cycle
  ptr = head
  while ptr != slow:
    ptr = ptr.next
    slow = slow.next

  # walk the cycle to find the node pointing back to the entry
  runner = slow
  while runner.next != slow:
    runner = runner.next
  runner.next = None

  return head

Interviewers can usually tell quickly whether you're describing a real failure or a disguised humblebrag ("my code was so good the review took forever"). Name the actual mistake, a wrong technical assumption, a scope you underestimated, a communication gap with another team, and be specific about what it cost. A missed deadline. A rewrite. An incident.

The answers that land describe a concrete process change, not a restated lesson. "I learned to test more" is forgettable. "I added a staging rollout step because we'd been shipping straight to prod, and that's exactly how a null-pointer regression got through" is a story an interviewer remembers. Have a second example ready too, they sometimes ask whether you've actually applied the lesson since, on a different project.

Be concrete about what you cut, why, and what it cost. "We shipped a prototype without error handling to hit a demo deadline, then paid down the tech debt in sprint 3" is the kind of answer that lands. Vague answers ("I always try to balance both") tell the interviewer nothing and signal that you haven't made a real trade-off consciously.

The follow-up is almost always: "Did that turn out to be the right call?" Have a real answer, even if it's "no, we underestimated the maintenance cost."

Interviewers want to see that you take partial ownership even when the failure was partly someone else's fault, and that you made a structural change to prevent recurrence. "We added a weekly sync" is acceptable. "We added a weekly sync and a shared doc that captured the decision and its rationale" is better because it shows you thought about the failure mode, not just the symptom.

Every company asks this. Nvidia's version is harder because they want specificity about the technical direction, Blackwell architecture, the CUDA ecosystem, Triton Inference Server, DGX Cloud, NIM microservices, or whatever the team you're interviewing with actually works on. Generic "AI is the future" answers are transparent and don't work.

Research what the specific team ships. For a CUDA platform role, that's the CUDA toolkit release notes. For an inference role, that's the recent TensorRT-LLM changelog and the Triton Inference Server release notes. For a chip architecture role, that's the Blackwell architecture paper from Hot Chips 2024.

Nvidia's own answer is that most candidates have a decision within a matter of weeks from their first interview. The in-person onsite is usually the step that sets the pace, because it has to be scheduled at an Nvidia office. If you are juggling other offers, tell your recruiter early.

Nvidia does not publish a difficulty level. What it does say is that technical candidates may get a coding exercise, usually on HackerRank, and that interviews run 30 to 60 minutes each. In a 30 to 60 minute slot, the realistic target is a medium-difficulty problem solved cleanly, tested, and defended when the interviewer pushes on complexity or edge cases.

It depends heavily on the team. For GPU architecture, CUDA platform, driver, or compiler roles, yes, you'll get detailed CUDA questions and need real hands-on experience. For application-layer SWE roles (developer tools, platform infrastructure, data engineering), the CUDA questions are lighter and more conceptual. Ask your recruiter which category your loop falls into. It's a fair question and they'll tell you.

Nvidia says interns typically complete phone interviews only, while full-time hires, new grads included, do phone interviews followed by virtual or in-person interviews and an in-person onsite before an offer. A HackerRank coding exercise is common for technical roles. Openings and events for students are listed on Nvidia's University Recruiting page.

Very important for systems and GPU roles. Interviewers will go deep on memory management, RAII, virtual dispatch, threading primitives, and template mechanics. For ML systems or application-layer roles, Python is acceptable in coding rounds but interviewers often ask C++ conceptual questions anyway. If you're applying to any role with "systems," "compiler," "driver," or "CUDA" in the title, C++ fluency is effectively a prerequisite.

Nvidia does not publish this, and its hiring page says the number of interviews depends on the role. Expect design questions to carry more weight as the level goes up, and ask your recruiter directly whether your loop has a dedicated system design round. It is a normal question and recruiters answer it.

Use a HashMap for O(1) key lookups combined with a doubly linked list to track recency order. The list head holds the most recently used entry; the tail holds the least recently used. On a get, move the node to the head. On a put that exceeds capacity, evict the tail node and remove its key from the map.

A natural extension is making it thread-safe. A plain read-write lock does not work as well as it sounds, because every get also moves the entry to the front of the list, so reads mutate state too. Be ready to explain that and propose a mutex, lock striping or an approximate LRU.

from collections import OrderedDict

class LRUCache:
  def __init__(self, capacity: int):
    self.cap = capacity
    self.cache = OrderedDict()

  def get(self, key: int) -> int:
    if key not in self.cache:
      return -1
    self.cache.move_to_end(key)
    return self.cache[key]

  def put(self, key: int, value: int) -> None:
    if key in self.cache:
      self.cache.move_to_end(key)
    self.cache[key] = value
    if len(self.cache) > self.cap:
      self.cache.popitem(last=False)

Keep a hash map of counts that you update as each element arrives. When someone asks for the top k, run a size-k min-heap over the distinct elements, which costs O(m log k) for m distinct values. Trying to keep a live size-k heap updated on every arrival is harder than it looks, because an element already inside the heap carries a stale count, and saying so out loud is part of a good answer.

The stream aspect is the real test. Candidates who hard-code a "collect all, then sort" approach get pushed on it immediately. Interviewers want to see you reason about bounded memory and online processing.

import heapq
from collections import defaultdict

class TopKStream:
  def __init__(self, k):
    self.k = k
    self.freq = defaultdict(int)

  def add(self, num):
    self.freq[num] += 1

  def top_k(self):
    # O(m log k) over m distinct elements
    return [x for x, _ in heapq.nlargest(
      self.k, self.freq.items(), key=lambda kv: kv[1])]

DFS with a recursion stack (a set of nodes currently on the active call stack) and a visited set. If DFS reaches a node already in the recursion stack, a cycle exists. After finishing DFS from a node, remove it from the recursion stack.

A common follow-up: "how does this change for an undirected graph?" The undirected version only needs a visited set and parent tracking, not a separate recursion stack.

def has_cycle(graph):
  visited = set()
  rec_stack = set()

  def dfs(node):
    visited.add(node)
    rec_stack.add(node)
    for neighbor in graph.get(node, []):
      if neighbor not in visited:
        if dfs(neighbor):
          return True
      elif neighbor in rec_stack:
        return True
    rec_stack.remove(node)
    return False

  for node in graph:
    if node not in visited:
      if dfs(node):
        return True
  return False

Modified binary search. At each step, determine which half of the array is sorted (compare mid to the boundaries). If the target falls within the sorted half, search there. Otherwise, search the other half.

Follow-ups often ask about the case with duplicate elements, which breaks the standard approach and requires a fallback to linear search in the worst case (O(n)).

Two valid approaches, and a strong answer covers both along with their trade-offs. A min-heap of size k gives O(n log k) time and holds only the k largest elements seen so far, evicting the smallest whenever the heap exceeds size k. Quickselect, a variant of quicksort's partitioning step, gives O(n) average time by recursing into only the half of the array that contains the kth position, but degrades to O(n^2) on adversarial input unless the pivot is randomized.

The natural follow-up asks which approach fits a streaming input (heap, since quickselect needs the full array in memory) versus a fixed array you're allowed to mutate in place (quickselect, since it skips the heap's log factor).

import random

def find_kth_largest(nums, k):
  target = len(nums) - k # index if nums were sorted ascending

  def partition(left, right, pivot_index):
    pivot = nums[pivot_index]
    nums[pivot_index], nums[right] = nums[right], nums[pivot_index]
    store_index = left
    for i in range(left, right):
      if nums[i] < pivot:
        nums[store_index], nums[i] = nums[i], nums[store_index]
        store_index += 1
    nums[right], nums[store_index] = nums[store_index], nums[right]
    return store_index

  left, right = 0, len(nums) - 1
  while True:
    pivot_index = random.randint(left, right)
    new_pivot_index = partition(left, right, pivot_index)
    if new_pivot_index == target:
      return nums[new_pivot_index]
    elif new_pivot_index < target:
      left = new_pivot_index + 1
    else:
      right = new_pivot_index - 1

Interviewers present a short snippet and ask you to trace execution line by line: which constructor fires, when does the destructor run, is there a memory leak? Copy construction vs. the assignment operator is a common gotcha. RAII semantics and when a copy constructor gets implicitly called (e.g., passing by value) are the real focus.

The question tests whether you understand the object lifecycle, not just that you can write classes. The key is knowing when the destructor runs for stack-allocated vs. heap-allocated objects.

The core requirement: maintain a shared reference count on the heap alongside the managed object. The copy constructor increments the count; the destructor decrements it and frees the object when the count reaches zero. The assignment operator must handle self-assignment and correctly decrement the old referent.

This tests whether you actually understand RAII vs. whether you've just used std::shared_ptr. Interviewers follow up on circular references, two shared_ptr objects pointing at each other, both with refcount 1, which never reaches zero and leaks. The fix is std::weak_ptr.

Each class with virtual functions has a vtable: a static array of function pointers. Each object of that class holds a hidden vptr (pointer to its vtable). Virtual dispatch dereferences the vptr, looks up the function pointer in the vtable, and calls it, adding one indirection and potentially a cache miss compared to a direct call.

Overhead matters in tight GPU driver loops where call frequency is very high. Interviewers also cover lambda captures (closures vs. function pointers), template instantiation (compile-time polymorphism with zero runtime overhead), and move semantics (avoiding unnecessary copies of large buffers).

False sharing happens when two threads write to different variables that happen to sit on the same cache line, typically 64 bytes. The cache coherence protocol treats the whole line as dirty on every write, so it invalidates the other core's copy even though the two threads never touch the same variable. The result looks like a data race in a profiler, cores repeatedly re-fetching a line from a neighbor's cache, but it's really a memory layout problem, not a logic bug.

Fix it by padding hot per-thread counters or state to their own cache line (alignas(64) on the struct), or by reorganizing data so per-thread fields aren't adjacent, structure-of-arrays instead of array-of-structures for the hot fields. Driver and runtime code is prone to it, since independent per-thread counters and queue heads are exactly the pattern that triggers it.

#include <atomic>

struct PaddedCounter {
  alignas(64) std::atomic<long> count;
  char padding[64 - sizeof(std::atomic<long>)];
};

PaddedCounter counters[NUM_THREADS]; // one cache line per thread

SIMD applies one instruction to multiple data lanes in strict lockstep, the classic example being CPU vector units like AVX, where there's no real per-lane control flow. SIMT, Nvidia's model, groups 32 threads into a warp and issues one instruction to all of them, but each thread keeps its own program counter and register state conceptually, so threads can take different branches.

That flexibility isn't free. When threads in a warp diverge, the hardware runs both branch paths and masks off the threads that shouldn't be active for each one, which is why the warp divergence question later in this section matters so much. Interviewers tend to ask this one first, as a check on whether you actually know what a warp is before they ask you to optimize around it.

Nvidia's CUDA C++ Best Practices Guide puts it plainly: global memory loads and stores by the threads of a warp are coalesced into as few transactions as possible. When threads in a warp access contiguous addresses, that can be a single transaction. Non-coalesced access forces multiple transactions and tanks bandwidth, in pathological cases, you can lose 10x or more of available memory bandwidth.

The follow-up is almost always: "How do you restructure a matrix transpose kernel to achieve coalesced access?" The answer: load a tile of the input into shared memory using coalesced reads, then write from shared memory to the transposed output location (which would be non-coalesced from global memory directly).

Shared memory has roughly 100x lower latency than global memory but is limited per thread block, typically 48 to 96 KB on recent GPUs (on H100 a single thread block can opt in to up to 227 KB). Load data into shared memory when threads in a block will reuse it multiple times. The access pattern within shared memory also matters: bank conflicts occur when multiple threads in a warp access different addresses that map to the same memory bank, serializing those accesses.

Interviewers want you to connect shared memory decisions to coalescing decisions. They're two sides of the same optimization: coalesced global reads fill shared memory efficiently, and then shared memory provides low-latency reuse for the computation phase.

A warp is 32 threads executed in lockstep on a single streaming multiprocessor. When threads within a warp take different branch paths, the hardware serializes those paths, it executes the true branch with some threads active and the false branch with the remaining threads active, then merges. Both paths run, even the ones that aren't "taken" for a given thread.

Strategies to reduce divergence: restructure data so that threads within a warp process elements of the same type (reducing branch probability), use predicated instructions for short branches where both paths are cheap, and avoid dynamic indexing into shared memory where the index depends on thread ID in a non-uniform way.

Nvidia Nsight Systems for timeline analysis: CPU-GPU interaction, kernel launch overhead, synchronization points, and data transfer timing. Nsight Compute for kernel-level profiling: warp efficiency, memory throughput, achieved occupancy, and instruction mix. The distinction matters, Nsight Systems answers "where is time being spent at the application level?" while Nsight Compute answers "why is this specific kernel slow?"

Nvidia's own documentation splits the jobs the same way: Nsight Systems for the system-wide timeline and Nsight Compute for a single kernel. A natural follow-up is how you'd instrument a multi-GPU training run. For that case, the answer involves NCCL profiling hooks and per-rank timeline correlation in Nsight Systems.

GPU virtual memory via CUDA Unified Virtual Memory (UVM) allows CPU and GPU to share an address space. The cost: page faults trigger expensive migrations between host and device memory, with each fault adding microseconds of latency. For workloads that access memory in predictable patterns, pinned (page-locked) host memory with explicit cudaMemcpy transfers is almost always faster than relying on UVM page migration.

Interviewers want you to know when UVM is acceptable (prototyping, irregular access patterns that are too complex to manage manually, very large datasets that exceed GPU memory) vs. when it's a performance liability (hot-path inference, latency-sensitive kernels).

A switch from user mode into the kernel costs far more than a function call, because the CPU changes privilege level and often loses cache and TLB state. For GPU work the question is which CUDA calls pay it. Allocation calls such as cudaMalloc and cudaFree can involve the operating system and are slow, which is why production code pre-allocates a pool and reuses buffers. Kernel launches are mostly submitted from user space, but each still carries a few microseconds of launch overhead, which is what batching launches and CUDA Graphs are meant to cut. Streams then let you overlap the remaining transfers with compute.

This comes up for driver and compiler roles more than for application-layer roles, but Nvidia expects any systems candidate to know it.

The default stream serializes everything, a kernel launch and a memory copy issued on it run one after the other even if they don't depend on each other. Non-default streams change that. Issue a kernel on stream A and a cudaMemcpyAsync on stream B, and modern GPUs, which have separate copy engines from their compute engines, run them concurrently.

The catch that trips people up, and the Best Practices Guide states it directly: the asynchronous copy requires pinned (page-locked) host memory, allocated via cudaMallocHost or cudaHostAlloc. Pageable memory can't be transferred asynchronously, the driver stages it through a pinned buffer first, which silently serializes your "concurrent" transfer. Interviewers ask this as a follow-up almost every time, it's the detail that separates someone who's read the CUDA programming guide from someone who's actually pipelined a real workload.

Layer fusion combines adjacent operations (e.g., conv + batch norm + ReLU) into a single kernel, eliminating intermediate memory writes and reads. Quantization (INT8 or FP8 for inference where precision tolerance allows) reduces memory bandwidth and increases throughput on hardware with dedicated low-precision units. Kernel auto-tuning via TensorRT finds the best kernel implementation for a given input shape. Batching requests amortizes kernel launch overhead and maximizes GPU utilization between calls. For the serving layer, know what Triton Inference Server does, since it is Nvidia's own serving product.

Connect these optimizations to hardware realities rather than reciting a checklist. "Layer fusion reduces memory round-trips because each intermediate tensor otherwise sits in global memory between operations" is the level of reasoning they're looking for.

Be specific about the timeline, how you ramped (documentation? colleagues? experiments?), what you got wrong first, and what the outcome was. Hardware and software stacks at a company like Nvidia move fast, and internal documentation for new products can be thin. The question tests whether you can self-direct learning in novel territory, not whether you've memorized existing frameworks.

Interviewers follow up by asking what resources were most useful and what you'd do differently if you had to do it again. Vague answers ("I read the docs and asked questions") don't land here.

Interviewers want to see that you can hold a position with evidence and change your mind with evidence. Pure deference to seniority is a red flag. So is stubbornness that ignores new information when it arrives. The strongest answers describe a specific technical disagreement, the evidence you brought to the conversation, what the outcome was, and whether, in retrospect, the decision was correct.

This question frequently turns into a 20-minute technical discussion about the actual trade-off. Prepare your story with the same technical depth you'd bring to a system design question.

Strong answers name the actual tool you used to find it, perf, Nsight, flamegraph, custom instrumentation, a printf at the right place. Name what made it hard to see (it only appeared under production traffic patterns, it was masked by another bottleneck, the tooling didn't expose it directly). And give the actual size of the win.

Round numbers ("it was 2x faster") get follow-up questions about how you measured. If you don't have a rigorous measurement story, you'll get pressed on it, so bring the before and after numbers and how you measured them.

Real answer structure: first, identify what decision is actually irreversible vs. reversible, irreversible decisions (architectural choices, API contracts) warrant more information-gathering before committing; reversible decisions should be made faster. Second, bound your uncertainty with a quick experiment or proxy measurement rather than waiting for complete data. Third, commit with an explicit trigger to revisit (a date, a metric threshold, a milestone).

Vague answers ("I gather as much data as possible") don't land well here. Hardware and platform work is planned long before the product ships, so interviewers want to hear your actual process, not a principle.

For a sequence of length n and hidden dimension d, computing the attention scores QK^T costs O(n^2 * d), and the attention matrix itself takes O(n^2) memory. Double the context length and you quadruple the compute and memory cost of the attention block specifically, the feed-forward layers stay linear in n. That quadratic wall is why FlashAttention exists, it fuses the softmax and the two matmuls so the full n x n matrix never gets materialized in HBM, only computed in tiles that fit in on-chip SRAM.

Interviewers often connect this straight to the KV-cache question below. During generation, each new token only attends against the cached keys and values, so the quadratic cost gets paid once during prefill, not per token during decode. Candidates who conflate the two, and claim generation itself is quadratic per token, get corrected fast.

In autoregressive generation, each new token attends to all previous tokens. Without a cache, you'd recompute the key and value tensors for every previous token at every decoding step, O(n^2) total compute for a sequence of length n. KV-cache stores those key and value tensors after they're computed, so each decoding step only computes attention for the new token against the cached context.

The cost: memory grows linearly with sequence length and batch size, which is why PagedAttention (vLLM) and sliding window attention (Mistral) exist. A likely follow-up for LLM-focused roles by asking how you'd manage KV-cache memory when multiple requests have different sequence lengths in the same batch.

LoRA freezes base model weights and trains small rank-decomposed matrices injected into the attention layers. The update to a weight matrix W is expressed as W + AB, where A is (d x r) and B is (r x d), and r is much smaller than d. Much lower memory footprint than full fine-tuning, at a cost of some expressiveness for tasks that require large distributional shifts from the base model.

Used when you have limited GPU budget, when you need fast task-switching between many adapters (LoRA multiplexing in production serving), or when the base model is already close to the target task. Full fine-tuning is worth the cost when LoRA's rank limitation actually constrains performance, which is task-dependent and usually determined empirically.

Standard backprop keeps every intermediate activation from the forward pass around for the backward pass, so memory scales with layer count times batch size times sequence length. Gradient checkpointing throws most of that away and recomputes it on demand during the backward pass instead, trading roughly 20 to 30% more compute time for activation memory that's often the difference between a model fitting on your GPU budget and not fitting at all.

You reach for it when you're memory-bound rather than compute-bound, which describes most large transformer training runs on a fixed cluster. Interviewers sometimes push on placement, checkpointing every transformer block is the standard granularity. Checkpoint too finely and the recomputation overhead eats the memory savings you were trying to get.

A senior-level systems question. The expected approach uses atomic compare-and-swap operations on a free-list head pointer, with careful alignment to cache line boundaries. Pre-allocate a large contiguous slab and track free blocks with a lock-free stack of block pointers using std::atomic.

Interviewers care about the reasoning as much as the code. "Why lock-free?" is always a follow-up. The expected answer: mutex contention at GPU driver call frequency creates measurable latency spikes that a lock-free approach avoids, at the cost of more complex ABA-problem handling.

#include <atomic>
#include <cstddef>
#include <cstdlib>

struct Block {
  Block* next;
};

class LockFreePool {
  std::atomic<Block*> head;

public:
  LockFreePool() : head(nullptr) {}

  void* allocate(size_t size) {
    Block* old_head = head.load(std::memory_order_relaxed);
    while (old_head) {
      if (head.compare_exchange_weak(old_head, old_head->next,
        std::memory_order_acquire, std::memory_order_relaxed)) {
        return old_head;
      }
    }
    return std::aligned_alloc(64, size); // fallback, 64-byte aligned
  }

  void deallocate(void* ptr) {
    Block* block = static_cast<Block*>(ptr);
    Block* old_head = head.load(std::memory_order_relaxed);
    do {
      block->next = old_head;
    } while (!head.compare_exchange_weak(old_head, block,
         std::memory_order_release, std::memory_order_relaxed));
  }
};

Start with Nsight Compute to get a kernel-level profile: achieved occupancy, memory throughput, warp efficiency, and instruction-level breakdown. Then reason through the likely causes in priority order. Uncoalesced memory access is the most common culprit. Bank conflicts in shared memory are second. Thread divergence (branches within a warp) serializes execution and reduces effective throughput. Insufficient parallelism means the GPU has too few active warps to hide memory latency, check that your grid and block dimensions are appropriate for the problem size.

Name the tool before you name the fix, and be clear on the split between Nsight Systems (timeline, CPU-GPU interaction) and Nsight Compute (kernel internals). Using the wrong tool for the question is a red flag.

Start with INT8 or FP8 quantization for weights and activations where the model's accuracy tolerance allows, this cuts both memory bandwidth and compute. Apply kernel fusion via TensorRT or torch.compile to collapse attention and feed-forward layers. Use FlashAttention for the attention layers, which reduces HBM reads and writes for the attention matrix. For the KV cache, PagedAttention (as in vLLM) reduces fragmentation and allows larger effective batch sizes. For very long sequences, speculative decoding with a smaller draft model can improve tokens-per-second at the cost of some implementation complexity.

Expect follow-up questions about the latency-throughput trade-off in each choice, quantization helps latency but can hurt accuracy on certain tasks; larger batches improve throughput but increase per-request latency.

MoE replaces each dense feed-forward block with multiple specialist sub-networks ("experts"), with a gating network routing each token to a subset (typically top-2). Compute per token stays constant even as total parameter count grows, Mixtral 8x7B has 47B total parameters but activates roughly 13B per token. This gives MoE models a favorable FLOPs-per-quality ratio at training time.

Trade-offs: load imbalance across experts (some experts get routed to heavily, others sit idle), which requires auxiliary load-balancing losses during training. Increased communication overhead in distributed training, since experts may live on different devices and tokens need to be routed across the network. Serving complexity increases because some experts may be "cold" (not resident in GPU memory) if you're serving many adapters on one cluster.

Data parallelism replicates the full model on each GPU and shards the data. Simple to implement, but requires an all-reduce at each gradient step, as model size grows, this communication becomes the bottleneck. Model parallelism shards the model layers across GPUs, reducing per-GPU memory at the cost of communication for activations between layers. Pipeline parallelism assigns each GPU one stage of the computation and enables asynchronous micro-batching, but introduces pipeline bubbles (GPUs sitting idle waiting for the previous stage) that hurt GPU utilization.

In practice, Nvidia's Megatron-LM and Microsoft DeepSpeed use all three simultaneously (3D parallelism), with tensor parallelism within a node (NVLink), pipeline parallelism across nodes (InfiniBand), and data parallelism across replica groups. The optimal split depends on the model architecture and hardware topology.

Model GPU memory and compute as first-class constrained resources, not just CPU slots. Cover preemption strategy (checkpoint-and-restore vs. kill-and-resubmit, the latter being cheaper but harsher), priority queuing with aging to prevent starvation, and gang scheduling for jobs that need multiple GPUs simultaneously, a job requiring 8 GPUs must wait until all 8 are available, or fragment-and-stall becomes a real problem.

Interviewers push hard on the failure case: what happens when a job hangs? You need a watchdog process, a heartbeat from running jobs, and a kill mechanism that releases GPU memory even if the job process is unresponsive (which means you need a separate out-of-process memory manager, not just cleanup in the job itself).

Model sharding strategy first: tensor parallelism within a node (using NVLink for high-bandwidth inter-GPU communication), pipeline parallelism across nodes (using InfiniBand). Load balancing must be GPU-memory-aware, not just CPU-load-aware, a node with high CPU but low GPU memory utilization is still a valid target. Request batching with a configurable max wait time (e.g., 5ms) amortizes kernel launch overhead while keeping tail latency bounded.

At senior level, interviewers expect you to name TensorRT-LLM or vLLM as existing solutions and know their trade-offs. Be ready to compare them honestly: both are actively developed, so talk about the design choices (in-flight batching, paged KV-cache management, kernel fusion) rather than quoting benchmark numbers you cannot back up.

Parallelism strategy: tensor parallelism within a node using NVLink (8 GPUs per node), pipeline parallelism across nodes using InfiniBand. Checkpoint frequency is a trade-off: more frequent checkpoints reduce wasted compute on node failure but add synchronization overhead and storage I/O. With 128 GPUs, a node failure every few hours is realistic, plan for it. NCCL handles gradient reduction. ZeRO-3 optimizer state sharding (from DeepSpeed) spreads the optimizer states, gradients, and parameters across GPUs to fit larger models than any single GPU could hold.

Interviewers want to see that you can reason about the failure probability of 128 machines over a multi-day training run, not just describe the happy path.

Key design decisions: in-process memory vs. Redis vs. GPU-local caching. For sub-millisecond latency, in-process caching is usually necessary, a Redis round-trip adds 0.5 to 2ms of network latency even on a fast LAN, which violates the constraint. Cache key design for model inputs is hard: exact-match keys work for deterministic inputs, but near-duplicate inputs (semantically similar prompts) won't hit. Semantic caching using embedding similarity adds latency that may negate the cache benefit. Eviction policy matters: LRU works well for temporal locality in request patterns; LFU works better for long-tail distributions where some inputs recur at very high frequency.

Invalidation when model weights update requires either a versioned cache key (include model version in the key) or a cache flush, the versioned approach is better for gradual rollouts.