Nvidia Interview Questions (2026): What They Actually Ask

Sort the intervals by start time. Iterate through, and for each interval, check if it overlaps with the last merged interval (i.e., its start is less than or equal to the last merged end). If yes, extend the end. If no, add it as a new interval.

A staple that shows up across coding rounds and OAs alike. Expect edge cases: single-interval input, entirely overlapping intervals, and intervals that touch but do not overlap (e.g., [1,3] and [3,5], most interviewers want these merged).

Use two hash maps: one mapping characters from s to t, one mapping characters from t to s. For each character pair (c1, c2), verify both mappings are consistent. If c1 already maps to something other than c2, or c2 already maps to something other than c1, return false.

Reported in the same OA as the palindromic subsequence question. Seems like an easy warm-up but the bidirectional constraint catches people who only check one direction. Checking s-to-t alone allows "aa" and "ab" to incorrectly return true.

Stack-based approach. Push opening brackets onto the stack. When you encounter a closing bracket, check if the top of the stack is the matching opener. If yes, pop. If no, or if the stack is empty, return false. At the end, return true only if the stack is empty.

Reported from a 2026 onsite. Some interviewers extend it to nested conditions or multi-character brackets (e.g., matching <% %> tokens in template syntax).

Floyd's tortoise and hare. Move a slow pointer one node at a time and a fast pointer two nodes at a time. If the list has a cycle, they meet inside it. If the fast pointer hits null first, there's no cycle.

The part candidates fumble is finding the cycle's entry point. Once slow and fast meet, reset one pointer to the head and advance both one step at a time, they'll meet exactly at the entry node. To remove the cycle, walk from that entry point until you find the node whose next pointer points back to it, then set that pointer to null. Common on new-grad OAs and early phone screens, this one is a warm-up before the interviewer moves to something harder.

def detect_and_remove_cycle(head):
    slow = fast = head
    has_cycle = False

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            has_cycle = True
            break

    if not has_cycle:
        return head

    # find the entry point of the cycle
    ptr = head
    while ptr != slow:
        ptr = ptr.next
        slow = slow.next

    # walk the cycle to find the node pointing back to the entry
    runner = slow
    while runner.next != slow:
        runner = runner.next
    runner.next = None

    return head

Nvidia interviewers can tell within a sentence whether you're describing a real failure or a disguised humblebrag ("my code was so good the review took forever"). Name the actual mistake, a wrong technical assumption, a scope you underestimated, a communication gap with another team, and be specific about what it cost. A missed deadline. A rewrite. An incident.

The answers that land describe a concrete process change, not a restated lesson. "I learned to test more" is forgettable. "I added a staging rollout step because we'd been shipping straight to prod, and that's exactly how a null-pointer regression got through" is a story an interviewer remembers. Have a second example ready too, they sometimes ask whether you've actually applied the lesson since, on a different project.

Be concrete about what you cut, why, and what it cost. "We shipped a prototype without error handling to hit a demo deadline, then paid down the tech debt in sprint 3" is the kind of answer that lands. Vague answers ("I always try to balance both") tell the interviewer nothing and signal that you haven't made a real trade-off consciously.

The follow-up is almost always: "Did that turn out to be the right call?" Have a real answer, even if it's "no, we underestimated the maintenance cost."

The "One Team" cultural question in disguise. Interviewers want to see that you take partial ownership even when the failure was partly someone else's fault, and that you made a structural change to prevent recurrence. "We added a weekly sync" is acceptable. "We added a weekly sync and a shared doc that captured the decision and its rationale" is better because it shows you thought about the failure mode, not just the symptom.

Every company asks this. Nvidia's version is harder because they want specificity about the technical direction, Blackwell architecture, the CUDA ecosystem, the Triton compiler, DGX Cloud, NIM microservices, or whatever the team you're interviewing with actually works on. Generic "AI is the future" answers are transparent and don't work.

Research what the specific team ships. For a CUDA platform role, that's the CUDA toolkit release notes. For an inference role, that's recent TensorRT-LLM changelog and the Triton blog. For a chip architecture role, that's the Blackwell architecture paper from Hot Chips 2024.

Reported from multiple 2025 behavioral rounds. Answer with a real example of a ship-vs-quality call you made, including what you deferred and whether deferring it turned out to be the right call. If you deferred something that later caused an incident, say so, that's a more credible and interesting answer than one where every trade-off worked out perfectly.

Most candidates report 4 to 8 weeks. If you have a referral, that compresses to 3 to 4 weeks in some cases. The longest part is usually waiting between the onsite and the team-matching calls, which can take 1 to 2 weeks after the loop completes.

Mostly medium difficulty, but with a genuine emphasis on correctness, follow-up questions, and the reasoning behind your choices. A handful of reported questions are hard-difficulty (Special Binary String, Basic Calculator III), but the median difficulty in the coding rounds is medium. What separates Nvidia from pure LeetCode shops is the follow-up questioning, a clean medium solution that you can defend at depth is worth more than a stumbling hard solution you can't explain.

It depends heavily on the team. For GPU architecture, CUDA platform, driver, or compiler roles, yes, you'll get detailed CUDA questions and need real hands-on experience. For application-layer SWE roles (developer tools, platform infrastructure, data engineering), the CUDA questions are lighter and more conceptual. Ask your recruiter which category your loop falls into. It's a fair question and they'll tell you.

New grads start with a HackerRank online assessment: 2 to 3 coding problems and 25 multiple-choice questions, 70 to 90 minutes. Topics include C/C++ fundamentals, OS basics, DSA, and probability. Candidates who pass get a recruiter call, then 3 to 4 technical and behavioral rounds. Nvidia reportedly reviews resumes before sending the OA link, so it's not a pure volume-screener. New grad internship conversion rate is 60 to 70% for those who complete a summer internship.

Very important for systems and GPU roles. Interviewers will go deep on memory management, RAII, virtual dispatch, threading primitives, and template mechanics. For ML systems or application-layer roles, Python is acceptable in coding rounds but interviewers often ask C++ conceptual questions anyway. If you're applying to any role with "systems," "compiler," "driver," or "CUDA" in the title, C++ fluency is effectively a prerequisite.

Usually not in full, the new grad loop is more coding and conceptual questions. At senior and staff levels, system design is a full dedicated round (60 minutes). Mid-level loops vary by team. Some teams include a lighter design conversation in the hiring manager round even for junior roles.

Use a HashMap for O(1) key lookups combined with a doubly linked list to track recency order. The list head holds the most recently used entry; the tail holds the least recently used. On a get, move the node to the head. On a put that exceeds capacity, evict the tail node and remove its key from the map.

Nvidia interviewers frequently extend this question into a follow-up about GPU memory caching semantics, or ask how you'd make it thread-safe using a read-write lock. The thread-safety extension is specifically reported from 2025 onsites for systems roles.

from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity: int):
        self.cap = capacity
        self.cache = OrderedDict()

    def get(self, key: int) -> int:
        if key not in self.cache:
            return -1
        self.cache.move_to_end(key)
        return self.cache[key]

    def put(self, key: int, value: int) -> None:
        if key in self.cache:
            self.cache.move_to_end(key)
        self.cache[key] = value
        if len(self.cache) > self.cap:
            self.cache.popitem(last=False)

Maintain a min-heap of size k alongside a hash map of element frequencies. For each element, update its count in the map. If the heap has fewer than k entries, push it. If the element's count exceeds the heap's minimum, pop the minimum and push the new element.

The stream aspect is the real test. Candidates who hard-code a "collect all, then sort" approach get pushed on it immediately. Interviewers want to see you reason about bounded memory and online processing.

import heapq
from collections import defaultdict

def top_k_frequent_stream(stream, k):
    freq = defaultdict(int)
    # min-heap: (count, element)
    heap = []
    seen = set()

    for num in stream:
        freq[num] += 1
        if num not in seen:
            seen.add(num)
            heapq.heappush(heap, (freq[num], num))
        else:
            # rebuild heap with updated freq (simplified approach)
            heap = [(freq[x], x) for x in seen]
            heapq.heapify(heap)

    # return top k
    return [x for _, x in heapq.nlargest(k, heap)]

DFS with a recursion stack (a set of nodes currently on the active call stack) and a visited set. If DFS reaches a node already in the recursion stack, a cycle exists. After finishing DFS from a node, remove it from the recursion stack.

Reported in multiple online assessment write-ups from 2024-2026. Common follow-up: "how does this change for an undirected graph?" The undirected version only needs a visited set and parent tracking, not a separate recursion stack.

def has_cycle(graph):
    visited = set()
    rec_stack = set()

    def dfs(node):
        visited.add(node)
        rec_stack.add(node)
        for neighbor in graph.get(node, []):
            if neighbor not in visited:
                if dfs(neighbor):
                    return True
            elif neighbor in rec_stack:
                return True
        rec_stack.remove(node)
        return False

    for node in graph:
        if node not in visited:
            if dfs(node):
                return True
    return False

Modified binary search. At each step, determine which half of the array is sorted (compare mid to the boundaries). If the target falls within the sorted half, search there. Otherwise, search the other half.

Reported as a phone screen question. Follow-ups often ask about the case with duplicate elements, which breaks the standard approach and requires a fallback to linear search in the worst case (O(n)).

Bottom-up DP table where dp[i][j] represents the length of the longest palindromic subsequence in s[i..j]. If s[i] == s[j], dp[i][j] = dp[i+1][j-1] + 2. Otherwise, dp[i][j] = max(dp[i+1][j], dp[i][j-1]). Fill diagonally from shorter substrings to longer.

Reported in an OA from a candidate in 2024. Bottom-up DP is preferred over memoized recursion because it's easier to talk through under time pressure and avoids stack overhead on long inputs.

Two valid approaches, and Nvidia interviewers want to hear both along with their trade-offs. A min-heap of size k gives O(n log k) time and holds only the k largest elements seen so far, evicting the smallest whenever the heap exceeds size k. Quickselect, a variant of quicksort's partitioning step, gives O(n) average time by recursing into only the half of the array that contains the kth position, but degrades to O(n^2) on adversarial input unless the pivot is randomized.

Reported as a phone screen and OA staple. The follow-up almost always asks which approach fits a streaming input (heap, since quickselect needs the full array in memory) versus a fixed array you're allowed to mutate in place (quickselect, since it skips the heap's log factor).

import random

def find_kth_largest(nums, k):
    target = len(nums) - k  # index if nums were sorted ascending

    def partition(left, right, pivot_index):
        pivot = nums[pivot_index]
        nums[pivot_index], nums[right] = nums[right], nums[pivot_index]
        store_index = left
        for i in range(left, right):
            if nums[i] < pivot:
                nums[store_index], nums[i] = nums[i], nums[store_index]
                store_index += 1
        nums[right], nums[store_index] = nums[store_index], nums[right]
        return store_index

    left, right = 0, len(nums) - 1
    while True:
        pivot_index = random.randint(left, right)
        new_pivot_index = partition(left, right, pivot_index)
        if new_pivot_index == target:
            return nums[new_pivot_index]
        elif new_pivot_index < target:
            left = new_pivot_index + 1
        else:
            right = new_pivot_index - 1

Interviewers present a short snippet and ask you to trace execution line by line: which constructor fires, when does the destructor run, is there a memory leak? Copy construction vs. the assignment operator is a common gotcha. RAII semantics and when a copy constructor gets implicitly called (e.g., passing by value) are the real focus.

The question tests whether you understand the object lifecycle, not just that you can write classes. Interviewers at Nvidia care about whether you know when the destructor runs for stack-allocated vs. heap-allocated objects.

The core requirement: maintain a shared reference count on the heap alongside the managed object. The copy constructor increments the count; the destructor decrements it and frees the object when the count reaches zero. The assignment operator must handle self-assignment and correctly decrement the old referent.

Reported from a 2025 onsite. Tests whether you actually understand RAII vs. whether you've just used std::shared_ptr. Interviewers follow up on circular references, two shared_ptr objects pointing at each other, both with refcount 1, which never reaches zero and leaks. The fix is std::weak_ptr.

Create two semaphores initialized to 1 and 0. Thread A waits on semaphore 1, prints its output, then signals semaphore 2. Thread B waits on semaphore 2, prints its output, then signals semaphore 1. This enforces strict alternation.

Reported from an internship technical round. The problem is easy to state but hard to get exactly right without introducing a race condition. Interviewers watch for incorrect initialization values and for code that signals before printing, which allows the other thread to proceed before the output is actually written.

Each class with virtual functions has a vtable: a static array of function pointers. Each object of that class holds a hidden vptr (pointer to its vtable). Virtual dispatch dereferences the vptr, looks up the function pointer in the vtable, and calls it, adding one indirection and potentially a cache miss compared to a direct call.

Overhead matters in tight GPU driver loops where call frequency is very high. Interviewers also cover lambda captures (closures vs. function pointers), template instantiation (compile-time polymorphism with zero runtime overhead), and move semantics (avoiding unnecessary copies of large buffers).

False sharing happens when two threads write to different variables that happen to sit on the same cache line, typically 64 bytes. The cache coherence protocol treats the whole line as dirty on every write, so it invalidates the other core's copy even though the two threads never touch the same variable. The result looks like a data race in a profiler, cores repeatedly re-fetching a line from a neighbor's cache, but it's really a memory layout problem, not a logic bug.

Fix it by padding hot per-thread counters or state to their own cache line (alignas(64) on the struct), or by reorganizing data so per-thread fields aren't adjacent, structure-of-arrays instead of array-of-structures for the hot fields. Nvidia driver and runtime code hits this constantly, since independent per-thread counters and queue heads are exactly the pattern that triggers it.

#include <atomic>

struct PaddedCounter {
    alignas(64) std::atomic<long> count;
    char padding[64 - sizeof(std::atomic<long>)];
};

PaddedCounter counters[NUM_THREADS]; // one cache line per thread

SIMD applies one instruction to multiple data lanes in strict lockstep, the classic example being CPU vector units like AVX, where there's no real per-lane control flow. SIMT, Nvidia's model, groups 32 threads into a warp and issues one instruction to all of them, but each thread keeps its own program counter and register state conceptually, so threads can take different branches.

That flexibility isn't free. When threads in a warp diverge, the hardware runs both branch paths and masks off the threads that shouldn't be active for each one, which is why the warp divergence question later in this section matters so much. Interviewers tend to ask this one first, as a check on whether you actually know what a warp is before they ask you to optimize around it.

When threads in a warp access contiguous memory addresses, the GPU combines those into a single memory transaction. Non-coalesced access forces multiple transactions and tanks bandwidth, in pathological cases, you can lose 10x or more of available memory bandwidth.

The follow-up is almost always: "How do you restructure a matrix transpose kernel to achieve coalesced access?" The answer: load a tile of the input into shared memory using coalesced reads, then write from shared memory to the transposed output location (which would be non-coalesced from global memory directly).

Shared memory has roughly 100x lower latency than global memory but is limited per thread block, typically 48 to 96 KB on recent GPUs (H100 offers up to 228 KB with dynamic configuration). Load data into shared memory when threads in a block will reuse it multiple times. The access pattern within shared memory also matters: bank conflicts occur when multiple threads in a warp access different addresses that map to the same memory bank, serializing those accesses.

Interviewers want you to connect shared memory decisions to coalescing decisions. They're two sides of the same optimization: coalesced global reads fill shared memory efficiently, and then shared memory provides low-latency reuse for the computation phase.

A warp is 32 threads executed in lockstep on a single streaming multiprocessor. When threads within a warp take different branch paths, the hardware serializes those paths, it executes the true branch with some threads active and the false branch with the remaining threads active, then merges. Both paths run, even the ones that aren't "taken" for a given thread.

Strategies to reduce divergence: restructure data so that threads within a warp process elements of the same type (reducing branch probability), use predicated instructions for short branches where both paths are cheap, and avoid dynamic indexing into shared memory where the index depends on thread ID in a non-uniform way.

Nvidia Nsight Systems for timeline analysis: CPU-GPU interaction, kernel launch overhead, synchronization points, and data transfer timing. Nsight Compute for kernel-level profiling: warp efficiency, memory throughput, achieved occupancy, and instruction mix. The distinction matters, Nsight Systems answers "where is time being spent at the application level?" while Nsight Compute answers "why is this specific kernel slow?"

The candidate account that reported this said the interviewer specifically asked when you'd use each tool and followed up by asking how you'd instrument a multi-GPU training run. For that case, the answer involves NCCL profiling hooks and per-rank timeline correlation in Nsight Systems.

GPU virtual memory via CUDA Unified Virtual Memory (UVM) allows CPU and GPU to share an address space. The cost: page faults trigger expensive migrations between host and device memory, with each fault adding microseconds of latency. For workloads that access memory in predictable patterns, pinned (page-locked) host memory with explicit cudaMemcpy transfers is almost always faster than relying on UVM page migration.

Interviewers want you to know when UVM is acceptable (prototyping, irregular access patterns that are too complex to manage manually, very large datasets that exceed GPU memory) vs. when it's a performance liability (hot-path inference, latency-sensitive kernels).

Context switches between kernel and user mode carry overhead, typically in the hundreds of nanoseconds range. For GPU work, this surfaces in driver calls, every cudaMalloc, every kernel launch, every cudaMemcpy crosses the kernel-user boundary. Minimizing cudaMalloc frequency (pre-allocate a pool, reuse buffers), batching kernel launches, and using CUDA streams to overlap compute with data transfers all reduce this overhead.

This comes up for driver and compiler roles more than for application-layer roles, but Nvidia expects any systems candidate to know it.

The default stream serializes everything, a kernel launch and a memory copy issued on it run one after the other even if they don't depend on each other. Non-default streams change that. Issue a kernel on stream A and a cudaMemcpyAsync on stream B, and modern GPUs, which have separate copy engines from their compute engines, run them concurrently.

The catch that trips people up: this only works with pinned (page-locked) host memory, allocated via cudaMallocHost or cudaHostAlloc. Pageable memory can't be transferred asynchronously, the driver stages it through a pinned buffer first, which silently serializes your "concurrent" transfer. Interviewers ask this as a follow-up almost every time, it's the detail that separates someone who's read the CUDA programming guide from someone who's actually pipelined a real workload.

Layer fusion combines adjacent operations (e.g., conv + batch norm + ReLU) into a single kernel, eliminating intermediate memory writes and reads. Quantization (INT8 or FP8 for inference where precision tolerance allows) reduces memory bandwidth and increases throughput on hardware with dedicated low-precision units. Kernel auto-tuning via TensorRT finds the best kernel implementation for a given input shape. Batching requests amortizes kernel launch overhead and maximizes GPU utilization between calls. For the serving layer, Triton Inference Server comes up frequently in reported questions.

Interviewers at Nvidia want you to connect these optimizations to hardware realities, not just recite a checklist. "Layer fusion reduces memory round-trips because each intermediate tensor otherwise sits in global memory between operations" is the level of reasoning they're looking for.

Be specific about the timeline, how you ramped (documentation? colleagues? experiments?), what you got wrong first, and what the outcome was. Nvidia engineers work on emerging hardware and software stacks where the documentation is sometimes a week old or nonexistent. The question tests whether you can self-direct learning in novel territory, not whether you've memorized existing frameworks.

Interviewers follow up by asking what resources were most useful and what you'd do differently if you had to do it again. Vague answers ("I read the docs and asked questions") don't land here.

The "intellectual honesty" value in practice. Interviewers want to see that you can hold a position with evidence and change your mind with evidence. Pure deference to seniority is a red flag. So is stubbornness that ignores new information when it arrives. The strongest answers describe a specific technical disagreement, the evidence you brought to the conversation, what the outcome was, and whether, in retrospect, the decision was correct.

This question frequently turns into a 20-minute technical discussion about the actual trade-off. Prepare your story with the same technical depth you'd bring to a system design question.

Strong answers name the actual tool you used to find it, perf, Nsight, flamegraph, custom instrumentation, a printf at the right place. Name what made it hard to see (it only appeared under production traffic patterns, it was masked by another bottleneck, the tooling didn't expose it directly). And give the actual size of the win.

Round numbers ("it was 2x faster") get follow-up questions about how you measured. If you don't have a rigorous measurement story, you'll get pressed on it. Nvidia engineers profile things; they don't estimate.

Real answer structure: first, identify what decision is actually irreversible vs. reversible, irreversible decisions (architectural choices, API contracts) warrant more information-gathering before committing; reversible decisions should be made faster. Second, bound your uncertainty with a quick experiment or proxy measurement rather than waiting for complete data. Third, commit with an explicit trigger to revisit (a date, a metric threshold, a milestone).

Vague answers ("I gather as much data as possible") don't land well here. Nvidia is building hardware on 3-year cycles, every decision at some level is made with incomplete information. They want to know your actual process.

For a sequence of length n and hidden dimension d, computing the attention scores QK^T costs O(n^2 * d), and the attention matrix itself takes O(n^2) memory. Double the context length and you quadruple the compute and memory cost of the attention block specifically, the feed-forward layers stay linear in n. That quadratic wall is why FlashAttention exists, it fuses the softmax and the two matmuls so the full n x n matrix never gets materialized in HBM, only computed in tiles that fit in on-chip SRAM.

Interviewers often connect this straight to the KV-cache question below. During generation, each new token only attends against the cached keys and values, so the quadratic cost gets paid once during prefill, not per token during decode. Candidates who conflate the two, and claim generation itself is quadratic per token, get corrected fast.

In autoregressive generation, each new token attends to all previous tokens. Without a cache, you'd recompute the key and value tensors for every previous token at every decoding step, O(n^2) total compute for a sequence of length n. KV-cache stores those key and value tensors after they're computed, so each decoding step only computes attention for the new token against the cached context.

The cost: memory grows linearly with sequence length and batch size, which is why PagedAttention (vLLM) and sliding window attention (Mistral) exist. Reported from a phone screen for an LLM-focused role. Interviewers often follow up by asking how you'd manage KV-cache memory when multiple requests have different sequence lengths in the same batch.

LoRA freezes base model weights and trains small rank-decomposed matrices injected into the attention layers. The update to a weight matrix W is expressed as W + AB, where A is (d x r) and B is (r x d), and r is much smaller than d. Much lower memory footprint than full fine-tuning, at a cost of some expressiveness for tasks that require large distributional shifts from the base model.

Used when you have limited GPU budget, when you need fast task-switching between many adapters (LoRA multiplexing in production serving), or when the base model is already close to the target task. Full fine-tuning is worth the cost when LoRA's rank limitation actually constrains performance, which is task-dependent and usually determined empirically.

Standard backprop keeps every intermediate activation from the forward pass around for the backward pass, so memory scales with layer count times batch size times sequence length. Gradient checkpointing throws most of that away and recomputes it on demand during the backward pass instead, trading roughly 20 to 30% more compute time for activation memory that's often the difference between a model fitting on your GPU budget and not fitting at all.

You reach for it when you're memory-bound rather than compute-bound, which describes most large transformer training runs on a fixed cluster. Interviewers sometimes push on placement, checkpointing every transformer block is the standard granularity. Checkpoint too finely and the recomputation overhead eats the memory savings you were trying to get.

This comes up in senior loops. The expected approach uses atomic compare-and-swap operations on a free-list head pointer, with careful alignment to cache line boundaries. Pre-allocate a large contiguous slab and track free blocks with a lock-free stack of block pointers using std::atomic.

Interviewers care about the reasoning as much as the code. "Why lock-free?" is always a follow-up. The expected answer: mutex contention at GPU driver call frequency creates measurable latency spikes that a lock-free approach avoids, at the cost of more complex ABA-problem handling.

#include <atomic>
#include <cstddef>
#include <cstdlib>

struct Block {
    Block* next;
};

class LockFreePool {
    std::atomic<Block*> head;

public:
    LockFreePool() : head(nullptr) {}

    void* allocate(size_t size) {
        Block* old_head = head.load(std::memory_order_relaxed);
        while (old_head) {
            if (head.compare_exchange_weak(old_head, old_head->next,
                std::memory_order_acquire, std::memory_order_relaxed)) {
                return old_head;
            }
        }
        return std::aligned_alloc(64, size); // fallback, 64-byte aligned
    }

    void deallocate(void* ptr) {
        Block* block = static_cast<Block*>(ptr);
        Block* old_head = head.load(std::memory_order_relaxed);
        do {
            block->next = old_head;
        } while (!head.compare_exchange_weak(old_head, block,
                 std::memory_order_release, std::memory_order_relaxed));
    }
};

Start with Nsight Compute to get a kernel-level profile: achieved occupancy, memory throughput, warp efficiency, and instruction-level breakdown. Then reason through the likely causes in priority order. Uncoalesced memory access is the most common culprit. Bank conflicts in shared memory are second. Thread divergence (branches within a warp) serializes execution and reduces effective throughput. Insufficient parallelism means the GPU has too few active warps to hide memory latency, check that your grid and block dimensions are appropriate for the problem size.

Reported from a senior systems role onsite in 2025. The interviewer wanted to hear Nsight Compute named specifically and wanted the candidate to distinguish between Nsight Systems (timeline, CPU-GPU interaction) and Nsight Compute (kernel internals). Using the wrong tool for the question is a red flag.

Start with INT8 or FP8 quantization for weights and activations where the model's accuracy tolerance allows, this cuts both memory bandwidth and compute. Apply kernel fusion via TensorRT or torch.compile to collapse attention and feed-forward layers. Use FlashAttention for the attention layers, which reduces HBM reads and writes for the attention matrix. For the KV cache, PagedAttention (as in vLLM) reduces fragmentation and allows larger effective batch sizes. For very long sequences, speculative decoding with a smaller draft model can improve tokens-per-second at the cost of some implementation complexity.

Reported from a 2025 ML systems onsite. The interviewer asked follow-up questions specifically about the latency-throughput trade-off in each choice, quantization helps latency but can hurt accuracy on certain tasks; larger batches improve throughput but increase per-request latency.

MoE replaces each dense feed-forward block with multiple specialist sub-networks ("experts"), with a gating network routing each token to a subset (typically top-2). Compute per token stays constant even as total parameter count grows, Mixtral 8x7B has 47B total parameters but activates roughly 13B per token. This gives MoE models a favorable FLOPs-per-quality ratio at training time.

Trade-offs: load imbalance across experts (some experts get routed to heavily, others sit idle), which requires auxiliary load-balancing losses during training. Increased communication overhead in distributed training, since experts may live on different devices and tokens need to be routed across the network. Serving complexity increases because some experts may be "cold" (not resident in GPU memory) if you're serving many adapters on one cluster.

Data parallelism replicates the full model on each GPU and shards the data. Simple to implement, but requires an all-reduce at each gradient step, as model size grows, this communication becomes the bottleneck. Model parallelism shards the model layers across GPUs, reducing per-GPU memory at the cost of communication for activations between layers. Pipeline parallelism assigns each GPU one stage of the computation and enables asynchronous micro-batching, but introduces pipeline bubbles (GPUs sitting idle waiting for the previous stage) that hurt GPU utilization.

In practice, Nvidia's Megatron-LM and Microsoft DeepSpeed use all three simultaneously (3D parallelism), with tensor parallelism within a node (NVLink), pipeline parallelism across nodes (InfiniBand), and data parallelism across replica groups. The optimal split depends on the model architecture and hardware topology.

Two heaps: a max-heap for the lower half of values, a min-heap for the upper half. On each insertion, add to the appropriate heap based on whether the value is above or below the current median, then rebalance if the two heaps differ in size by more than one. The median is either the top of the larger heap, or the average of both tops if they're equal in size.

Reported from a 2025 onsite. The GPU-specific framing, "kernel execution times arriving in real-time", is unusual but the underlying pattern is the classic sliding window median. The insertion is effectively O(log n), not O(1), but the question is testing whether you know this approach at all and can implement it correctly.

Model GPU memory and compute as first-class constrained resources, not just CPU slots. Cover preemption strategy (checkpoint-and-restore vs. kill-and-resubmit, the latter being cheaper but harsher), priority queuing with aging to prevent starvation, and gang scheduling for jobs that need multiple GPUs simultaneously, a job requiring 8 GPUs must wait until all 8 are available, or fragment-and-stall becomes a real problem.

Interviewers push hard on the failure case: what happens when a job hangs? You need a watchdog process, a heartbeat from running jobs, and a kill mechanism that releases GPU memory even if the job process is unresponsive (which means you need a separate out-of-process memory manager, not just cleanup in the job itself).

Model sharding strategy first: tensor parallelism within a node (using NVLink for high-bandwidth inter-GPU communication), pipeline parallelism across nodes (using InfiniBand). Load balancing must be GPU-memory-aware, not just CPU-load-aware, a node with high CPU but low GPU memory utilization is still a valid target. Request batching with a configurable max wait time (e.g., 5ms) amortizes kernel launch overhead while keeping tail latency bounded.

At senior level, interviewers expect you to name TensorRT-LLM or vLLM as existing solutions and know their trade-offs. TensorRT-LLM has better raw latency for fixed-shape inputs; vLLM has better throughput for variable-length sequences via PagedAttention. Reported from a 2025 onsite.

Parallelism strategy: tensor parallelism within a node using NVLink (8 GPUs per node), pipeline parallelism across nodes using InfiniBand. Checkpoint frequency is a trade-off: more frequent checkpoints reduce wasted compute on node failure but add synchronization overhead and storage I/O. With 128 GPUs, a node failure every few hours is realistic, plan for it. NCCL handles gradient reduction. ZeRO-3 optimizer state sharding (from DeepSpeed) spreads the optimizer states, gradients, and parameters across GPUs to fit larger models than any single GPU could hold.

Interviewers want to see that you can reason about the failure probability of 128 machines over a multi-day training run, not just describe the happy path.

Key design decisions: in-process memory vs. Redis vs. GPU-local caching. For sub-millisecond latency, in-process caching is usually necessary, a Redis round-trip adds 0.5 to 2ms of network latency even on a fast LAN, which violates the constraint. Cache key design for model inputs is hard: exact-match keys work for deterministic inputs, but near-duplicate inputs (semantically similar prompts) won't hit. Semantic caching using embedding similarity adds latency that may negate the cache benefit. Eviction policy matters: LRU works well for temporal locality in request patterns; LFU works better for long-tail distributions where some inputs recur at very high frequency.

Invalidation when model weights update requires either a versioned cache key (include model version in the key) or a cache flush, the versioned approach is better for gradual rollouts.