Scale AI Interview Questions (2026): What They Actually Ask

Use a Counter or a dict to tally character frequencies, then sort by frequency descending (stable sort keeps alphabetical order for ties). Return the sorted character list.

Reported as an early warm-up question in Scale AI's technical screen. Trivial with Python's Counter but interviewers follow up by asking how you'd handle a stream of characters where the full word isn't known upfront, which points toward a heap-based approach for the top-k version of the problem.

Sort the sessions by start time. Walk through the sorted list once, keeping a running merged interval. If the next session's start is less than or equal to the current merged interval's end, extend the end to cover it. Otherwise close out the current interval, add its length to the total, and start a new one. The final total is the sum of the merged interval lengths, not the sum of raw session lengths, since overlapping sessions would otherwise get counted twice.

Reported as a warm-up problem on Scale AI's HackerRank assessment in 2025. The bug most candidates ship is using strict inequality when checking overlap, which fails on sessions that touch exactly at the boundary. Ask the interviewer whether a session ending at 2:00pm and one starting at 2:00pm count as overlapping before you write the comparison.

def total_active_time(sessions):
    """
    sessions: list of (start, end) tuples, unsorted
    Returns total merged active time.
    """
    if not sessions:
        return 0

    sessions = sorted(sessions, key=lambda s: s[0])
    total = 0
    cur_start, cur_end = sessions[0]

    for start, end in sessions[1:]:
        if start <= cur_end:
            cur_end = max(cur_end, end)
        else:
            total += cur_end - cur_start
            cur_start, cur_end = start, end

    total += cur_end - cur_start
    return total

Since the scores are monotonically non-increasing, binary search directly for the boundary instead of scanning linearly. On each step, check the midpoint score. If it's still above the threshold, the answer lies to the right; if it's at or below the threshold, the answer could be the midpoint or something to the left, so narrow the search there. The loop ends when the search window collapses to the boundary index.

Reported as a quick technical screen question in 2024 and 2025, usually solved in under ten minutes so interviewers can spend the rest of the round on what you'd do operationally once you find that day, like whether to freeze the affected batch or re-route it for review.

def first_day_below_threshold(scores, threshold):
    lo, hi = 0, len(scores) - 1
    result = -1

    while lo <= hi:
        mid = (lo + hi) // 2
        if scores[mid] <= threshold:
            result = mid
            hi = mid - 1
        else:
            lo = mid + 1

    return result

If policy violations show up in roughly 0.5 percent of content, a classifier that predicts "not a violation" on everything scores 99.5 percent accuracy while catching nothing. Accuracy is the wrong metric whenever classes are this imbalanced.

Precision tells you, of everything flagged as a violation, how much actually was one; recall tells you, of all real violations, how many got caught. Which one matters more depends on the cost of each error type. For a moderation system where missing a real violation is worse than a false alarm, optimize for recall and accept lower precision. For a system where wrongly flagging legitimate content damages user trust, weight precision higher. F1 is the harmonic mean of the two and is a reasonable single number when you don't have a strong reason to favor one error type over the other.

Don't be modest here. This question is genuinely asking for your best work, and interviewers have calibrated expectations. A vague "I led a migration" answer will prompt follow-up questions: migration of what, how big, what broke, what was the measured outcome? Prepare to go technical quickly. The behavioral round at Scale AI is more technically substantive than behavioral rounds at most other companies. Your story should have a real number in it: latency reduced by X%, cost cut by $Y per month, labeling throughput improved from Z tasks/hour to Z' tasks/hour.

Most candidates report three to five weeks. The process is tighter than comparably sized companies. Referrals or warm introductions sometimes compress this to two to three weeks. The longest wait is typically between the virtual onsite and the offer decision, which usually comes within five to seven business days. Scale AI moves faster than most companies on post-onsite decisions.

Python is strongly preferred and effectively required for ML engineer roles. For software engineering roles, Python is the dominant language in the coding rounds, though some candidates report using TypeScript or Go for backend-focused roles. The debugging round typically uses Python or TypeScript. If Python is not your primary language, practice the standard data structures and the standard library (collections.deque, heapq, itertools) in Python specifically before your interview.

New grad loops typically include the HackerRank online assessment, a recruiter screen, a hiring manager call, and two to three onsite rounds covering coding and behavioral. The system design round is often lighter or absent for new grads. Senior and staff-level loops add a full system design round (the AI-specific pipeline design questions), may include a take-home for ML roles, and feature a more technically substantive Credo round. At senior levels, the Credo questions frequently turn into extended technical discussions about the specific decisions described in your stories.

More important than at most companies. Scale AI's engineering blog and recruiter conversations consistently indicate that even software engineers who aren't writing ML code work closely enough with annotation pipelines, evaluation infrastructure, and data quality systems that ML fluency matters. Expect conceptual questions about model training, evaluation metrics, and data quality challenges in the hiring manager screen even for backend SWE roles. You don't need to have trained a model, but you do need to understand why data quality affects model quality and what metrics measure it.

Different, not uniformly harder. Anthropic and OpenAI skew toward deeper ML research questions and more theoretical depth. Scale AI's process is harder on applied engineering and practical pipeline design, and the Credo round is more pointed than most behavioral interviews. The debugging round has no direct equivalent at most other AI companies. Candidates who do well at pure algorithm interviews sometimes struggle at Scale AI specifically because of the domain-specific framing and the debugging requirement. Candidates who have real data pipeline experience often find Scale's interviews map closely to actual work they've done.

Mostly medium difficulty. The coding problems have a practical framing tied to annotation pipelines and data stream processing, which makes them feel different from abstract LeetCode mediums even when the underlying pattern is the same. Hard-difficulty algorithmic problems are occasionally reported but are not the norm. What makes Scale's coding rounds challenging is the implementation round (card game or equivalent) and the debugging round, neither of which maps to standard LeetCode preparation.

The classic approach uses three pointers: prev, curr, and next. You walk the list once, and at each node you flip curr.next to point backward at prev, then advance all three pointers forward. When curr becomes null, prev is the new head.

def reverse_list(head):
    prev = None
    curr = head
    while curr:
        nxt = curr.next
        curr.next = prev
        prev = curr
        curr = nxt
    return prev

This runs in O(n) time and O(1) space, you're just rewiring pointers, no extra data structure needed. A recursive version exists too, but it costs O(n) stack space, which isn't what I'd want for a list that could hold tens of thousands of nodes.

Watch the edge cases: an empty list (head is None, just return None) and a single-node list (the loop runs once, prev ends up pointing at that node with next set to None, which is correct). If it's a doubly linked list, you also need to swap the prev and next pointers on every node, not just next.

Push opening brackets onto a stack. When you hit a closing bracket, check that the stack isn't empty and that the top of the stack is the matching opener, if either check fails, the string is invalid. Pop the matching opener off the stack. At the end, if the stack is empty, every open bracket got closed correctly.

def is_balanced(s):
    pairs = {')': '(', ']': '[', '}': '{'}
    stack = []
    for ch in s:
        if ch in '([{':
            stack.append(ch)
        elif ch in pairs:
            if not stack or stack.pop() != pairs[ch]:
                return False
    return not stack

This is O(n) time and O(n) space in the worst case, like a string that's all opening brackets. The detail people miss is checking that the stack isn't empty before popping. Skip that guard and a string that starts with a closing bracket either throws an index error or gets handled inconsistently depending on the language. Also remember to check that the stack is empty at the end, a string with leftover unmatched openers is invalid even if every closing bracket you saw along the way had a match.

A process is an independent unit of execution with its own memory address space, file descriptors, and OS-level resources. A thread is a unit of execution that lives inside a process and shares that process's memory with any other threads in the same process. Spawning a new process is expensive, the OS has to allocate a fresh address space and set up new resource tables, while spawning a thread is cheap since it just needs a new stack and register set within memory that already exists.

Because threads share memory, communication between them is fast, just read or write a shared variable, but it's also risky. You need locks, mutexes, or other synchronization to stop two threads from stepping on the same data at the same time. Processes talk to each other through IPC mechanisms like pipes, sockets, or shared memory segments, which is slower but safer, since one process crashing doesn't corrupt another process's memory.

I'd reach for multiple processes when I want fault isolation, for example running independent labeling-worker instances where one crashing doesn't take the others down with it, or when I need to get around a language runtime's global interpreter lock for CPU-bound work. I'd reach for threads when the work is I/O bound within a single service, like a server handling many concurrent requests that mostly wait on network calls, where the shared memory and lower overhead pay off and there's no CPU-bound bottleneck to work around.

An INNER JOIN returns only the rows where the join condition matches in both tables. A LEFT JOIN returns every row from the left table and fills in NULLs for the right table's columns wherever there's no match.

SELECT l.name, COUNT(t.id) AS task_count
FROM labelers l
LEFT JOIN tasks t ON t.labeler_id = l.id
GROUP BY l.name;

Say you have a labelers table and a tasks table joined on labeler_id. Swap that LEFT JOIN for an INNER JOIN and you'd only get labelers who have at least one task, anyone sitting idle with zero assigned tasks disappears from the result entirely.

That distinction matters most in reporting. If you want a utilization report that includes idle labelers with a task count of zero, an INNER JOIN silently drops them and undercounts your workforce. You need the LEFT JOIN so every labeler row survives, with COUNT correctly returning zero instead of a missing row.

Classic k-way merge using a min-heap. Each element in the heap stores the current value, the worker index it came from, and the position within that worker's stream. Pop the minimum, output it, then push the next element from that same worker. Time complexity is O(n log k) where n is total elements across all streams and k is the number of workers.

Reported from Scale AI online assessments in 2024-2025. The stream framing is the tell: candidates who try to collect all values first then sort them get immediately challenged on memory usage and latency. The heap approach works because it processes elements online as they arrive.

import heapq

def merge_sorted_streams(streams):
    """
    streams: list of iterators, each yielding sorted values
    Returns a single sorted iterator over all values.
    """
    heap = []
    # Initialize heap with first element from each stream
    for i, stream in enumerate(streams):
        try:
            val = next(stream)
            heapq.heappush(heap, (val, i, stream))
        except StopIteration:
            pass

    while heap:
        val, idx, stream = heapq.heappop(heap)
        yield val
        try:
            next_val = next(stream)
            heapq.heappush(heap, (next_val, idx, stream))
        except StopIteration:
            pass

Use a deque to maintain the sliding window of the last N values. On each new element, update the window, compute the current mean and standard deviation, and check if the new element falls outside mean +/- 2 * std. Computing std from scratch each iteration is O(window_size). A running sum and running sum-of-squares lets you compute mean and variance in O(1) per step instead.

Reported from Scale AI coding rounds in 2025. The O(1) running stats optimization is the follow-up interviewers typically push for. If you implement the naive approach first, be ready to explain how you'd improve it. The question is testing whether you understand that annotation pipelines process data continuously, not in static batches.

from collections import deque
import math

def detect_anomalies(stream, window_size, threshold=2.0):
    window = deque()
    running_sum = 0.0
    running_sum_sq = 0.0
    anomalies = []

    for i, val in enumerate(stream):
        window.append(val)
        running_sum += val
        running_sum_sq += val * val

        if len(window) > window_size:
            removed = window.popleft()
            running_sum -= removed
            running_sum_sq -= removed * removed

        n = len(window)
        if n < 2:
            continue

        mean = running_sum / n
        variance = (running_sum_sq / n) - (mean ** 2)
        std = math.sqrt(max(variance, 0))

        if std > 0 and abs(val - mean) > threshold * std:
            anomalies.append((i, val, mean, std))

    return anomalies

Greedy assignment using a min-heap keyed by current load. For each incoming task, pop the least-loaded worker from the heap, assign the task, update the worker's load, and push it back. This gives each task to the currently least-busy worker, minimizing the maximum load across all workers.

Reported in Scale AI interview prep guides citing 2025 onsite experiences. The real follow-up is about what "load" means: is it task count, estimated completion time, or a quality-weighted score? Interviewers want to see you ask that clarifying question before implementing.

import heapq

def assign_tasks(tasks, num_workers):
    """
    tasks: list of task durations
    Returns list of (task_id, worker_id) assignments.
    """
    # min-heap: (current_load, worker_id)
    heap = [(0, i) for i in range(num_workers)]
    heapq.heapify(heap)
    assignments = []

    for task_id, duration in enumerate(tasks):
        load, worker_id = heapq.heappop(heap)
        assignments.append((task_id, worker_id))
        heapq.heappush(heap, (load + duration, worker_id))

    return assignments

Maintain a max-heap keyed by priority. At each time step, push all jobs whose release time has passed, then pop and execute the highest-priority job. If jobs also have deadlines, maintain a deadline tracker and skip or fail jobs that can no longer meet their deadline.

Reported from the applied engineering round in 2024 and 2025. The real complexity is in the state machine: a job can be PENDING, READY, RUNNING, DONE, or EXPIRED. Interviewers watch for candidates who forget to handle the EXPIRED state, which is what happens in a real annotation pipeline when a labeler holds a task too long.

This is cycle detection on a directed graph, solvable with Kahn's algorithm. Build an adjacency list and an in-degree count for each task. Push every task with in-degree zero onto a queue, then repeatedly pop a task, decrement the in-degree of its dependents, and push any dependent that reaches zero. If every task gets processed by the end, a valid order exists. If any task never reaches in-degree zero, there's a cycle and no valid order is possible.

Reported in 2025 SWE onsite loops, usually as a pivot question right after the stream merge or heap problems. Interviewers use it to see whether a candidate can move from array and heap thinking to graph thinking without re-deriving the algorithm from first principles. Mention the connection to build systems or dependency resolution and interviewers tend to move on faster.

from collections import deque, defaultdict

def can_complete_all_tasks(num_tasks, dependencies):
    """
    dependencies: list of (task, depends_on) pairs
    Returns True if all tasks can be ordered without a cycle.
    """
    graph = defaultdict(list)
    in_degree = [0] * num_tasks

    for task, depends_on in dependencies:
        graph[depends_on].append(task)
        in_degree[task] += 1

    queue = deque(t for t in range(num_tasks) if in_degree[t] == 0)
    processed = 0

    while queue:
        node = queue.popleft()
        processed += 1
        for neighbor in graph[node]:
            in_degree[neighbor] -= 1
            if in_degree[neighbor] == 0:
                queue.append(neighbor)

    return processed == num_tasks

The specific game changes, but the pattern is consistent: a multi-rule specification, multiple state transitions, and scoring logic. Spend the first 8 to 10 minutes reading and mapping the spec to classes and methods before writing anything. Candidates who start coding immediately consistently fail to handle edge cases buried in the spec's later sections.

The canonical structure: a Deck class that shuffles and deals, a Card class with suit/value, a Player class that holds a hand, and a Game class that manages state transitions and enforces the rules. The Game class is where the spec complexity lands. Build it incrementally and test each rule as you implement it rather than writing everything first.

What interviewers are watching for is not perfect code. They're watching whether you can translate a dense written spec into working software while narrating your reasoning. The thinking-aloud piece is as important as the code here.

import random
from enum import Enum
from dataclasses import dataclass, field
from typing import List, Optional

class Suit(Enum):
    HEARTS = "Hearts"
    DIAMONDS = "Diamonds"
    CLUBS = "Clubs"
    SPADES = "Spades"

@dataclass
class Card:
    suit: Suit
    value: int  # 1=Ace, 11=Jack, 12=Queen, 13=King

    def display_value(self) -> str:
        names = {1: "A", 11: "J", 12: "Q", 13: "K"}
        return names.get(self.value, str(self.value))

    def __repr__(self):
        return f"{self.display_value()}{self.suit.value[0]}"

class Deck:
    def __init__(self):
        self.cards = [
            Card(suit, val)
            for suit in Suit
            for val in range(1, 14)
        ]
        random.shuffle(self.cards)

    def deal(self, n: int) -> List[Card]:
        if len(self.cards) < n:
            raise ValueError("Not enough cards to deal")
        dealt = self.cards[:n]
        self.cards = self.cards[n:]
        return dealt

@dataclass
class Player:
    name: str
    hand: List[Card] = field(default_factory=list)
    score: int = 0

    def receive(self, cards: List[Card]):
        self.hand.extend(cards)

    def play_card(self, index: int) -> Card:
        return self.hand.pop(index)

class Game:
    """
    Skeleton: fill in game-specific rules from the spec.
    The key is getting this structure right first,
    then layering spec rules on top.
    """
    def __init__(self, player_names: List[str], cards_per_hand: int = 5):
        self.deck = Deck()
        self.players = [Player(name) for name in player_names]
        self.current_round = 0
        self._deal_hands(cards_per_hand)

    def _deal_hands(self, n: int):
        for player in self.players:
            player.receive(self.deck.deal(n))

    def play_round(self):
        # Implement spec-specific round logic here
        self.current_round += 1

    def winner(self) -> Optional[Player]:
        return max(self.players, key=lambda p: p.score)

Don't read top to bottom. Start with the function signatures and docstrings to understand the intended behavior, then look at the test cases (if provided) to see what the code is supposed to produce. Run the code if you can, and look at where the output diverges from the expected output. Work backward from the divergence point.

For logical bugs specifically, the most common patterns are: off-by-one errors in loop bounds or slice indices, incorrect comparison operators (greater than vs. greater than or equal), state that isn't reset between iterations, and mutation of a container while iterating over it. Scale AI's debugging problems tend to involve annotation pipeline logic or stream processing code, so also watch for index tracking bugs and incorrect handling of empty input.

Reported across multiple 2024-2025 candidate accounts on Blind. One candidate noted they finished only one of two bugs and still got an offer because they articulated exactly where the second bug was and how they would fix it. The explanation counted as much as the fix.

Start by checking the formula implementation against the spec. Cohen's Kappa, the standard inter-annotator metric, is (Po - Pe) / (1 - Pe), where Po is observed agreement and Pe is expected agreement by chance. A common bug is computing Pe incorrectly by summing class proportions instead of summing the products of marginal probabilities for each class. Another common bug is dividing by the wrong denominator in Po when the annotation matrix is non-square.

Add print statements or assertions at each intermediate step: check Po, check Pe, check the final formula. If Pe is inflated, kappa will be lower than expected. If Pe is deflated (the more common bug), kappa will be higher than expected, which matches the symptom described. This question tests whether you know enough about the domain to know where to look.

Cohen's Kappa measures agreement between exactly two raters, corrected for agreement by chance. Formula: (Po - Pe) / (1 - Pe). A Kappa of 0.6 to 0.8 is typically considered "substantial agreement"; above 0.8 is "near perfect." For annotation tasks at Scale AI, Kappa below 0.4 usually means the task instructions are ambiguous, not that the annotators are bad.

Fleiss' Kappa extends this to multiple raters. Use it when each task gets routed to more than two annotators. Weighted Kappa (for ordinal scales, e.g., quality ratings from 1 to 5) penalizes large disagreements more than small ones, which is appropriate for preference scoring in RLHF tasks.

Reported as a direct question in Scale AI ML engineer technical screens in 2025. Interviewers want to hear both the formula and the practical interpretation: what does it mean in the context of a labeling job when Kappa drops from 0.7 to 0.4 overnight?

Active learning selects which unlabeled examples to annotate next, prioritizing examples where the current model is most uncertain. This reduces labeling costs by focusing human effort on the most informative data points rather than labeling randomly.

Three uncertainty sampling strategies: least confidence (label the example the model assigns the lowest max class probability to), margin sampling (minimize the gap between the top two class probabilities), and entropy sampling (maximize the entropy of the predicted class distribution). Entropy sampling is generally preferred because it accounts for uncertainty across all classes, not just the top one.

In a real labeling pipeline, this means the model runs inference on a batch of unlabeled candidates, scores each by uncertainty, and routes the highest-uncertainty examples to human annotators. The pipeline then re-trains on the new labels and repeats. Interviewers follow up by asking how you'd prevent the model from fixating on a particular region of input space (diversity constraints: ensure the selected batch covers diverse examples, not just the single hardest cluster).

import numpy as np
from typing import List

def entropy_sampling(probs: np.ndarray, n_samples: int) -> List[int]:
    """
    probs: shape (n_examples, n_classes), probability distributions
    Returns indices of the n_samples most uncertain examples.
    """
    # Clip to avoid log(0)
    probs = np.clip(probs, 1e-10, 1.0)
    entropy = -np.sum(probs * np.log(probs), axis=1)
    # Return indices sorted by entropy, highest first
    ranked = np.argsort(entropy)[::-1]
    return ranked[:n_samples].tolist()

def margin_sampling(probs: np.ndarray, n_samples: int) -> List[int]:
    """
    Returns indices with the smallest gap between top two class probs.
    """
    sorted_probs = np.sort(probs, axis=1)[:, ::-1]
    margins = sorted_probs[:, 0] - sorted_probs[:, 1]
    ranked = np.argsort(margins)  # smallest margin = most uncertain
    return ranked[:n_samples].tolist()

From a data platform perspective, RLHF is primarily a data quality and throughput problem. The model research team cares about the training algorithm. You care about: how do you collect enough high-quality preference data, fast enough, with enough consistency between annotators, to actually train a good reward model?

The preference collection pipeline generates pairs of model outputs per prompt. Human annotators compare them and pick the better one. Noisy preferences (annotators who disagree with each other, annotators who are inconsistent day-to-day, adversarial annotators who click randomly) degrade the reward model's quality more than people expect. A reward model trained on 10K high-quality preferences often outperforms one trained on 100K noisy preferences.

The downstream effect: if your data platform produces high-kappa, low-bias preference data, the model team's PPO or DPO training run works. If your data platform produces inconsistent data, no algorithm fixes it. This is the frame Scale AI interviewers want to see when they ask about RLHF, not the RL theory.

Three types of shift matter here: covariate shift (the distribution of input examples changes), label shift (the distribution of labels changes, which can happen when annotator populations change), and concept drift (the underlying annotation criteria change, often due to guideline updates).

Detection: monitor the KL divergence between the current batch's feature distribution and the historical baseline. Track per-class label frequencies over time. Run periodic calibration tasks (tasks with known correct labels) and track annotator accuracy against those ground-truth examples. A sudden drop in calibration accuracy with no change in annotator Kappa usually signals concept drift, not annotator quality degradation.

Response: flag drift early and escalate to the guidelines team before the model is retrained on the shifted data. Reweighting by importance weights can partially correct for covariate shift if you can quantify it. Label shift requires retraining the classifier head.

RAG augments a language model's generation with retrieved documents, pulling relevant context from a knowledge base at inference time rather than relying solely on the model's parametric memory. The quality of the knowledge base directly limits RAG output quality, which makes data quality a first-class problem, and is why Scale AI's ML engineer interviews specifically cover this.

Building a high-quality RAG knowledge base requires: document ingestion and chunking (split documents into semantically coherent chunks, not fixed-size windows), embedding generation (using a sentence encoder to produce dense vectors for each chunk), relevance annotation (having human annotators judge whether retrieved chunks are actually relevant to given queries), and iterative refinement (retrain the retriever on the annotated relevance judgments). The annotation task here is essentially building a relevance dataset: (query, chunk) pairs labeled as relevant or not relevant. This is a Scale AI problem domain.

Bias is the error from a model being too simple to capture the true pattern. Variance is the error from a model being sensitive to the specific noise in its training set, so it would produce a meaningfully different result if trained on a different sample from the same distribution. A high-bias model underfits; a high-variance model overfits.

Noisy preference labels push a reward model toward higher variance. If two annotators disagree on the same pair 30 percent of the time, the model trained on that data partially memorizes the disagreement pattern rather than the true underlying preference signal, and its predictions become sensitive to exactly which labeled examples happened to end up in the training set. More data helps, but cleaner data helps more per example. Interviewers ask this specifically to see whether you'd reach for "collect more labels" as the only lever, when reducing label noise at the source usually matters more than adding volume.

Encode each task (the text prompt, or the image, depending on modality) into a dense vector using a sentence or vision encoder. Two tasks that are semantically similar produce vectors that point in a similar direction in embedding space, regardless of surface-level differences like word order or minor cropping. Cosine similarity measures the angle between two vectors and is a standard choice here because it's invariant to vector magnitude, which matters when comparing embeddings that weren't normalized the same way.

In practice this feeds the deduplication pipeline described earlier in this guide: new incoming tasks get embedded and compared against an approximate nearest-neighbor index of already-seen tasks. Anything above a similarity threshold gets flagged before it's routed to a labeler, which saves the cost of paying to label content that's already been labeled once under slightly different framing.

The framing of this question matters: they assume you shipped under pressure and want to know how you reasoned about the trade-offs, not whether you did it. The answer structure that lands: state the deadline and the constraint, name what you deprioritized and what you kept (and why each choice), describe the outcome, and say whether in hindsight the call was right. If it wasn't, say that too. "We shipped on time but the error rate was 3x for two weeks afterward and I should have pushed back on the deadline" is a better answer than a story where every trade-off was optimal.

Scale AI's ownership value specifically includes "ownership of outcomes, not just tasks." That means owning the downstream consequences of your shipping decision, not just the act of shipping.

This is the "ownership is the job" value in practice. The answer structure that fails: spending most of the answer explaining why it wasn't really your fault before getting to what you did. The answer structure that works: brief context (here's what happened), here's the part I owned (even though X, Y, Z also contributed), here's what I changed, here's how it resolved. Interviewers specifically watch for how long the attribution section is. Long attribution means low ownership. Short attribution means high ownership, even if the actual failure was partly caused by someone else.

Scale AI's "intellectual rigor" value means they want to see you push back with evidence when you believe something is wrong. Pure deference ("I trusted the manager's judgment") reads as low-agency. So does "I raised it once and then just did it." The answer they're looking for: you identified a specific problem with the prioritization call, gathered data or ran an experiment to quantify the risk, presented it, and either changed the decision or got a clear rationale for why the original call stood. "I changed my mind because the argument was good" is also a fine outcome. Updating based on new information is intellectual rigor, not weakness.

Scale AI's "run through walls" value is really about moving despite incomplete information, not about ignoring constraints. The answer that resonates: identify which parts of the problem are genuinely ambiguous vs. which parts you're treating as ambiguous because you haven't decided yet. For the genuinely ambiguous parts, make a reversible decision quickly, document the assumption, and move. For the irreversible parts (API contracts, data schemas, architectural choices), take the time to resolve the ambiguity before committing. The distinction between reversible and irreversible decisions is the key insight here.

Scale AI's clarity value shows up directly in how this question gets evaluated. Vague answers ("I told them to be more careful") don't hold up under follow-up. The answer that works names the specific gap (a pattern of missed edge cases, a recurring bug type, a review comment ignored twice), describes exactly what you said and when, and reports what changed afterward, including if nothing changed and what you did next.

One thing interviewers listen for: whether you gave the feedback privately and directly, or whether you routed around the person by escalating to a manager first. Scale AI's culture rewards the former. Escalating without a direct conversation first tends to read as an avoidance pattern, not diligence.

This is a variation on the ownership theme that shows up throughout the Credo round, but the specific angle here is about decision-making under uncertainty, not about outcomes you didn't cause. Interviewers want the moment you realized the call was wrong, not just the moment you made it. Answers that skip straight from "I decided X" to "and it worked out in the end" usually get pushed on: how did you find out it was wrong, and what would you do differently with the same information available at the time?

The strongest answers separate two things: whether the decision was reasonable given what you knew then, and whether the outcome was good. A reasonable decision that turned out badly because of information you couldn't have had is a different story than a decision you rushed past a knowable risk. Scale AI interviewers are listening for which one you think it was, and whether your answer changes anything about how you'd approach ambiguity next time.

OFFSET/LIMIT pagination asks the database to scan past the first N rows and discard them every single time, so page 1 is fast but page 10,000 means scanning and sorting through 10,000 rows before the database can hand back the 20 you actually want. On a table with millions of annotation records, that scan cost grows with the page number, and it gets worse fast if there's no index covering the sort, since the database has to sort the entire matching set before it can apply the offset.

There's a correctness problem here too, not just a speed one. If rows get inserted or deleted while someone's paging through results, offset based pagination can skip rows or show duplicates, because the position of a given row shifts as the underlying data changes underneath it.

Cursor based pagination fixes both. Instead of asking for skip N, take M, you ask for the M rows after a specific value, usually the primary key or a (created_at, id) tuple encoded as an opaque cursor.

SELECT *
FROM annotations
WHERE (created_at, id) > (:last_created_at, :last_id)
ORDER BY created_at, id
LIMIT 20;

An index on (created_at, id) satisfies that query directly, no scanning past rows you're going to throw away. The tradeoff is you lose the ability to jump straight to an arbitrary page number, which is fine for infinite scroll UIs and API consumers but is a real product limitation if someone wants a jump-to-page-500 control.

Gold tasks with known answers catch labelers who are careless or rushing, but they don't catch a ring that's actually paying attention to each other's answers, since a gold task looks like any other task to labelers who are guessing together. What you need is a signal that doesn't depend on planted answers at all: correlated disagreement with the consensus.

The core idea is to look at pairs of labelers and measure how often they agree with each other specifically on tasks where the group consensus disagrees with both of them. If two labelers genuinely work independently, their mistakes should be roughly uncorrelated, when they're both wrong, they're usually wrong in different ways. If they're colluding, sharing answers through a side channel, their mistakes will land on the same wrong answer far more often than chance predicts, even on tasks they were never both deliberately assigned as a check.

Concretely, for every pair of labelers with enough overlapping tasks, compute their agreement rate conditioned on both disagreeing with the consensus label, and compare that to a baseline rate expected given the overall error rate and number of label classes. Flag pairs whose correlated wrongness sits several standard deviations above that baseline. This needs a real sample size to have any statistical power, so it works better as a weekly batch job over the labeling graph than a live check, and I'd build the labeler pairs into a graph rather than scoring pairs in isolation, since rings are often three or more people and clustering on that correlated error graph surfaces groups that pairwise scoring alone would miss. The false positive risk is labelers who happen to share a shift and an easy task type, which produces genuinely higher agreement for legitimate reasons, so you want to control for task difficulty and task type overlap before flagging anyone for human review.

The most common cause is that the offline eval set doesn't represent the actual distribution of outputs the policy will produce once it starts optimizing against the reward model. A reward model can score well on a static, human-labeled comparison set while being poorly calibrated on the out-of-distribution outputs the policy generates as it exploits whatever the reward model happens to reward, classic reward hacking, where the policy finds outputs the reward model over-scores that a human rater would judge as worse.

I'd start by sampling actual rollouts from the RLHF-trained policy and scoring those same rollouts with both reward models, then having humans rate them too. If the new reward model scores those rollouts much higher than human raters do, relative to how the old reward model scored the same rollouts, that's a strong signal of reward hacking or a distribution mismatch between the eval set and what the policy is actually producing.

Second, I'd check whether the offline eval set has label noise that got averaged away into a single accuracy number. A reward model can hit higher pairwise accuracy on preference comparisons while learning a subtly different decision boundary that aligns worse with what actually matters downstream, especially if it was trained on preference data sourced from a different batch of labelers with different instructions.

Third, I'd check for length or style bias. Reward models are notorious for learning to reward longer or more confident-sounding responses regardless of actual quality, and if the new model has a stronger length bias than the old one, RLHF will push the policy toward longer outputs that score well offline but read as padded to real users. Plotting reward score against response length for both models on the same held-out set usually surfaces this quickly, if the correlation is much steeper for the new model, that's the answer.

Combine a hash map with a doubly linked list. The hash map gives O(1) lookup from document ID to its node in the list. The linked list keeps nodes ordered by recency, with the most recently used document at the head and the least recently used at the tail. On a cache hit, move the accessed node to the head. On a miss, insert at the head and evict the tail node if the cache is over capacity.

Python's OrderedDict does most of this for you with move_to_end and popitem(last=False), which is the answer most interviewers expect for a 20-minute version of this problem. They will ask you to implement it from scratch with a manual linked list if you have extra time or if you're interviewing for a systems-adjacent role.

Reported from applied engineering rounds in 2025, usually surfacing as a follow-up once the card game exercise wraps early. The follow-up question interviewers ask most: what happens to cache hit rate if guideline documents get revised mid-day and old cached versions go stale? That's a cache invalidation question, not a data structure question, and candidates who only defend the LRU logic tend to miss it.

from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity: int):
        self.capacity = capacity
        self.cache = OrderedDict()

    def get(self, doc_id: str):
        if doc_id not in self.cache:
            return None
        self.cache.move_to_end(doc_id)
        return self.cache[doc_id]

    def put(self, doc_id: str, document) -> None:
        if doc_id in self.cache:
            self.cache.move_to_end(doc_id)
        self.cache[doc_id] = document
        if len(self.cache) > self.capacity:
            self.cache.popitem(last=False)

A language detection system needs to handle very short inputs (as short as one word) where statistical models struggle, and very long inputs where character n-gram models are wasteful. A two-stage approach works: a fast, lightweight n-gram classifier (fastText's language detection model, for instance) handles the common case. A slower but more accurate transformer-based model handles ambiguous cases that the n-gram model is uncertain about (confidence below some threshold).

At inference scale, the fast path needs to be in-process and latency under 5ms. Cache common inputs. The slow path can be async if exact language is not needed in real-time. Reported as a Scale AI system design question from 2024 onsite candidates.

Mix a small percentage of gold tasks, tasks with a verified correct answer, into the normal queue so they're visually and structurally indistinguishable from real tasks. Track each labeler's accuracy against the gold answers over a rolling window and use that as a quality signal independent of inter-annotator agreement.

The design tension is sampling rate. Inject too many gold tasks and you waste real labeling capacity and labelers may start to recognize patterns in the gold set. Inject too few and the quality signal is noisy, especially for newer labelers who haven't hit enough gold tasks yet to get a reliable score. A reasonable starting point is 2 to 5 percent of a labeler's queue, adjusted upward for new labelers until their trust score stabilizes and downward for established labelers with a long track record.

Reported as a follow-up to the task distribution question in 2025 loops. Interviewers want to hear that you'd rotate the gold set periodically, since a static set of known tasks eventually gets recognized by experienced labelers who've seen them repeat.

Core components: an ingestion API that accepts annotation tasks, a task queue (likely Redis or a purpose-built queue), a worker management layer that tracks labeler availability and skill levels, a task router that matches tasks to appropriate labelers, a quality control engine, and a results aggregator.

The quality angle is where this diverges from a standard job queue problem. You need consensus logic: route each task to multiple labelers and compute agreement. Use Cohen's Kappa or simple majority vote depending on the task type. Flag tasks with low agreement for expert review rather than silently accepting noisy labels. Track per-labeler quality scores over time and reduce routing weight for consistently low-quality workers. Also plan for adversarial labelers: consistency checks (re-route a known ground-truth task occasionally and compare the labeler's response against the gold answer).

At 100K concurrent workers, the bottleneck is the task router, not the queue. A centralized task router becomes a hot path. Shard the router by task type or labeler geography. Interviewers at Scale AI will push on this failure mode: what happens when the router is slow?

The pipeline has to generate response pairs per prompt, route them to human annotators, collect preferences (A is better / B is better / Tie / Both bad), run quality checks, and feed the resulting preference dataset to a reward model training job.

Key design choices: how do you generate response pairs that are actually informative for the reward model? Random pairs are low-signal. Pairs where one response is clearly worse than the other teach the model less than pairs that are genuinely close calls. Interviewers want to see you argue for using the current policy model to generate challenging pairs, not random sampling.

Quality control: compute Cohen's Kappa per annotator per task type. Flag annotators with Kappa below 0.6 for re-training or removal. Watch for verbosity bias (annotators systematically preferring longer responses) and anchoring effects (annotators changing their preference based on response order). Randomize response order before display.

For throughput: experienced annotators produce roughly 50 preference pairs per hour. At scale, you need thousands of concurrent annotators to generate enough signal for training runs. The task routing system from the previous question becomes a dependency here. Reported as a senior-level design prompt at Scale AI onsites in 2025.

# Pseudocode for RLHF preference collection service

class PreferencePipeline:
    def generate_pairs(self, prompt: str, model, n_pairs: int = 4):
        """
        Generate n_pairs response pairs for a given prompt.
        Use beam search with temperature variation to produce
        diverse candidates, then select pairs by embedding distance
        (prefer pairs that are semantically close but diverge
        on quality signals).
        """
        candidates = model.sample(prompt, n=n_pairs * 2, temperature=0.8)
        return self._select_informative_pairs(candidates, n_pairs)

    def _select_informative_pairs(self, candidates, n):
        # Select pairs with moderate semantic distance
        # (not identical, not completely different)
        # Implementation uses cosine similarity on embeddings
        pass

    def collect_preference(self, pair_id: str, annotator_id: str,
                           preference: str):
        """preference: 'A' | 'B' | 'tie' | 'both_bad'"""
        self._validate_annotator_quality(annotator_id)
        self._store_preference(pair_id, annotator_id, preference)
        self._check_consensus(pair_id)

    def _check_consensus(self, pair_id: str):
        prefs = self._get_preferences(pair_id)
        if len(prefs) >= 3:
            kappa = self._compute_kappa(prefs)
            if kappa < 0.4:
                # Disagreement: route to expert reviewer
                self._escalate(pair_id)
            elif len(prefs) >= 5:
                # Sufficient signal: finalize
                self._finalize_preference(pair_id)

Requirements to clarify upfront: comparison modes (A/B head-to-head, absolute rating, or ranking), aggregation method (majority vote, mean score, confidence intervals), latency SLA for results, whether evaluations are human-only or hybrid human-plus-model, and versioning requirements for tracking evaluation results across model iterations.

Architecture: an ingestion API accepts (prompt, response-A, response-B, evaluation-config) tuples. A router decides which evaluations go to ML scoring (fast, cheap, less reliable) and which go to human evaluators (slow, expensive, ground truth). A task queue (Kafka or Kinesis at this throughput) buffers evaluation requests. A consensus engine aggregates results and computes agreement metrics. A versioned dataset store (with lineage tracking so you know which model version generated which responses) holds all results.

At 10K evaluations per minute: that's roughly 167 evaluations per second. If each evaluation generates one task for the queue, peak throughput is achievable with a horizontally scaled worker pool. The bottleneck is typically the consensus computation, not the ingestion. Shard the consensus engine by evaluation batch ID.

This is a multi-modal annotation problem, which makes it harder than a text annotation pipeline. Each labeling job contains synchronized LiDAR point clouds, camera frames, and video clips that must be annotated in a spatially and temporally consistent way. A bounding box annotated in frame 10 must be consistent with the same object's position in frame 11.

Storage first: large media files go to object storage (S3 or equivalent) with chunked upload for video. Metadata (file references, annotation schemas, job status) goes to a relational DB. Annotation data (the actual labels) go to a separate labeled-data store optimized for batch reads during model training.

The unique challenge here is temporal and spatial consistency. Two different annotators labeling the same object across consecutive frames might disagree on its boundary by 5 pixels. That's a quality control problem that requires temporal consensus logic, not just per-frame agreement. Scale AI interviewers specifically want to see candidates address consistency across time as a first-class concern.

Exact-match deduplication is the easy 10 percent of this problem: hash each example and drop repeats. The hard 90 percent is near-duplicates, the same underlying image or prompt submitted with slightly different cropping, phrasing, or formatting by two vendors who don't know about each other. Exact hashing misses all of these.

The standard approach is to encode each example with an embedding model (a vision encoder for images, a sentence encoder for text) and compute similarity against a nearest-neighbor index rather than every other example in the dataset. Brute-force pairwise comparison is O(n squared) and doesn't scale past a few hundred thousand examples. Use an approximate nearest-neighbor index (FAISS or a hosted vector database) and flag pairs above a similarity threshold for review rather than auto-deleting, since false positives here silently remove valid training data.

Threshold tuning is where interviewers push hardest. Too low and near-duplicates slip through and get labeled twice at full cost. Too high and genuinely distinct examples get flagged and either wasted on manual review or wrongly dropped. Reported as a senior ML system design prompt in 2025, frequently asked as a follow-up once vendor diversity comes up in the RLHF pipeline question.

Two very different read patterns compete here: operators watching a live number tick during an incident, and analysts pulling a week-over-week trend. Serving both from the same raw event stream doesn't scale, so split them. Raw labeling events (task completed, task flagged, quality check result) stream through Kafka or Kinesis into a stream processor that maintains rolling aggregates, tasks per minute, error rate, and average time per task, at multiple time granularities.

Write the pre-aggregated rollups to a time-series store rather than querying raw events on every dashboard refresh. For the live view, keep the last few minutes of aggregates in an in-memory cache the dashboard polls directly. For historical trends, the time-series store handles downsampled queries over days or weeks without touching raw event volume.

Reported from staff-level system design rounds in 2025 and 2026. The follow-up interviewers ask most often: what happens when a labeling pool goes offline and throughput drops to zero, does your dashboard distinguish "no work available" from "system outage"? Candidates who only design for the happy path tend to miss this.

The core problem this solves is training-serving skew: a model trained on features computed one way in a batch pipeline behaves differently in production if the online feature computation drifts even slightly from the offline version. The fix is a shared feature definition that both paths read from, not two separately maintained pipelines that happen to agree today.

Architecture: an offline store (a data warehouse or Parquet files on object storage) holds historical feature values computed in batch, used for training. An online store (a low-latency key-value store like Redis or DynamoDB) holds the current value of each feature, used at inference time. Both are populated from the same feature transformation code, ideally shared as a library rather than reimplemented per pipeline.

Point-in-time correctness matters more than most candidates initially account for. When you're building a training set, you have to join each label with the feature values as they existed at the time the label was generated, not the current feature values. Using today's feature values to train on a label from three months ago leaks future information into the model. Reported from ML engineer onsites in 2025, particularly for candidates whose take-home covered feature engineering.

I'd treat this as three separate versioning problems that get tied together at training time: the dataset, the labeling instructions, and the code and config that trained the model.

For the dataset, every batch of labeled examples gets an immutable snapshot, a content-addressed hash of the exact rows and labels at the moment they were pulled for training, not a mutable latest pointer that keeps changing underneath you. Storing this in a table format with time travel, like Delta Lake or Iceberg, means you can query the dataset exactly as it looked at a specific commit months later, even after upstream rows have since been relabeled or corrected.

For labeling instructions, the guideline documents need version numbers too, since the same raw data labeled under guideline v3 versus v4 can produce systematically different label distributions. I'd store a guideline_version field on every annotation row rather than only on the batch, because guidelines sometimes change partway through a batch.

At training time, the pipeline writes a manifest that pins the dataset snapshot hash, the guideline version, the preprocessing code's commit SHA, and the hyperparameter config, all tied to the resulting checkpoint's ID. To reproduce a checkpoint, you check out that manifest, pull the dataset by its hash, and rerun training against the pinned code. The failure mode I've seen without this in place: someone fixes a bug in a shared preprocessing script, every downstream model starts training on slightly different features, and there's no way to tell which checkpoints are affected until something breaks in production.

I'd start by confirming the blast radius before touching any code. Is it every task type in that region or just one? Is it every labeler in that region or a subset? Did it start at an exact timestamp or ramp up gradually? A hard cutover at a specific time points at a deploy, a config change, or an infra event. A gradual ramp points at something like disk fill-up, connection pool exhaustion, or replication lag creeping up over hours.

Since it's isolated to one region, I'd check the regional infrastructure first: is there a regional database replica serving that traffic, and is its replication lag spiking? Did a subset of app servers in that region start failing health checks, with traffic now piling onto the survivors? I'd pull p50, p95, and p99 latency broken down by hop, DNS, connection setup, time to first byte, database query time, since a network issue in that region usually shows up as higher connection time, while a database problem shows up as higher query time specifically.

If infra looks clean, the next suspect is data skew. Maybe that region is running a disproportionate share of one heavy task type, video annotation with large payloads, say, because of how work happened to get routed, and that task type saturates a smaller regional worker pool that wasn't sized for it. I'd check whether the task mix in that region changed right before the spike, since a routing bug that shifts task distribution is a far more common root cause in my experience than actual hardware failure. Last, I'd rule out a noisy neighbor, another job or batch process sharing the same regional database or cache that happened to kick off around the same time.

With a fixed pool of GPU workers, you're trying to smooth bursts into a rate the workers can actually sustain, rather than reject everything that crosses some threshold. I'd put a token bucket limiter in front of the API sized per customer to their contracted quota, so one customer's burst can't starve everyone else, plus a separate global token bucket sized to the aggregate throughput the GPU pool can sustain end to end.

Requests that pass the limiter land on a durable queue, SQS or a Kafka topic, rather than going straight to a worker. That decouples accepting a request from processing it. Workers pull from the queue at their own pace, so a traffic burst gets absorbed as queue depth instead of dropped requests or workers running out of memory. The API can return a 202 with a job ID right away and let the customer poll or get a webhook when the eval finishes, instead of holding an HTTP connection open for a GPU job that might run for minutes.

For the backpressure signal itself, I'd feed queue depth and estimated wait time back into the rate limiter, so once the queue grows past a threshold, the per-customer limiter starts returning 429s with a Retry-After header instead of letting requests keep piling into a queue that only gets longer. An unbounded queue turns a rejected-request problem into an unbounded-latency problem, which is worse. I'd also drive GPU pool autoscaling off queue depth and the age of the oldest message rather than GPU utilization alone, since eval jobs often show low utilization while waiting on data loading, and utilization-based autoscaling reacts too late to a real backlog.