Databricks Interview Questions (2026): What They Actually Ask

DENSE_RANK() over a partition by department, ordered by salary descending, then filter to rank 2. DENSE_RANK matters here, not ROW_NUMBER: if two people in a department tie for the top salary, ROW_NUMBER arbitrarily calls one of them rank 1 and the other rank 2, silently giving you the wrong answer for "second highest." DENSE_RANK gives both of them rank 1 and the next distinct salary rank 2, which is what the question actually means.

SELECT name, department, salary
FROM (
  SELECT name, department, salary,
         DENSE_RANK() OVER (
           PARTITION BY department ORDER BY salary DESC
         ) AS rnk
  FROM employees
) ranked
WHERE rnk = 2;

The classic trap version, MAX(salary) WHERE salary less than MAX(salary), breaks the same way on ties and also requires a self-join or subquery to scope per department. Interviewers ask candidates to name why it breaks, not just to write the working version.

SUM as a window function ordered by date, with an explicit frame from unbounded preceding to the current row.

SELECT
  sale_date,
  daily_revenue,
  SUM(daily_revenue) OVER (
    ORDER BY sale_date
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  ) AS running_total
FROM daily_sales
ORDER BY sale_date;

Worth knowing cold: if you omit the explicit ROWS clause and leave just ORDER BY, most engines default to a RANGE frame instead of ROWS, and RANGE groups tied order-by values together, so two rows with the same sale_date get summed into the same running total instead of accumulating one row at a time. That's a real bug people ship, not a trick question.

Every company asks a version of this. The Databricks version turns into a 20-30 minute technical deep-dive if the hiring manager is interested. Have a project ready where the hard part was technical, not organizational. The hiring manager will ask follow-up questions at the architecture level and won't stop at "we used Kafka for streaming." Know why you made the choices you made.

The honest answer to "what's the hardest part?" is usually something you got wrong or had to redo. That's the version that reads as credible. "Everything went smoothly" is a signal that you either didn't do the hardest part of the work or aren't being candid.

Real answers name specific sources: the Databricks engineering blog, the Delta Lake changelog, the Apache Spark release notes, Martin Kleppmann's "Designing Data-Intensive Applications" as a reference, specific papers (the original Dremel paper, the Bigtable paper, the Lakehouse paper from CIDR 2021). Generic answers like "I follow tech news and read blogs" are thin. Databricks engineers are close to the research, and they notice when a candidate hasn't engaged with the primary sources.

A mentoring and communication check, and a common one given how much of the actual job is explaining tricky concurrency reasoning to teammates who didn't build the system. Strong answers name the specific misconception, usually assuming a distributed operation is atomic when it isn't, or assuming a network call always succeeds, describe how you explained it (an analogy, a small reproducible example, tracing a failure case on a whiteboard), and how you confirmed they actually got it rather than just nodded.

Most candidates report four to eight weeks from application to verbal offer. The OA and phone screen can be completed in the first two weeks. The gap between onsite completion and offer letter is often longer than candidates expect, three to four weeks in some cases, because of the committee review and VP sign-off. Referrals typically move faster, compressing the early stages to one to two weeks.

Yes, based on multiple reported accounts from 2024 and 2025. It's a full 45-to-60-minute round focused specifically on concurrency and multithreading, not a coding round with one threading question added in. The format varies slightly by team, some run it as an implementation problem (build this thread-safe data structure), others as a diagnosis problem (here's a buggy concurrent program, find the race condition). Either way, plan to spend as much prep time on this round as on a standard coding round.

For software engineering roles (as opposed to data engineering roles), the coding and system design rounds don't require you to write Spark code. But interviewers will ask you about distributed systems concepts that Spark and Delta Lake embody: shuffle semantics, partitioning, ACID transactions, and fault tolerance. Knowing what Spark actually does under the hood, even if you've never written a Spark job, is a meaningful advantage in the system design and behavioral rounds.

Medium to hard, with hard problems appearing more often than at most comparable-stage companies. The OA typically includes two easy-to-medium problems and two medium-to-hard problems. Onsite coding rounds are predominantly medium-hard. Binary search is emphasized more than at comparable firms. The concurrency round adds a separate dimension that standard LeetCode medium prep doesn't address.

For SWE roles, the core coding and design rounds are general software engineering problems, not Databricks-platform-specific. The hiring manager call and behavioral rounds may probe your experience with large-scale data systems, but you won't be asked to write Delta Lake syntax or optimize a Spark query in a coding round. Data engineering roles (separate from SWE) do have heavier Spark and Delta Lake content. Confirm with your recruiter which track you're on.

Four to five rounds in the onsite, lasting four to five hours total. Standard split: two algorithm/data-structures coding rounds, one concurrency and multithreading round, one system design round, and one behavioral round. Senior and staff candidates sometimes see a second system design round. The onsite is fully virtual in 2026. System design is conducted in Google Docs rather than a diagramming tool, which is unusual enough to be worth practicing for.

DBFS is a distributed file system mounted into a Databricks workspace that sits on top of the cloud object storage backing that workspace, S3 on AWS, ADLS Gen2 on Azure, or GCS on GCP. It gives you a POSIX-like path structure such as /mnt or /FileStore so notebooks and jobs can read and write files with plain file paths instead of calling a cloud SDK directly.

In practice teams use it for staging small files, storing notebook-uploaded datasets under /FileStore, mounting external buckets so multiple users share the same storage location, and holding library jars or init scripts a cluster needs at startup. The catch is that the DBFS root is not meant to hold production data at scale. Most teams point real pipelines at cloud storage directly through Unity Catalog external locations, because DBFS root storage isn't governed by the same fine-grained access control as Unity Catalog managed tables, and treating it as your production data lake tends to turn into an access control mess later.

An all-purpose cluster is what you spin up for interactive work, attaching notebooks, running ad hoc queries, exploring a dataset cell by cell. It stays up until someone terminates it or an auto-termination policy kicks in, and multiple users and notebooks can attach to the same cluster at once, which is convenient for collaboration but means one person's runaway query can slow down everyone else's session.

A job cluster gets created automatically when a scheduled job runs, spins up with exactly the config that job specifies, runs only that job's tasks, and terminates the moment the job finishes. Nobody attaches to it interactively. Job clusters bill at a lower DBU rate than all-purpose clusters, and since every run gets a fresh cluster, you avoid noisy-neighbor problems or leftover state bleeding from one run into the next. A common junior mistake is pointing a production job at an all-purpose cluster to save setup time, which works fine until that cluster gets reused for something else and the job either fails or starves for resources next to unrelated interactive queries.

Unity Catalog is Databricks' governance layer for data and AI assets across an entire account, not just one workspace. Before it existed, access control lived at the workspace level through the Hive metastore, so ten workspaces meant ten separate places to define who could read which table, with no single audit trail spanning all of them. Unity Catalog introduces a three-level namespace, catalog.schema.table, and lets one metastore attach to many workspaces, so permissions, lineage, and audit logs stay consistent no matter which workspace someone connects from.

Concretely, it centralizes access control with GRANT and REVOKE statements that read like standard SQL permissions instead of workspace-specific ACLs, it generates automatic column and table level lineage so you can see what fed a table and what consumes it, and it gives admins one place to manage external locations and storage credentials instead of every workspace owner configuring their own mount points. It also underpins row filters and column masks, which is what most companies actually need once a table mixes PII with columns that are fine for a broader audience to see.

An RDD, resilient distributed dataset, is Spark's original low-level abstraction, a distributed collection of JVM objects with no schema, where transformations are written as lambdas and Spark has no idea what's inside those objects until it actually runs them. That means no query optimization beyond the transformation graph itself, since the optimizer can't reason about columns it can't see.

A DataFrame is a distributed collection of rows with a known schema, similar to a table. Because Spark knows the column names and types up front, the Catalyst optimizer can push down filters, reorder joins, and rewrite the plan before anything executes, which is why DataFrame code is almost always faster than the equivalent hand-written RDD code for anything past a trivial map or filter. The tradeoff is some loss of compile-time type safety, since a DataFrame is really Dataset[Row] under the hood in Scala, and a typo in a column name only shows up at runtime.

A Dataset adds compile-time type safety on top of the same execution engine, available in Scala and Java, where you work with typed case classes and the compiler catches a schema mismatch before you run anything. PySpark has no separate Dataset API because Python isn't statically typed, so in PySpark you work with DataFrames the whole time, and when someone says "Dataset" in a PySpark context they usually just mean DataFrame.

The core requirement: implement a set that supports add, remove, and snapshot operations. Snapshots must be immutable, meaning any adds or removes to the live set after a snapshot is taken should not affect iterators created from that snapshot.

Use a versioned structure, either a persistent data structure or a copy-on-write approach. A simple implementation copies the underlying set at snapshot time (O(n) per snapshot, O(1) iterator creation). A more sophisticated approach uses a version counter and stores mutations as a log, materializing the snapshot lazily. Interviewers at Databricks push toward the lazy approach as a follow-up when the naive copy version is working.

class SnapshotSet:
    def __init__(self):
        self._version = 0
        self._snapshots = {}  # version -> frozenset
        self._live = set()

    def add(self, val):
        self._live.add(val)

    def remove(self, val):
        self._live.discard(val)

    def snapshot(self) -> int:
        snap_id = self._version
        self._snapshots[snap_id] = frozenset(self._live)
        self._version += 1
        return snap_id

    def get_iterator(self, snap_id: int):
        if snap_id not in self._snapshots:
            raise ValueError("Invalid snapshot ID")
        return iter(self._snapshots[snap_id])

Convert each CIDR block to a network address and mask. For a CIDR like 192.168.1.0/24, the mask has 24 ones followed by 8 zeros. Apply the mask to both the target IP and the network address; if they match, the IP is in the range. Parse dotted-decimal notation into 32-bit integers for clean bitwise comparison.

Reported as an onsite problem. The key insight is that you're doing bitwise AND with the subnet mask, not string comparison. Candidates who try to solve this with string parsing hit edge cases in the last 10 minutes and don't recover. Work with integers from the start.

def ip_in_cidr(ip: str, cidr: str) -> bool:
    def ip_to_int(ip_str):
        parts = list(map(int, ip_str.split('.')))
        return (parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8) | parts[3]

    network, prefix_len = cidr.split('/')
    prefix_len = int(prefix_len)
    mask = ((1 << 32) - 1) ^ ((1 << (32 - prefix_len)) - 1)

    ip_int = ip_to_int(ip)
    net_int = ip_to_int(network)

    return (ip_int & mask) == (net_int & mask)

def ip_in_any_cidr(ip: str, cidrs: list) -> bool:
    return any(ip_in_cidr(ip, cidr) for cidr in cidrs)

The naive approach checks all rows, columns, and diagonals after each move, O(n) per move. The efficient approach maintains running sums per row, column, and the two diagonals. On each move, increment the relevant counters. When any counter reaches n, the mover wins. This gets the per-move complexity down to O(1).

Reported in the OA. The extension question: "Make it n-dimensional." That's almost never asked in full, but the framing tells you the interviewer wants to see you generalize your solution beyond the 2D case. Acknowledging the generalization and sketching how you'd represent n-dimensional indices in a flat counter structure is usually enough.

Count the frequency of each task. The bottleneck is the most frequent task. The minimum time is max(total_tasks, (max_freq - 1) * (cooldown + 1) + count_of_tasks_with_max_freq). You don't need to simulate the schedule explicitly. The formula works because you're filling idle slots with lower-frequency tasks or accepting actual idle time when no tasks are available.

LeetCode 621. Reported in multiple Databricks OA write-ups. The formula-based approach is cleaner than a priority queue simulation, but the priority queue approach is valid and easier to verify during an interview. If you can't remember the formula, simulate with a max-heap: always schedule the most frequent available task.

Build a general graph from the binary tree by adding bidirectional edges (parent-to-child and child-to-parent). Then run BFS from the target node. The first leaf node reached is the closest. The bidirectional edge requirement means you need to track the parent of each node during the initial tree traversal before BFS starts.

LeetCode 742. The key insight is converting the tree to a graph. Candidates who try to solve this purely recursively in tree-traversal style hit cases where the closest leaf is reachable only by going up through an ancestor and back down on the other side. BFS on the graph representation handles this naturally.

Min-heap of size k. Push the first element of each list along with its list index. Pop the minimum, append it to the result, then push the next element from whichever list that minimum came from. Runs in O(N log k) where N is the total element count across all lists, since every push and pop touches a heap of size k rather than the full N.

LeetCode 23, reframed. Databricks interviewers describe it as merging per-executor log files by timestamp for a distributed trace viewer, which is close to what actually happens when Spark reconciles shuffle output files. A divide-and-conquer pairwise merge gets the same O(N log k) bound with a smaller constant factor and no heap at all; mentioning both approaches and picking one with a stated reason is what separates a pass from a borderline pass here.

import heapq

def merge_k_sorted(lists: list) -> list:
    heap = []
    for i, lst in enumerate(lists):
        if lst:
            heapq.heappush(heap, (lst[0], i, 0))

    result = []
    while heap:
        val, list_idx, elem_idx = heapq.heappop(heap)
        result.append(val)
        if elem_idx + 1 < len(lists[list_idx]):
            next_val = lists[list_idx][elem_idx + 1]
            heapq.heappush(heap, (next_val, list_idx, elem_idx + 1))
    return result

Maintain a min-heap capped at size k. Every new value gets pushed; if the heap grows past k, pop the smallest. The top of the heap is always the kth largest value seen so far, and each insert costs O(log k) instead of resorting the entire history.

import heapq

class KthLargest:
    def __init__(self, k: int, nums: list):
        self.k = k
        self.heap = nums
        heapq.heapify(self.heap)
        while len(self.heap) > k:
            heapq.heappop(self.heap)

    def add(self, val: int) -> int:
        heapq.heappush(self.heap, val)
        if len(self.heap) > self.k:
            heapq.heappop(self.heap)
        return self.heap[0]

LeetCode 703. The Databricks framing is a cluster health monitor tracking the kth-highest CPU sample across nodes without storing full history, which is the actual constraint: you can't afford to keep every sample from every node in memory. Reported in OA summaries as one of the "easy" problems that trips people up anyway, because candidates over-engineer it with a sorted list before someone points out heapq already does the work.

dp[i] is true if the substring ending at index i can be fully segmented into dictionary words. dp[0] starts true (empty prefix). For each i, check every j less than i: if dp[j] is true and s[j:i] is in the dictionary, dp[i] is true. Naive complexity is O(n^2 times average word length); a trie built from the dictionary cuts the substring-lookup cost.

LeetCode 139. Reported as a DP warm-up before the harder onsite rounds, not usually the headline problem on its own. Databricks frames it as validating whether a raw line concatenated from a corrupted Kafka topic reconstructs into known event-type tokens, which is a real failure mode when a producer flushes a partial buffer.

Flood fill. Scan the grid; whenever you hit an unvisited healthy node, run BFS or DFS to mark every connected healthy node as visited and increment the cluster count. Each cell gets visited once, so the whole scan is O(rows times cols).

LeetCode 200, reported in a phone screen. The follow-up that actually separates strong candidates: "what if nodes join or leave the cluster after your initial scan?" A full re-scan on every membership change doesn't hold up at scale, so the expected pivot is union-find with incremental unions on join and, trickier, some form of lazy re-evaluation on leave, since union-find doesn't support efficient splits.

GROUP BY every column HAVING COUNT(*) greater than 1 is the textbook answer, and it's the wrong one at Databricks scale: grouping on every column of a multi-terabyte table forces a full shuffle across all of them. The better pattern, and the one that matches how Delta Lake MERGE-based deduplication actually works in production, is ROW_NUMBER partitioned by a natural key, ordered by ingestion timestamp descending, keeping only rank 1.

SELECT *
FROM (
  SELECT *,
         ROW_NUMBER() OVER (
           PARTITION BY event_id
           ORDER BY ingestion_ts DESC
         ) AS rn
  FROM raw_events
) deduped
WHERE rn = 1;

Reported as a data-engineering-track question. The follow-up is usually "how would you prevent the duplicates instead of cleaning them up after the fact," which points toward an idempotent MERGE INTO keyed on event_id at write time rather than a nightly dedup job.

A correlated subquery re-evaluates for every row of the outer query, which at Databricks scale over a Delta table means re-scanning (or at least re-planning against) the inner table once per outer row. Pre-aggregating the inner query once and joining on the result computes the same thing a single time.

-- Slow: correlated subquery re-evaluated per outer row
SELECT o.order_id, o.customer_id
FROM orders o
WHERE o.total > (
  SELECT AVG(o2.total)
  FROM orders o2
  WHERE o2.customer_id = o.customer_id
);

-- Faster: pre-aggregate once, then join
SELECT o.order_id, o.customer_id
FROM orders o
JOIN (
  SELECT customer_id, AVG(total) AS avg_total
  FROM orders
  GROUP BY customer_id
) avgs
  ON avgs.customer_id = o.customer_id
WHERE o.total > avgs.avg_total;

Spark's Catalyst optimizer can decorrelate some subqueries automatically, so the rewritten version doesn't always run faster in practice. Interviewers care less about the runtime and more about whether you can read an EXPLAIN plan and reason about shape, not just correctness.

The naive version acquires a mutex before writing, releases after. That's correct but the interviewer will push: "What if logging is slow because it's writing to disk? You're blocking all threads on I/O." The expected step-up: a bounded blocking queue as a write buffer. Each writer enqueues a log entry (fast, in-memory, with a lock); a dedicated logger thread dequeues and writes to disk. This decouples write throughput from disk I/O latency.

Reported from the Databricks concurrency round. Follow-up: "What happens if the queue fills up?" Options: block the producer (backpressure), drop logs with a counter (lossy but non-blocking), or flush synchronously when the queue is full. The trade-off is availability vs. completeness of logs, and Databricks interviewers want to hear you name that trade-off explicitly.

import threading
import queue

class AsyncLogger:
    def __init__(self, maxsize=1000):
        self._q = queue.Queue(maxsize=maxsize)
        self._stop = threading.Event()
        self._worker = threading.Thread(target=self._flush_loop, daemon=True)
        self._worker.start()

    def log(self, message: str):
        try:
            self._q.put_nowait(message)
        except queue.Full:
            pass  # drop under backpressure; counter omitted for brevity

    def _flush_loop(self):
        while not self._stop.is_set() or not self._q.empty():
            try:
                msg = self._q.get(timeout=0.05)
                print(msg)  # replace with file write in production
                self._q.task_done()
            except queue.Empty:
                continue

    def shutdown(self):
        self._stop.set()
        self._worker.join()

BFS over the URL graph, but parallelized. A shared visited set (protected by a lock or using a ConcurrentSet equivalent) tracks fetched URLs. A thread pool processes URLs from a work queue. Each thread fetches a page, extracts links, filters already-visited URLs, and enqueues new ones. The tricky part is termination: you need to know when all work is complete and the queue is empty, not just when the queue is momentarily empty between enqueues.

LeetCode 1242. Reported in Databricks coding rounds. The termination condition is where most candidates get it wrong. Use an active-task counter alongside the queue: increment when enqueuing, decrement when a task completes. Signal completion when both the counter reaches zero and the queue is empty.

HashMap's internal state (the bucket array, the size counter, and the rehash logic) is not atomically updated. Two threads calling put simultaneously can corrupt the internal linked list during a bucket collision, or both trigger a rehash and lose each other's writes. In Java's HashMap specifically, two concurrent puts during rehash can create an infinite loop in the linked list. Options in Java: ConcurrentHashMap (striped locking, O(1) per operation at low contention); Collections.synchronizedMap (single lock, simple but lower throughput); or explicit ReadWriteLock if reads dominate. In Python, the GIL makes dict operations thread-safe for single operations but not for compound read-modify-write sequences.

Reported as a conceptual question in the Databricks concurrency round. The Python-specific GIL nuance matters here: Python candidates often claim dict is thread-safe and they're right for atomic operations but wrong for compound operations like "check-then-insert."

A shuffle occurs when Spark needs to redistribute data across partitions, typically triggered by operations like groupBy, join, or distinct. The process: map-side tasks write their output partitioned by the shuffle key to local disk files; reduce-side tasks fetch the relevant partitions from all map tasks over the network and re-sort the data for the downstream operation.

The expense comes from three sources: disk I/O (writing intermediate shuffle files and reading them back), network I/O (all-to-all data transfer between executors, which scales as O(m x r) where m is map tasks and r is reduce tasks), and GC pressure from materializing large shuffle blocks in memory. Mitigation: reduce shuffle size through predicate pushdown and projection before the shuffle operation; use broadcast joins to eliminate shuffles entirely when one side is small enough to fit in executor memory (typically under 10MB by default in Spark, configurable).

Delta Lake stores metadata as a series of JSON log files in the _delta_log directory alongside the data files (Parquet). Each transaction appends a new JSON entry to the log describing which files were added, which were removed, and what schema was applied. The current state of the table is derived by replaying the log from the last checkpoint forward.

ACID compliance works as follows. Atomicity: either all files in a transaction are committed to the log or none are. Consistency: schema enforcement validates each write against the declared schema before committing. Isolation: optimistic concurrency control with conflict detection during the commit protocol, two writers who attempt to modify the same files will fail the one that commits second with a conflict error. Durability: once the log entry is written, the transaction is durable. Databricks interviewers probe on what happens during concurrent writes from two Spark jobs and how the conflict is surfaced to the application layer.

Z-ordering is a multi-dimensional data clustering technique that co-locates related data in the same set of files. When you run OPTIMIZE ... ZORDER BY (column_a, column_b), Delta Lake rewrites the data files so that rows with similar values in those columns are stored together. Subsequent queries with filters on those columns can skip files that don't contain relevant ranges, a technique called data skipping.

Use it when you have high-cardinality columns that frequently appear in query filters and you're not partitioning by those columns (because partitioning by a high-cardinality column creates too many small files). The trade-off: OPTIMIZE with Z-ordering is an expensive write operation. It's appropriate for tables that are read much more often than written, and where file-skipping savings exceed the rewrite cost. Not appropriate for streaming tables with continuous low-latency writes.

Catalyst is Spark SQL's query optimizer. It parses your query into a logical plan, applies rule-based transformations (predicate pushdown, constant folding, column pruning), then generates one or more physical plans and picks the cheapest one using statistics gathered before the job runs. All of that happens once, up front, based on estimates.

Adaptive Query Execution, on by default since Spark 3.0, re-optimizes mid-flight using the actual statistics a completed stage produced. It can flip a sort-merge join to a broadcast join once it sees the real size of one side, coalesce shuffle partitions that turned out smaller than expected, and split a partition that turned out skewed. The follow-up worth preparing for: "what can't AQE fix?" AQE re-plans between stages, so it can't rescue a single task that's already OOMing inside a stage that hasn't finished yet. That's still a query-design problem, not a runtime one.

A broadcast join sends a full copy of the smaller table to every executor up front, so the join happens locally with no shuffle at all. The default size threshold is 10MB (spark.sql.autoBroadcastJoinThreshold), though teams commonly raise it well past 100MB for wide dimension tables once they've confirmed executor memory can absorb it.

from pyspark.sql.functions import broadcast

result = large_orders_df.join(
    broadcast(small_dim_table_df),
    on="dim_key",
    how="inner",
)

The trade-off interviewers push on: broadcasting a table that's actually too large duplicates it across every executor simultaneously, which can be worse than the shuffle it was meant to avoid. AQE will convert a sort-merge join to a broadcast join at runtime if it detects, after the fact, that one side turned out small enough, even if the static plan picked shuffle originally.

cache() is shorthand for persist() with a default storage level, MEMORY_AND_DISK for the DataFrame and Dataset APIs (worth noting: the older RDD API defaults to MEMORY_ONLY instead, a distinction interviewers sometimes probe just to see if you know it). persist() lets you choose explicitly: serialized in-memory to cut GC pressure, disk-only when memory is tight, or replicated storage levels (the _2 suffix) that keep a second copy on another node for faster recovery.

from pyspark.storagelevel import StorageLevel

df.cache()  # shorthand for persist(StorageLevel.MEMORY_AND_DISK)

df.persist(StorageLevel.MEMORY_ONLY_SER)  # serialized, memory-only
df.persist(StorageLevel.DISK_ONLY_2)      # replicated to 2 nodes, disk only

Recomputation happens because Spark's execution graph is lazy and lineage-based: if an executor holding a cached partition is lost mid-job, Spark recomputes just that partition from its lineage rather than failing the whole job. That's what makes Spark fault-tolerant without needing synchronous replication of every cached dataset.

Databricks interviewers probe for the specific technical disagreement, not just the interpersonal resolution. Describe what the two positions were, what evidence you brought, what evidence they brought, and what the outcome actually was. "We talked it out and found a middle ground" is a non-answer here. The interviewers want to know whether the resolution was technically correct and whether you were willing to be wrong.

The follow-up is frequently: "Looking back, who was right?" Being honest about a case where your colleague was right is more credible than a story where you were right and they came around. Databricks teams work on complex systems where reasonable engineers disagree constantly. The ability to update your view is as valued as the ability to hold a position.

This question is about engineering judgment under change, not about your project management skills. Interviewers want to hear: what was the original design? What specifically changed? What was the cost of the original design in the new world? What did you do, and what did you defer? The answer should have specific technical content, not just "we re-prioritized."

A strong answer names concrete trade-offs. "We had built the pipeline assuming append-only writes; when requirements changed to include deletes, we had to introduce a CDC layer. We chose to accept eventual consistency on the delete path for 48 hours rather than block the existing functionality while rebuilding."

This is a diagnostic reasoning question. The best answers have a structured approach: first, confirm the problem is real and reproducible; second, narrow the scope by checking whether the problem is isolated to one node, one partition, one job, or one user; third, examine logs and metrics for anomalies at the time the failure started; fourth, form a hypothesis and design a targeted test. Generic answers like "I look at the logs and ask colleagues" don't land well.

Databricks-specific context helps here. Mentioning Spark UI for DAG visualization, executor-level logs for task failures, and Delta Lake transaction log inspection for data integrity issues shows that you've actually worked with distributed systems rather than just studied them.

This one probes data skepticism directly, which matters more at a data infrastructure company than most. Interviewers want the specific tip-off: a number that didn't reconcile against a second source, a metric that moved with no corresponding event upstream. Then what you actually did to verify it, traced the pipeline back to the raw source, checked for a silent schema change, diffed against an independent report, not just "I raised a flag."

Weak answers stop at "I told the team something looked off." Strong ones name what was actually broken: a silent schema evolution in an upstream topic, a timezone bug in a date-truncation step, a join that fanned out rows for weeks before anyone noticed the totals were inflated.

Name the trade-off explicitly. What was the shortcut, what would the "right" version have cost in time, and what did the debt actually cost downstream, a manual reconciliation someone ran weekly, a job that hit a wall at a certain volume and needed a rewrite anyway. A good answer also says whether the debt got paid down later, and if not, why not.

Less about the emotional response, more about triage under pressure. Interviewers want your decision process for fix-forward versus rollback, how you communicate status while still mid-debug (a short update every 15 to 20 minutes beats silence, even if the update is "still investigating"), and what the postmortem looked like afterward.

One detail that signals real experience: whether you mention checkpoint recovery for Structured Streaming jobs specifically, can you just restart from the last checkpoint, or did the checkpoint itself get corrupted, which turns a bad night into a much worse one. Candidates who've actually operated one of these pipelines bring that up unprompted.

In-place solution using the array as a hash table. For each value v in range [1, n], place it at index v-1. Make a pass to swap elements into their target positions. Then scan the array; the first index i where array[i] != i+1 means i+1 is missing. If all positions are correct, the answer is n+1. O(n) time, O(1) space.

LeetCode 41. Reported in Databricks onsite coding rounds. The hard part is the in-place constraint. Most candidates default to a hash set approach (O(n) space), which works but gets a follow-up: "Can you do this without extra space?" The index-as-hash-table insight is the only O(1)-space solution.

def first_missing_positive(nums: list) -> int:
    n = len(nums)
    # place each num in its correct position
    for i in range(n):
        while 1 <= nums[i] <= n and nums[nums[i] - 1] != nums[i]:
            nums[nums[i] - 1], nums[i] = nums[i], nums[nums[i] - 1]

    for i in range(n):
        if nums[i] != i + 1:
            return i + 1
    return n + 1

Two pointers starting at both ends, tracking a running left_max and right_max. At each step, advance whichever pointer has the smaller max, since the water trapped at that position is bounded by the smaller of the two maxes regardless of what's happening on the other side. O(n) time, O(1) space.

LeetCode 42. Interviewing.io lists this among the hardest onsite problems reported for Databricks, and the reason isn't the algorithm itself, it's that most candidates only know the stack-based O(n) O(n) solution and freeze when asked for O(1) space on the spot.

def trap(height: list) -> int:
    if not height:
        return 0
    left, right = 0, len(height) - 1
    left_max, right_max = height[left], height[right]
    water = 0
    while left < right:
        if left_max <= right_max:
            left += 1
            left_max = max(left_max, height[left])
            water += left_max - height[left]
        else:
            right -= 1
            right_max = max(right_max, height[right])
            water += right_max - height[right]
    return water

LAG gives you the previous event time per user. Flag a new session whenever the gap since the prior event exceeds 30 minutes, or when there is no prior event at all. Then a cumulative SUM of that flag, ordered by time, produces a stable session ID per user.

WITH gaps AS (
  SELECT
    user_id,
    event_time,
    LAG(event_time) OVER (
      PARTITION BY user_id ORDER BY event_time
    ) AS prev_event_time
  FROM user_events
),
flagged AS (
  SELECT
    user_id,
    event_time,
    CASE
      WHEN prev_event_time IS NULL
        OR event_time > prev_event_time + INTERVAL 30 MINUTES
      THEN 1 ELSE 0
    END AS new_session
  FROM gaps
)
SELECT
  user_id,
  event_time,
  SUM(new_session) OVER (
    PARTITION BY user_id ORDER BY event_time
  ) AS session_id
FROM flagged;

DataCamp's 2026 Databricks guide flags this as one of the most repeated hard SQL questions across data-platform loops, and the reason is structural: it forces you to chain two window function patterns (LAG for gap detection, cumulative SUM for grouping) rather than apply one pattern in isolation. Candidates who've only practiced single window functions stall on the second step.

Combine a doubly linked list and a HashMap as in a standard LRU cache, but add a ReentrantReadWriteLock: read lock for get operations (to allow parallel reads), write lock for put and eviction. For TTL, store the expiry timestamp alongside each value in the cache. On get, check the timestamp; if expired, evict that entry and return miss. A background thread (daemon thread) can periodically scan for and evict stale entries, using the write lock during cleanup.

Reported from the 2025 Databricks concurrency round. The interviewer probed locking granularity: "Why a read-write lock rather than a simple mutex?" The expected answer: reads are far more frequent than writes in a cache workload, and a mutex serializes all reads unnecessarily. The read-write lock allows concurrent reads and only blocks when a write is in progress.

import threading
import time
from collections import OrderedDict

class TTLLRUCache:
    def __init__(self, capacity: int, ttl_seconds: float):
        self.capacity = capacity
        self.ttl = ttl_seconds
        self.cache = OrderedDict()  # key -> (value, expiry_time)
        self._lock = threading.RLock()

    def _is_expired(self, key) -> bool:
        return time.monotonic() > self.cache[key][1]

    def get(self, key):
        with self._lock:
            if key not in self.cache:
                return -1
            if self._is_expired(key):
                del self.cache[key]
                return -1
            self.cache.move_to_end(key)
            return self.cache[key][0]

    def put(self, key, value):
        with self._lock:
            expiry = time.monotonic() + self.ttl
            if key in self.cache:
                self.cache.move_to_end(key)
            self.cache[key] = (value, expiry)
            if len(self.cache) > self.capacity:
                self.cache.popitem(last=False)

Salting. Add a random integer suffix to the skewed join key on the large side, then explode the small side into one copy per possible salt value, and join on the combined (key, salt) pair. This spreads the one dominant key's rows across multiple partitions instead of funneling all of them into a single executor that never finishes while every other task sits idle.

from pyspark.sql import functions as F

NUM_SALTS = 20

skewed_salted = skewed_df.withColumn(
    "salt", (F.rand() * NUM_SALTS).cast("int")
)

small_salted = small_df.withColumn(
    "salt", F.explode(F.array([F.lit(i) for i in range(NUM_SALTS)]))
)

joined = skewed_salted.join(
    small_salted,
    on=["join_key", "salt"],
    how="inner",
)

AQE has built-in skew join handling since Spark 3.0 (spark.sql.adaptive.skewJoin.enabled), which splits an oversized partition automatically without any code change. The follow-up interviewers ask: "when would you still salt manually?" When the skewed side is itself the larger table and broadcasting isn't an option, or when you're on a version or config where AQE's skew handling isn't active.

Structured Streaming tracks progress through a checkpointed write-ahead log of offsets and processes each micro-batch deterministically: if a batch fails partway through, the exact same input range gets replayed on retry, not a new range. That guarantees exactly-once processing on the read side by itself.

End-to-end exactly-once only holds if the sink is also idempotent. Writing to Delta Lake through foreachBatch with a MERGE keyed on a unique event ID gives you that; a plain append sink will double-write rows on replay after a driver restart, because the append doesn't know a previous attempt already wrote some of that batch. Watermarking (withWatermark) bounds how late an event can arrive before Structured Streaming drops it from state, which is what keeps a stateful aggregation like sessionization from growing unbounded, but it also means data that arrives past the watermark is silently discarded, not queued. Interviewers want to hear that trade-off named directly, not assumed away.

Two approaches depending on what the interviewer emphasizes. If they want simplicity: a circular buffer of size 300 (one slot per second). Each slot stores the timestamp of the last write to that slot and the count at that timestamp. On hit, update the slot if the timestamp matches, else reset it. O(1) get and record. If they want sorted timestamps with range queries: store timestamps in a deque, evict entries older than 300 seconds on each call, and return the length. O(n) in the worst case but fine for real-world hit rates.

Reported in the OA and phone screen. The follow-up for the circular buffer approach: "What if you need per-minute breakdown, not just the 5-minute aggregate?" That requires bucketing at minute granularity rather than second granularity, adjusting the buffer size to 5 and the comparison window to 60 seconds per bucket.

Reported specifically from a Databricks onsite (interviewing.io account). The product fetches prices from multiple third-party APIs with different latency and reliability profiles, deduplicates results by ISBN, and returns a ranked list. Core design: async fan-out to all distributors with a configurable timeout; cache results in Redis with a TTL based on how often prices change (negotiated with product; probably 5 to 15 minutes); return partial results if some distributors time out rather than blocking on the slowest one.

The follow-ups were about consistency: "What if two distributors return different prices for the same ISBN from the same catalog?" And about scale: "What if you have 50 million ISBNs and 20 distributors polling every 10 minutes?" That last case pushes you toward a pull-based queue of refresh jobs rather than synchronous fan-out on every user request.

Sliding window counter with Redis as the shared state store. Each client has a key in Redis. On each request, use INCR and EXPIRE, or a sorted set with score as timestamp and ZREMRANGEBYSCORE to evict expired entries. The sliding window (as opposed to fixed window) prevents the burst at window boundaries that fixed-window approaches allow.

For millions of RPS, a single Redis instance becomes the bottleneck. Shard by client ID (consistent hashing across a Redis cluster). For fairness across clients, use token bucket per client rather than a global counter. Handle network failures with local in-memory fallback counters: if Redis is unreachable, apply a more conservative limit locally and log the degraded state. Reported from a 2025 Databricks onsite. The interviewer specifically asked about request bursts and what happens when a client sends 10x their limit in the first 100ms of a window.

A job scheduler for Spark needs to treat cluster resources (executor cores, memory per executor) as first-class constraints, not just slot counts. Key components: a job queue with priority tiers; a resource manager that tracks available executors per node; a scheduler that matches jobs to resources using a bin-packing algorithm; and a health monitor that detects and reschedules failed tasks.

Spark's own FAIR and FIFO schedulers are worth mentioning as a baseline, then explain what you'd add: preemption (kill lower-priority jobs to free resources for SLA-bound jobs), gang scheduling for jobs that need a minimum number of executors to make progress (a Spark job that needs 10 executors to avoid spilling to disk should wait rather than start with 3 and thrash), and dead-letter handling for tasks that repeatedly fail on the same node, which usually signals hardware issues.

Two planes. Offline: Delta Lake tables computed by scheduled Spark batch jobs, versioned by timestamp, joined "as of" the label's timestamp during training rather than against the latest value, to avoid leaking future information into the training set. Online: a low-latency key-value store, Redis, DynamoDB, or a managed online-tables product, synced from the offline store on a schedule or via CDC, serving single-key lookups in single-digit milliseconds for inference.

The part interviewers dig into is consistency between the two planes when a feature's definition changes, not just its value. The fix is a feature registry that versions the transformation logic itself, so both the batch job and the online sync path recompute from the same definition instead of drifting apart over time as someone quietly edits one pipeline and forgets the other.