Databricks Interview Questions (2026): What They Actually Ask

DENSE_RANK() over a partition by department, ordered by salary descending, then filter to rank 2. DENSE_RANK matters here, not ROW_NUMBER: if two people in a department tie for the top salary, ROW_NUMBER arbitrarily calls one of them rank 1 and the other rank 2, silently giving you the wrong answer for "second highest." DENSE_RANK gives both of them rank 1 and the next distinct salary rank 2, which is what the question actually means.

SELECT name, department, salary
FROM (
 SELECT name, department, salary,
     DENSE_RANK() OVER (
      PARTITION BY department ORDER BY salary DESC
     ) AS rnk
 FROM employees
) ranked
WHERE rnk = 2;

The classic trap version, MAX(salary) WHERE salary less than MAX(salary), breaks the same way on ties and also requires a self-join or subquery to scope per department. Expect to be asked why it breaks, not just to write the version that works.

SUM as a window function ordered by date, with an explicit frame from unbounded preceding to the current row.

SELECT
 sale_date,
 daily_revenue,
 SUM(daily_revenue) OVER (
  ORDER BY sale_date
  ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
 ) AS running_total
FROM daily_sales
ORDER BY sale_date;

Worth knowing cold: if you omit the explicit ROWS clause and leave just ORDER BY, most engines default to a RANGE frame instead of ROWS, and RANGE groups tied order-by values together, so two rows with the same sale_date get summed into the same running total instead of accumulating one row at a time. That's a real bug people ship, not a trick question.

This is the core of the Cross-Functional / Hiring Manager interview. Databricks' engineering prep guide says it is run by an engineering leader, potentially your direct manager, and covers your career path, why you are looking, significant projects and how they tied to company goals. The guide adds that it is not a technical assessment, but you should be ready to explain your projects with technical depth and without internal jargon from your current company.

So pick a project where the hard part was technical, not organisational, and rehearse explaining it to someone outside your team. Expect follow-ups at the architecture level: why this storage choice, why this consistency model, what you would change now. The honest answer to "what was the hardest part?" is usually something you got wrong or had to redo, and that version reads as credible.

Real answers name specific sources: the Databricks engineering blog, the Delta Lake changelog, the Apache Spark release notes, Martin Kleppmann's "Designing Data-Intensive Applications" as a reference, specific papers (the original Dremel paper, the Bigtable paper, the Lakehouse paper from CIDR 2021). Generic answers like "I follow tech news and read blogs" are thin. If you are on the database track, this matters twice: Databricks runs a separate Database Papers interview (covered below) where you discuss a paper in depth.

A mentoring and communication check, and a common one given how much of the actual job is explaining tricky concurrency reasoning to teammates who didn't build the system. Strong answers name the specific misconception, usually assuming a distributed operation is atomic when it isn't, or assuming a network call always succeeds, describe how you explained it (an analogy, a small reproducible example, tracing a failure case on a whiteboard), and how you confirmed they actually got it rather than just nodded.

Databricks' own FAQ says the timeline typically ranges from two to three months, depending on role, region and hiring team, and that holidays, executive hiring, offsites and travel can stretch it. Third-party guides such as interviewing.io describe up to eight weeks, so treat the shorter figure as the best case rather than the norm.

Two official details help you plan. Databricks aims to share feedback within 48 hours of your final interview, and says to contact your recruiter if you have heard nothing after a week or need a faster answer because of a competing offer. For back-end engineers, the guide also lists Team Matching and a Hiring Committee as internal steps between the panel and references, which is where most of the waiting happens.

Yes. Databricks' engineering prep guide lists a System Programming interview for back-end and database engineers. It covers multi-threading, synchronisation, I/O and performance, asks you to design and implement system components in pseudocode, and covers both single-threaded and multi-threaded approaches. The guide says specific language proficiency is not required, but you should be ready to write detailed pseudocode and explore trade-offs.

It is also graded on code design, not just correctness: separation of concerns, cohesion, coupling and the single responsibility principle are named in the description. Third-party reports, including interviewing.io, call this the concurrency round and flag it as one of the harder parts of the loop. Give it the same prep time as a coding round.

Not for the general software engineering loop. Databricks says its engineering interviews are language agnostic, but it expects fluency in the language you choose: core data structures, performance and memory, and language constructs. None of the back-end interview descriptions require Spark code.

Distributed systems knowledge still shows up. The Architecture interview explicitly covers storage systems, distributed computing, replication, consistency models and failure handling, and the Domain Deep Dive is built around scaling problems you have solved. Knowing what Spark and Delta Lake do underneath (shuffles, partitioning, the transaction log) gives you better material for both. Data engineering, field engineering and database roles lean further into the platform itself.

Databricks does not publish a difficulty level. What it does publish is how the two coding-style interviews differ. The Coding interview is implementation-focused: working code with clean organisation, test coverage, careful edge cases and a Big-O analysis. The Algorithm interview is about fundamentals and data structure choice, again with time and space complexity and a testing strategy.

interviewing.io reports LeetCode medium to hard, with hard appearing more than medium. The safer reading of both sources: solve medium and hard problems cleanly, write tests as you go, and say your complexity out loud. Coding happens in CoderPad, and Databricks links its CoderPad sandbox from the guide, so practise in that editor rather than your own IDE.

Yes. Databricks lists a separate set for Database Software Engineers: Coding, Algorithms, Architecture or Storage, System Programming, Domain Deep Dive, Database Papers and Cross-Functional. The Storage interview is an open-ended design session for the storage layer team, and a Stream Processing interview covers ingestion pipelines for real-time systems, storage for high-throughput workloads and low-latency query processing.

Front-end and full-stack candidates get a different set again (Front-End Code, Front-End Systems, Product Design, Front-End Infrastructure, Cross-Functional). Ask your recruiter which track your loop follows before you plan your prep.

Databricks' FAQ says the onsite loop typically consists of four to six interviews with various team members. For back-end engineers the engineering guide lists the full sequence as a 30-minute recruiter screen, a one-hour technical screen, team matching, a full panel of four to six interviews at one hour each, a hiring committee, references, then the offer.

All interviews are virtual on Google Meet unless your recruiter says otherwise. The Architecture interview uses CoderPad Draw as a whiteboard, per the April 2025 guide. Older third-party write-ups describe system design in Google Docs, so confirm the tool with your recruiter and practise both drawing and writing out a design in prose.

Databricks' interview prep page lists best practices meant to protect you, your current or former employer and Databricks. Do not bring or share any trade secret or confidential information from a current or past employer, and remember your employer likely owns anything you built while working there.

The practical rules: do not use your current work computer or employer materials to create any Databricks assignment or presentation, do not bring a work computer to a Databricks office or connect one to Databricks Wi-Fi, do not forward employer information, and do not attend internal Databricks events while employed elsewhere. It also suggests checking your contract for non-compete or non-solicitation clauses. For the Domain Deep Dive, this means describing past systems in general terms rather than showing internal documents.

Both appear in Databricks' back-end engineering guide as internal review steps, each marked "nothing candidates need to prepare for". Team Matching comes after the technical screen and before the full panel. The Hiring Committee comes after the panel and before references.

What this means for you: the panel may include people from the team you were matched to, and the decision is made by a committee reading all your interviewers' feedback, not by one person. A weak round is not automatically fatal, and a strong one does not carry the others. If the process goes quiet after the panel, the committee step is the usual reason.

DBFS was Databricks' file system layer over the cloud object storage behind a workspace (S3, ADLS Gen2 or GCS), giving notebooks and jobs POSIX-like paths such as /mnt or /FileStore instead of cloud SDK calls. Teams used it for uploaded datasets, mounted buckets, library jars and init scripts.

The current answer matters more than the history. Databricks' documentation now says DBFS root and DBFS mounts are deprecated and not recommended, that new accounts are provisioned without access to them, and that you should use Unity Catalog volumes, external locations or workspace files instead. In an interview for a data engineering or solutions role, explaining why (governance: mounts sit outside Unity Catalog's access control and audit) is stronger than describing /FileStore.

An all-purpose cluster is what you spin up for interactive work, attaching notebooks, running ad hoc queries, exploring a dataset cell by cell. It stays up until someone terminates it or an auto-termination policy kicks in, and multiple users and notebooks can attach to the same cluster at once, which is convenient for collaboration but means one person's runaway query can slow down everyone else's session.

A job cluster gets created automatically when a scheduled job runs, spins up with exactly the config that job specifies, runs only that job's tasks, and terminates the moment the job finishes. Nobody attaches to it interactively. Job compute is priced at a lower DBU rate than all-purpose compute, and since every run gets a fresh cluster, you avoid noisy-neighbor problems or leftover state bleeding from one run into the next. A common mistake is pointing a production job at an all-purpose cluster to save setup time, which works fine until that cluster gets reused for something else and the job either fails or starves for resources next to unrelated interactive queries.

Unity Catalog is Databricks' governance layer for data and AI assets across an entire account, not just one workspace. Before it existed, access control lived at the workspace level through the Hive metastore, so ten workspaces meant ten separate places to define who could read which table, with no single audit trail spanning all of them. Unity Catalog introduces a three-level namespace, catalog.schema.table, and lets one metastore attach to many workspaces, so permissions, lineage, and audit logs stay consistent no matter which workspace someone connects from.

Concretely, it centralizes access control with GRANT and REVOKE statements that read like standard SQL permissions instead of workspace-specific ACLs, it generates automatic column and table level lineage so you can see what fed a table and what consumes it, and it gives admins one place to manage external locations and storage credentials instead of every workspace owner configuring their own mount points. It also underpins row filters and column masks, which is what most companies actually need once a table mixes PII with columns that are fine for a broader audience to see.

An RDD, resilient distributed dataset, is Spark's original low-level abstraction, a distributed collection of JVM objects with no schema, where transformations are written as lambdas and Spark has no idea what's inside those objects until it actually runs them. That means no query optimization beyond the transformation graph itself, since the optimizer can't reason about columns it can't see.

A DataFrame is a distributed collection of rows with a known schema, similar to a table. Because Spark knows the column names and types up front, the Catalyst optimizer can push down filters, reorder joins, and rewrite the plan before anything executes, which is why DataFrame code is almost always faster than the equivalent hand-written RDD code for anything past a trivial map or filter. The tradeoff is some loss of compile-time type safety, since a DataFrame is really Dataset[Row] beneath that in Scala, and a typo in a column name only shows up at runtime.

A Dataset adds compile-time type safety on top of the same execution engine, available in Scala and Java, where you work with typed case classes and the compiler catches a schema mismatch before you run anything. PySpark has no separate Dataset API because Python isn't statically typed, so in PySpark you work with DataFrames the whole time, and when someone says "Dataset" in a PySpark context they usually just mean DataFrame.

Convert each CIDR block to a network address and mask. For a CIDR like 192.168.1.0/24, the mask has 24 ones followed by 8 zeros. Apply the mask to both the target IP and the network address; if they match, the IP is in the range. Parse dotted-decimal notation into 32-bit integers for clean bitwise comparison.

interviewing.io lists an IP-to-CIDR problem among the questions Databricks candidates report. The key insight is that you're doing a bitwise AND with the subnet mask, not string comparison. String-based solutions tend to break on edge cases late in the session, so convert to integers from the start, and test /0, /32 and an address just outside the range.

def ip_in_cidr(ip: str, cidr: str) -> bool:
  def ip_to_int(ip_str):
    parts = list(map(int, ip_str.split('.')))
    return (parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8) | parts[3]

  network, prefix_len = cidr.split('/')
  prefix_len = int(prefix_len)
  mask = ((1 << 32) - 1) ^ ((1 << (32 - prefix_len)) - 1)

  ip_int = ip_to_int(ip)
  net_int = ip_to_int(network)

  return (ip_int & mask) == (net_int & mask)

def ip_in_any_cidr(ip: str, cidrs: list) -> bool:
  return any(ip_in_cidr(ip, cidr) for cidr in cidrs)

The naive approach checks all rows, columns, and diagonals after each move, O(n) per move. The efficient approach maintains running sums per row, column, and the two diagonals. On each move, increment the relevant counters. When any counter reaches n, the mover wins. This gets the per-move complexity down to O(1).

interviewing.io lists a variable-size Tic-Tac-Toe implementation among reported Databricks questions. It fits the Coding interview Databricks describes, which grades clean code organisation, tests and edge cases, so write a small class with a move method, validate out-of-range and repeated moves, and add a few test calls. A natural follow-up is generalising to k-in-a-row or more dimensions; sketching how the counters change is usually enough.

Min-heap of size k. Push the first element of each list along with its list index. Pop the minimum, append it to the result, then push the next element from whichever list that minimum came from. Runs in O(N log k) where N is the total element count across all lists, since every push and pop touches a heap of size k rather than the full N.

LeetCode 23, framed around logs. It is good practice for the Algorithm interview, where Databricks asks you to justify data structure choice and complexity. A divide-and-conquer pairwise merge gets the same O(N log k) bound with no heap at all; naming both and picking one with a stated reason is the kind of trade-off discussion the guide describes.

import heapq

def merge_k_sorted(lists: list) -> list:
  heap = []
  for i, lst in enumerate(lists):
    if lst:
      heapq.heappush(heap, (lst[0], i, 0))

  result = []
  while heap:
    val, list_idx, elem_idx = heapq.heappop(heap)
    result.append(val)
    if elem_idx + 1 < len(lists[list_idx]):
      next_val = lists[list_idx][elem_idx + 1]
      heapq.heappush(heap, (next_val, list_idx, elem_idx + 1))
  return result

Maintain a min-heap capped at size k. Every new value gets pushed; if the heap grows past k, pop the smallest. The top of the heap is always the kth largest value seen so far, and each insert costs O(log k) instead of resorting the entire history.

import heapq

class KthLargest:
  def __init__(self, k: int, nums: list):
    self.k = k
    self.heap = list(nums)
    heapq.heapify(self.heap)
    while len(self.heap) > k:
      heapq.heappop(self.heap)

  def add(self, val: int) -> int:
    heapq.heappush(self.heap, val)
    if len(self.heap) > self.k:
      heapq.heappop(self.heap)
    return self.heap[0]

LeetCode 703. The useful framing is a monitor tracking the kth-highest CPU sample across nodes without storing full history, because you can't afford to keep every sample in memory. Copy the input list rather than heapifying the caller's list in place; mutating an argument is the kind of detail an implementation-focused interview notices.

GROUP BY every column HAVING COUNT(*) greater than 1 is the textbook answer, and it's the wrong one at this scale: grouping on every column of a multi-terabyte table forces a full shuffle across all of them. The usual pattern is ROW_NUMBER partitioned by a natural key, ordered by ingestion timestamp descending, keeping only rank 1.

SELECT *
FROM (
 SELECT *,
     ROW_NUMBER() OVER (
      PARTITION BY event_id
      ORDER BY ingestion_ts DESC
     ) AS rn
 FROM raw_events
) deduped
WHERE rn = 1;

A standard data engineering question. The follow-up is usually "how would you prevent the duplicates instead of cleaning them up after the fact," which points toward an idempotent MERGE INTO keyed on event_id at write time rather than a nightly dedup job.

A correlated subquery re-evaluates for every row of the outer query, which on a large table can mean re-scanning (or at least re-planning against) the inner table once per outer row. Pre-aggregating the inner query once and joining on the result computes the same thing a single time.

-- Slow: correlated subquery re-evaluated per outer row
SELECT o.order_id, o.customer_id
FROM orders o
WHERE o.total > (
 SELECT AVG(o2.total)
 FROM orders o2
 WHERE o2.customer_id = o.customer_id
);

-- Faster: pre-aggregate once, then join
SELECT o.order_id, o.customer_id
FROM orders o
JOIN (
 SELECT customer_id, AVG(total) AS avg_total
 FROM orders
 GROUP BY customer_id
) avgs
 ON avgs.customer_id = o.customer_id
WHERE o.total > avgs.avg_total;

Spark's Catalyst optimizer can decorrelate some subqueries automatically, so the rewritten version doesn't always run faster in practice. The stronger answer reads the EXPLAIN plan and reasons about its shape, not just about whether the query returns the right rows.

The naive version acquires a mutex before writing, releases after. That's correct but the interviewer will push: "What if logging is slow because it's writing to disk? You're blocking all threads on I/O." The expected step-up: a bounded blocking queue as a write buffer. Each writer enqueues a log entry (fast, in-memory, with a lock); a dedicated logger thread dequeues and writes to disk. This decouples write throughput from disk I/O latency.

interviewing.io lists "implement an efficient logger that processes messages in a queue" for the Databricks concurrency round, and it matches the System Programming description (multi-threading, buffering, I/O). Follow-up: "What happens if the queue fills up?" Options: block the producer (backpressure), drop logs with a counter (lossy but non-blocking), or flush synchronously when the queue is full. The trade-off is availability vs. completeness of logs. Name it explicitly, and say which you'd pick for audit logs versus debug logs.

import threading
import queue

class AsyncLogger:
  def __init__(self, maxsize=1000):
    self._q = queue.Queue(maxsize=maxsize)
    self._stop = threading.Event()
    self._worker = threading.Thread(target=self._flush_loop, daemon=True)
    self._worker.start()

  def log(self, message: str):
    try:
      self._q.put_nowait(message)
    except queue.Full:
      pass # drop under backpressure; counter omitted for brevity

  def _flush_loop(self):
    while not self._stop.is_set() or not self._q.empty():
      try:
        msg = self._q.get(timeout=0.05)
        print(msg) # replace with file write in production
        self._q.task_done()
      except queue.Empty:
        continue

  def shutdown(self):
    self._stop.set()
    self._worker.join()

BFS over the URL graph, but parallelized. A shared visited set (protected by a lock or using a ConcurrentSet equivalent) tracks fetched URLs. A thread pool processes URLs from a work queue. Each thread fetches a page, extracts links, filters already-visited URLs, and enqueues new ones. The tricky part is termination: you need to know when all work is complete and the queue is empty, not just when the queue is momentarily empty between enqueues.

LeetCode 1242, and good rehearsal for the System Programming interview, which covers single-threaded and multi-threaded approaches to the same component. The termination condition is the part that usually goes wrong. Use an active-task counter alongside the queue: increment when enqueuing, decrement when a task completes. Signal completion when both the counter reaches zero and the queue is empty.

HashMap's internal state (the bucket array, the size counter, and the rehash logic) is not atomically updated. Two threads calling put simultaneously can corrupt the internal linked list during a bucket collision, or both trigger a rehash and lose each other's writes. In Java 7 and earlier, two concurrent puts during a resize could even create a cycle in a bucket's linked list and hang a later get. Options in Java: ConcurrentHashMap (fine-grained per-bin locking plus CAS since Java 8, O(1) per operation at low contention); Collections.synchronizedMap (single lock, simple but lower throughput); or explicit ReadWriteLock if reads dominate. In CPython with the GIL, single dict operations are effectively atomic, but compound read-modify-write sequences are not.

A conceptual warm-up that fits the System Programming interview's focus on synchronisation. The Python nuance matters here: Python candidates often claim dict is thread-safe and they're right for atomic operations but wrong for compound operations like "check-then-insert."

A shuffle occurs when Spark needs to redistribute data across partitions, typically triggered by operations like groupBy, join, or distinct. The process: map-side tasks write their output partitioned by the shuffle key to local disk files; reduce-side tasks fetch the relevant partitions from all map tasks over the network and re-sort the data for the downstream operation.

The expense comes from three sources: disk I/O (writing intermediate shuffle files and reading them back), network I/O (all-to-all data transfer between executors, which scales as O(m x r) where m is map tasks and r is reduce tasks), and GC pressure from materializing large shuffle blocks in memory. Mitigation: reduce shuffle size through predicate pushdown and projection before the shuffle operation; use broadcast joins to eliminate shuffles entirely when one side is small enough to fit in executor memory (typically under 10MB by default in Spark, configurable).

Delta Lake stores metadata as a series of JSON log files in the _delta_log directory alongside the data files (Parquet). Each transaction appends a new JSON entry to the log describing which files were added, which were removed, and what schema was applied. The current state of the table is derived by replaying the log from the last checkpoint forward.

ACID compliance works as follows. Atomicity: either all files in a transaction are committed to the log or none are. Consistency: schema enforcement validates each write against the declared schema before committing. Isolation: optimistic concurrency control with conflict detection during the commit protocol, two writers who attempt to modify the same files will fail the one that commits second with a conflict error. Durability: once the log entry is written, the transaction is durable. A likely follow-up is what happens when two Spark jobs write concurrently and how the conflict reaches the application (a concurrent modification exception it must retry or surface).

Z-ordering is a multi-dimensional data clustering technique that co-locates related data in the same set of files. When you run OPTIMIZE... ZORDER BY (column_a, column_b), Delta Lake rewrites the data files so that rows with similar values in those columns are stored together. Subsequent queries with filters on those columns can skip files that don't contain relevant ranges, a technique called data skipping.

Use it when you have high-cardinality columns that frequently appear in query filters and you're not partitioning by those columns (because partitioning by a high-cardinality column creates too many small files). The trade-off: OPTIMIZE with Z-ordering is an expensive write operation. It's appropriate for tables that are read much more often than written, and where file-skipping savings exceed the rewrite cost. Not appropriate for streaming tables with continuous low-latency writes.

Know the current answer too. Databricks' documentation (updated 25 September 2026) describes liquid clustering as replacing table partitioning and ZORDER, and recommends it for all new tables, including streaming tables. Explaining Z-ordering and then saying you would use liquid clustering on a new Databricks table shows you know the platform as it is now.

Catalyst is Spark SQL's query optimizer. It parses your query into a logical plan, applies rule-based transformations (predicate pushdown, constant folding, column pruning), then generates one or more physical plans and picks the cheapest one using statistics gathered before the job runs. All of that happens once, up front, based on estimates.

Adaptive Query Execution, introduced in Spark 3.0 and on by default since Spark 3.2.0, re-optimizes mid-flight using the actual statistics a completed stage produced. It can flip a sort-merge join to a broadcast join once it sees the real size of one side, coalesce shuffle partitions that turned out smaller than expected, and split a partition that turned out skewed. The follow-up worth preparing for: "what can't AQE fix?" AQE re-plans between stages, so it can't rescue a single task that's already OOMing inside a stage that hasn't finished yet. That's still a query-design problem, not a runtime one.

A broadcast join sends a full copy of the smaller table to every executor up front, so the join happens locally with no shuffle at all. The default size threshold is 10MB (spark.sql.autoBroadcastJoinThreshold), and some teams raise it for wide dimension tables once they've confirmed driver and executor memory can absorb it.

from pyspark.sql.functions import broadcast

result = large_orders_df.join(
  broadcast(small_dim_table_df),
  on="dim_key",
  how="inner",
)

The trade-off interviewers push on: broadcasting a table that's actually too large duplicates it across every executor simultaneously, which can be worse than the shuffle it was meant to avoid. AQE will convert a sort-merge join to a broadcast join at runtime if it detects, after the fact, that one side turned out small enough, even if the static plan picked shuffle originally.

cache() is shorthand for persist() with the default storage level. For DataFrames that default is MEMORY_AND_DISK_DESER in current PySpark (the docs note it changed to match Scala in Spark 3.0), while the older RDD API defaults to MEMORY_ONLY, a distinction interviewers sometimes probe. persist() lets you choose explicitly: memory only, disk only, or replicated levels (the _2 suffix) that keep a second copy on another node.

from pyspark.storagelevel import StorageLevel

df.cache()                          # default level, MEMORY_AND_DISK_DESER
df.persist(StorageLevel.MEMORY_ONLY)  # evicted blocks are recomputed
df.persist(StorageLevel.DISK_ONLY_2)  # replicated to 2 nodes, disk only

PySpark has no _SER levels; those exist in the Scala and Java APIs. A cached DataFrame is recomputed from lineage in three common cases: an executor holding cached blocks is lost, blocks are evicted under memory pressure at a memory-only level, or the query you run does not match the cached plan (a different filter or column set applied before the cache point). That lineage fallback is what makes Spark fault-tolerant without synchronously replicating every cached dataset.

Databricks says its behavioral interviews evaluate every candidate against the same core competencies, using prompts like "Tell me about a time". For this one, give the specific technical disagreement, not just the interpersonal resolution. Describe what the two positions were, what evidence you brought, what evidence they brought, and what the outcome actually was. "We talked it out and found a middle ground" is a non-answer here. The interviewers want to know whether the resolution was technically correct and whether you were willing to be wrong.

The follow-up is frequently: "Looking back, who was right?" Being honest about a case where your colleague was right is more credible than a story where you were right and they came around. Showing that you can update your view matters as much as showing you can hold a position.

This question is about engineering judgment under change, not about your project management skills. Interviewers want to hear: what was the original design? What specifically changed? What was the cost of the original design in the new world? What did you do, and what did you defer? The answer should have specific technical content, not just "we re-prioritized."

A strong answer names concrete trade-offs. "We had built the pipeline assuming append-only writes; when requirements changed to include deletes, we had to introduce a CDC layer. We chose to accept eventual consistency on the delete path for 48 hours rather than block the existing functionality while rebuilding."

This is a diagnostic reasoning question. The best answers have a structured approach: first, confirm the problem is real and reproducible; second, narrow the scope by checking whether the problem is isolated to one node, one partition, one job, or one user; third, examine logs and metrics for anomalies at the time the failure started; fourth, form a hypothesis and design a targeted test. Generic answers like "I look at the logs and ask colleagues" don't land well.

Tool-specific detail helps here. Mentioning Spark UI for DAG visualization, executor-level logs for task failures, and Delta Lake transaction log inspection for data integrity issues shows that you've actually worked with distributed systems rather than just studied them.

This one probes data skepticism directly, which matters more at a data infrastructure company than most. Interviewers want the specific tip-off: a number that didn't reconcile against a second source, a metric that moved with no corresponding event upstream. Then what you actually did to verify it, traced the pipeline back to the raw source, checked for a silent schema change, diffed against an independent report, not just "I raised a flag."

Weak answers stop at "I told the team something looked off." Strong ones name what was actually broken: a silent schema evolution in an upstream topic, a timezone bug in a date-truncation step, a join that fanned out rows for weeks before anyone noticed the totals were inflated.

Name the trade-off explicitly. What was the shortcut, what would the "right" version have cost in time, and what did the debt actually cost downstream, a manual reconciliation someone ran weekly, a job that hit a wall at a certain volume and needed a rewrite anyway. A good answer also says whether the debt got paid down later, and if not, why not.

Less about the emotional response, more about triage under pressure. Interviewers want your decision process for fix-forward versus rollback, how you communicate status while still mid-debug (a short update every 15 to 20 minutes beats silence, even if the update is "still investigating"), and what the postmortem looked like afterward.

One detail that signals real experience: whether you mention checkpoint recovery for Structured Streaming jobs specifically, can you just restart from the last checkpoint, or did the checkpoint itself get corrupted, which turns a bad night into a much worse one. Raising that yourself, before being asked, shows you have operated a streaming job rather than read about one.

Databricks describes it as a seminar-style interview for engineers with database experience, focused on query optimisation and execution strategies. It is open-book: you can refer to your notes and the original paper during the discussion. Expect questions on storage optimisation techniques, runtime performance improvements and general database performance methods.

Because it is open-book, recall is not what's tested. Read the paper for its assumptions and limits: what workload it was designed for, what it trades away, and what you would change on modern cloud object storage. Ask your recruiter which paper, if they tell you; if not, the Lakehouse paper (CIDR 2021) and the Spark SQL paper (SIGMOD 2015) are sensible starting points for a Databricks panel.

Per Databricks' engineering guide, the Front-End Code interview has you build a browser-based application that fetches and displays remote data, testing asynchronous operations, efficient data fetching and state management across components. Supported options are vanilla JS/DOM, React, Angular, Vue and Svelte; if you want another framework, Databricks asks you to learn how to configure it in CoderPad beforehand.

Prepare the unglamorous parts: loading and error states, cancelling or ignoring stale requests when the user types quickly, pagination, and not refetching data you already have. The separate Front-End Systems interview is open-ended and includes hands-on coding, so expect to clarify requirements before designing.

LAG gives you the previous event time per user. Flag a new session whenever the gap since the prior event exceeds 30 minutes, or when there is no prior event at all. Then a cumulative SUM of that flag, ordered by time, produces a stable session ID per user.

WITH gaps AS (
 SELECT
  user_id,
  event_time,
  LAG(event_time) OVER (
   PARTITION BY user_id ORDER BY event_time
  ) AS prev_event_time
 FROM user_events
),
flagged AS (
 SELECT
  user_id,
  event_time,
  CASE
   WHEN prev_event_time IS NULL
    OR event_time > prev_event_time + INTERVAL 30 MINUTES
   THEN 1 ELSE 0
  END AS new_session
 FROM gaps
)
SELECT
 user_id,
 event_time,
 SUM(new_session) OVER (
  PARTITION BY user_id ORDER BY event_time
 ) AS session_id
FROM flagged;

A common hard SQL question for data-focused roles, and the difficulty is structural: it forces you to chain two window function patterns (LAG for gap detection, cumulative SUM for grouping) rather than apply one pattern in isolation. Candidates who've only practiced single window functions stall on the second step.

Combine a doubly linked list and a hash map as in a standard LRU cache, and store an expiry timestamp alongside each value. On get, check the timestamp; if expired, evict that entry and return a miss. A background thread can also sweep stale entries periodically, holding the lock while it does.

The locking follow-up is the interesting part, and it fits the System Programming interview's focus on synchronisation and caching. A read-write lock looks attractive because caches are read-heavy, but an LRU get is not a pure read: it moves the key to the front of the recency list, so it needs exclusive access too. Options are a single lock (simple, what the code below does), lock striping across shards of the key space, or approximate recency (sampling or a clock algorithm) so reads don't have to reorder anything.

import threading
import time
from collections import OrderedDict

class TTLLRUCache:
  def __init__(self, capacity: int, ttl_seconds: float):
    self.capacity = capacity
    self.ttl = ttl_seconds
    self.cache = OrderedDict() # key -> (value, expiry_time)
    self._lock = threading.RLock()

  def _is_expired(self, key) -> bool:
    return time.monotonic() > self.cache[key][1]

  def get(self, key):
    with self._lock:
      if key not in self.cache:
        return -1
      if self._is_expired(key):
        del self.cache[key]
        return -1
      self.cache.move_to_end(key)
      return self.cache[key][0]

  def put(self, key, value):
    with self._lock:
      expiry = time.monotonic() + self.ttl
      if key in self.cache:
        self.cache.move_to_end(key)
      self.cache[key] = (value, expiry)
      if len(self.cache) > self.capacity:
        self.cache.popitem(last=False)

Salting. Add a random integer suffix to the skewed join key on the large side, then explode the small side into one copy per possible salt value, and join on the combined (key, salt) pair. This spreads the one dominant key's rows across multiple partitions instead of funneling all of them into a single executor that never finishes while every other task sits idle.

from pyspark.sql import functions as F

NUM_SALTS = 20

skewed_salted = skewed_df.withColumn(
  "salt", (F.rand() * NUM_SALTS).cast("int")
)

small_salted = small_df.withColumn(
  "salt", F.explode(F.array([F.lit(i) for i in range(NUM_SALTS)]))
)

joined = skewed_salted.join(
  small_salted,
  on=["join_key", "salt"],
  how="inner",
)

AQE has built-in skew join handling since Spark 3.0 (spark.sql.adaptive.skewJoin.enabled), which splits an oversized partition automatically without any code change. The follow-up interviewers ask: "when would you still salt manually?" When the skewed side is itself the larger table and broadcasting isn't an option, or when you're on a version or config where AQE's skew handling isn't active.

Structured Streaming tracks progress through a checkpointed write-ahead log of offsets and processes each micro-batch deterministically: if a batch fails partway through, the exact same input range gets replayed on retry, not a new range. That guarantees exactly-once processing on the read side by itself.

End-to-end exactly-once only holds if the sink is also idempotent. Writing to Delta Lake through foreachBatch with a MERGE keyed on a unique event ID gives you that; a plain append sink will double-write rows on replay after a driver restart, because the append doesn't know a previous attempt already wrote some of that batch. Watermarking (withWatermark) bounds how late an event can arrive before Structured Streaming drops it from state, which is what keeps a stateful aggregation like sessionization from growing unbounded, but it also means data that arrives past the watermark is silently discarded, not queued. Interviewers want to hear that trade-off named directly, not assumed away.

Two approaches depending on what the interviewer emphasizes. If they want simplicity: a circular buffer of size 300 (one slot per second). Each slot stores the timestamp of the last write to that slot and the count at that timestamp. On hit, update the slot if the timestamp matches, else reset it. O(1) get and record. If they want sorted timestamps with range queries: store timestamps in a deque, evict entries older than 300 seconds on each call, and return the length. O(n) in the worst case but fine for real-world hit rates.

A common warm-up design question. A natural follow-up for the circular buffer approach: "What if you need per-minute breakdown, not just the 5-minute aggregate?" That requires bucketing at minute granularity rather than second granularity, adjusting the buffer size to 5 and the comparison window to 60 seconds per bucket.

interviewing.io lists a design question about a service that finds customers the cheapest copy of a book they're searching for. The product fetches prices from multiple third-party APIs with different latency and reliability profiles, deduplicates results by ISBN, and returns a ranked list. Core design: async fan-out to all distributors with a configurable timeout; cache results in Redis with a TTL based on how often prices change (negotiated with product; probably 5 to 15 minutes); return partial results if some distributors time out rather than blocking on the slowest one.

Likely follow-ups are about consistency: "What if two distributors return different prices for the same ISBN from the same catalog?" And about scale: "What if you have 50 million ISBNs and 20 distributors polling every 10 minutes?" That last case pushes you toward a pull-based queue of refresh jobs rather than synchronous fan-out on every user request.

Sliding window counter with Redis as the shared state store. Each client has a key in Redis. On each request, use INCR and EXPIRE, or a sorted set with score as timestamp and ZREMRANGEBYSCORE to evict expired entries. The sliding window (as opposed to fixed window) prevents the burst at window boundaries that fixed-window approaches allow.

For millions of RPS, a single Redis instance becomes the bottleneck. Shard by client ID (consistent hashing across a Redis cluster). For fairness across clients, use token bucket per client rather than a global counter. Handle network failures with local in-memory fallback counters: if Redis is unreachable, apply a more conservative limit locally and log the degraded state. A useful follow-up to rehearse: what happens when a client sends 10x its limit in the first 100ms of a window, and which algorithm handles that burst the way you want.

A job scheduler for Spark needs to treat cluster resources (executor cores, memory per executor) as first-class constraints, not just slot counts. Key components: a job queue with priority tiers; a resource manager that tracks available executors per node; a scheduler that matches jobs to resources using a bin-packing algorithm; and a health monitor that detects and reschedules failed tasks.

Spark's own FAIR and FIFO schedulers are worth mentioning as a baseline, then explain what you'd add: preemption (kill lower-priority jobs to free resources for SLA-bound jobs), gang scheduling for jobs that need a minimum number of executors to make progress (a Spark job that needs 10 executors to avoid spilling to disk should wait rather than start with 3 and thrash), and dead-letter handling for tasks that repeatedly fail on the same node, which usually signals hardware issues.

Two planes. Offline: Delta Lake tables computed by scheduled Spark batch jobs, versioned by timestamp, joined "as of" the label's timestamp during training rather than against the latest value, to avoid leaking future information into the training set. Online: a low-latency key-value store, Redis, DynamoDB, or a managed online-tables product, synced from the offline store on a schedule or via CDC, serving single-key lookups in single-digit milliseconds for inference.

The part interviewers dig into is consistency between the two planes when a feature's definition changes, not just its value. The fix is a feature registry that versions the transformation logic itself, so both the batch job and the online sync path recompute from the same definition instead of drifting apart over time as someone quietly edits one pipeline and forgets the other.

This mirrors the Stream Processing interview Databricks lists for database engineers: ingestion pipelines for real-time systems, storage for high-throughput environments and low-latency query processing, with trade-offs, scalability and fault tolerance. A workable outline: a partitioned log (Kafka or similar) as the entry point, stateless enrichment, then a streaming writer that commits small batches to a table format with atomic commits so readers never see half a batch.

The trade-off to name is freshness against file count. Committing every second makes data fresh but creates many small files that slow every query, so you need background compaction, and compaction then competes with ingestion for the same table. Be ready to say how you'd handle late events (watermarks and what happens past them), replays after failure (idempotent writes keyed on event ID) and a hot partition when one tenant suddenly sends ten times its usual volume.