Apache Spark Interview Questions (2026): Must-Know Q&A

Spark is a distributed, general-purpose engine for processing data across a cluster of machines, with one core execution engine underneath separate APIs for batch processing, SQL, streaming, machine learning (MLlib), and graph processing (GraphX). It started as a research project at UC Berkeley's AMPLab in 2009 and became a top-level Apache project in February 2014.

The "general-purpose" part matters in interviews. Before Spark, a shop running batch ETL, streaming, and ML each needed its own specialized system. Spark's pitch was one engine that could do all three reasonably well instead of three engines that each did one thing best and nothing else.

An RDD (resilient distributed dataset) is Spark's original abstraction: a distributed collection of JVM or Python objects with no schema Spark can see into. A DataFrame is a distributed collection organized into named columns with a known schema, conceptually closer to a database table or a pandas DataFrame, but spread across a cluster. A Dataset adds compile-time type safety on top of a DataFrame's schema, and it only exists in Scala and Java, because Python's dynamic typing gives a compiler nothing to check.

A PySpark DataFrame is functionally close to Dataset[Row] in Scala, minus the typed part. If a job posting says PySpark, Datasets don't come up as a coding concept, only as a conceptual question about how Scala shops differ.

A transformation builds a new RDD or DataFrame from an existing one, map, filter, select, groupBy, and none of them touch actual data when you call them. An action triggers real computation and returns a result to the driver or writes it somewhere, count(), collect(), show(), write.parquet().

Lineage is the recorded chain of transformations used to build an RDD from its original source, a recipe rather than a copy of the data itself. If a partition is lost, Spark doesn't need a replicated copy sitting somewhere else, it just replays the lineage for that one partition against the source and rebuilds it. That's a deliberate trade-off against something like HDFS, which achieves fault tolerance by physically replicating each block three times instead of recomputing it.

A narrow transformation, map, filter, union, needs data from only one parent partition to compute each output partition, so it runs entirely within an executor with no data movement. A wide transformation, groupByKey, join, repartition, distinct, needs data from multiple partitions, potentially every partition, to compute a single output partition, which forces a shuffle: data gets written out, sent across the network, and read back in by a different set of tasks.

A partition is a chunk of a dataset processed by a single task on a single executor core. After a shuffle, a groupBy or a join, Spark uses spark.sql.shuffle.partitions to decide how many output partitions to create, and the default is 200, a number picked years ago that's usually wrong for both very small and very large jobs.

cache() is just persist() called with a default storage level, nothing more. For an RDD, that default is MEMORY_ONLY. For a DataFrame or Dataset, the default is MEMORY_AND_DISK, deliberately more forgiving, since Spark expects a DataFrame might not fully fit in memory and would rather spill part of it to disk than silently drop it and force a full recompute later.

Catalyst takes DataFrame code or a SQL query through four phases: analysis (resolving column and table names against the catalog), logical optimization (rule-based rewrites like predicate pushdown and constant folding), physical planning (choosing actual join and aggregation strategies, then picking the cheapest by cost), and code generation, where Tungsten generates JVM bytecode for the chosen plan instead of interpreting it row by row.

Structured Streaming treats a stream as an unbounded table that keeps growing, and the same DataFrame and SQL operations that work against a static, bounded table run against it unchanged. By default it runs on a micro-batch engine, processing the stream as a series of small batch jobs with end-to-end latency as low as 100 milliseconds (Apache Spark Structured Streaming docs). A separate continuous processing mode, available since Spark 2.3, trades exactly-once for at-least-once guarantees and can push latency down to roughly 1 millisecond, but it only supports a narrower set of operations.

SparkSession is the single entry point introduced in Spark 2.0 that unifies SparkContext, SQLContext, and HiveContext into one object. Before that you had to juggle a SparkContext for RDD work and a separate SQLContext for DataFrame and SQL work, and if you wanted Hive support you needed HiveContext layered on top.

SparkSession.builder() gives you one object that exposes.sql(),.read(), and.createDataFrame(), and you can still reach the underlying SparkContext through spark.sparkContext when you need low-level RDD operations. In practice almost nobody constructs SparkContext directly anymore outside of legacy RDD-only code.

The driver is the process running your main function or the top of your notebook. It builds the DAG from your transformations, negotiates with the cluster manager for resources, schedules tasks onto executors, and collects results back for actions like collect() or count(). It holds the SparkContext and is a single point that can bottleneck or crash the whole job, for example if you collect() something too large.

Executors are JVM processes that run on worker nodes for the life of the application. Each executor runs the tasks assigned to it, one task per core slot, and keeps data in memory or spills it to local disk when needed. If an executor dies, its tasks get rescheduled elsewhere using lineage, not by restarting the whole job.

A DAG, a directed acyclic graph, is the logical plan of every RDD or DataFrame operation you've chained, built up lazily as you call transformations. Nothing runs until an action triggers it. The DAG scheduler then walks that graph backward from the action and splits it into stages at shuffle boundaries: operations that can run without moving data between partitions stay in one stage, and a new stage starts wherever a shuffle is required.

Each stage is then broken into tasks, one per partition, and those tasks are shipped to executors to run in parallel. So the hierarchy is job, one per action, then stages split at shuffles, then tasks split at partitions. That's what lets you look at a Spark UI page and actually understand why a job has, say, four stages and eight hundred tasks instead of just staring at a number.

Parquet is columnar and self-describing, which matters for a few concrete reasons. It stores a schema inside the file, so Spark doesn't have to infer types by scanning rows the way it does with CSV, where everything can silently end up typed as string. Being columnar also means a query touching 3 of 50 columns can skip reading the other 47 off disk entirely, which row-oriented formats like CSV and JSON can't do.

Parquet also supports predicate pushdown and stores min and max statistics per row group, so filters can skip whole chunks of data without ever decompressing them. It compresses better too, since similar values sit next to each other in a column. CSV and JSON still show up for ingesting data from external systems that only speak those formats, but the first step in most pipelines is converting to Parquet, or something like Delta or ORC, before any real processing happens.

Spark can run on its own standalone cluster manager, on YARN, on Kubernetes, or historically on Mesos, which is now deprecated. Standalone is the simplest to set up, just Spark's own master and worker processes, and works fine for a dedicated Spark cluster with nothing else competing for resources.

YARN shows up most in traditional Hadoop environments since it ships with Hadoop and lets Spark share a cluster with Hive, MapReduce, and other YARN applications. Kubernetes has become the default for cloud-native setups because Spark executors can run as pods alongside everything else already running on the same k8s cluster, using the same autoscaling and quota tooling. The practical thing interviewers care about is usually just whether you know which one your team uses and how deploy-mode and resource requests get configured for it.

spark-submit is the command-line tool used to launch a Spark application, whether the target is standalone, YARN, or Kubernetes. You point it at your jar or Python file, and it bootstraps the driver, ships your code and dependencies to the cluster, and requests executors from the cluster manager.

The arguments you touch most are --master, which picks the cluster manager and mode, --deploy-mode for client or cluster, --num-executors or dynamic allocation settings, --executor-memory and --executor-cores, and --conf for any Spark setting you need to override, like shuffle partitions or the serializer. For a Python job you'd also pass --py-files for extra modules, and for anything hitting external systems you'd add --jars or --packages for the connector.

MapReduce chains jobs by writing intermediate output to HDFS between every map and reduce stage, so an iterative algorithm that touches the same dataset twenty times pays for twenty round trips to disk. Spark represents a whole computation as a single DAG of stages and can keep data in memory across that DAG (spilling to disk only under real memory pressure), so a cached dataset reused across multiple iterations gets read from disk once, not once per pass.

That gap is real, but it isn't free. It's specifically iterative and interactive workloads where Spark pulls ahead. A single-pass batch job that reads once, transforms once, and writes once doesn't get much benefit from in-memory reuse, because there's nothing to reuse.

Two reasons, and interviewers usually want both. First, the Catalyst optimizer only understands DataFrame and Dataset operations; an RDD built from arbitrary Python or Scala lambdas is a black box Spark can't look inside, so it can't push a filter down, reorder a join, or skip reading a column nothing downstream touches. Second, DataFrames use Tungsten's off-heap binary row format instead of full JVM objects, which cuts the garbage collection overhead that used to be one of the biggest tuning headaches in RDD-era Spark.

case class Person(id: Long, name: String, age: Int)

val people: Dataset[Person] = spark.read
.parquet("s3://data/people/")
.as[Person] // compiles only if the schema matches Person

val adults = people.filter(_.age >= 18) // type-checked at compile time

That's the one place Datasets earn their keep in an interview answer: a typo in a field name or a type mismatch fails at compile time in Scala instead of blowing up three hours into a production run.

Deferring execution lets Spark see the entire chain of transformations before running anything, which is what makes the Catalyst optimizer possible for DataFrames. It can reorder a filter to run before an expensive join, skip reading columns nothing downstream touches, or fold several narrow transformations into one pass over the data. Run each line eagerly the moment you call it, and none of that whole-plan optimization is possible, because Spark would only ever see one operation at a time.

No, and this catches people. cache() and persist() are lazy too, they just mark the RDD or DataFrame to be cached the next time an action actually touches it. Call .cache() and never call an action afterward, and nothing gets cached at all.

Three costs stack on top of each other: disk I/O to write shuffle files on the map side, network transfer to move that data between executors, and serialization plus deserialization on both ends. A stage boundary in Spark's DAG is, almost by definition, a shuffle, so counting how many wide transformations sit in your plan is a rough proxy for how many expensive stage boundaries your job actually has.

repartition(n) does a full shuffle and can move a dataset to more or fewer partitions, roughly evenly balanced. coalesce(n) avoids a shuffle by merging adjacent existing partitions together, which is much cheaper, but it can only reduce partition count, and the result can end up uneven if the source partitions were already uneven in size.

Use coalesce when shrinking the output of a job, writing fewer, bigger files instead of thousands of tiny ones, and you don't need perfectly even partitions. Use repartition when balance actually matters, before a join where skewed partition sizes would bottleneck one task, for instance.

Salting is the manual fix: append a random suffix to the skewed key so it spreads across multiple partitions instead of piling onto one, join against a correspondingly exploded version of the smaller side, then strip the salt back off after. Since Spark 3.0, Adaptive Query Execution can also detect a skewed join at runtime using actual shuffle statistics instead of a static plan, and automatically split the oversized partition into smaller pieces without any salting code at all.

from pyspark.sql.functions import col, concat, lit, rand, floor

salted = df.withColumn(
  "salted_key", concat(col("customer_id"), lit("_"), floor(rand() * 10))
)

My honest preference: try flipping on AQE's skew join handling first, spark.sql.adaptive.skewJoin.enabled, since it's one config flag versus a real code change. Salting is still worth knowing cold for an interview, since not every shop runs with AQE fully enabled, and some skew patterns are ugly enough that AQE's default thresholds don't catch them.

MEMORY_ONLY keeps deserialized JVM objects in memory, fastest to read, most memory-hungry, and data that doesn't fit just gets dropped and recomputed from lineage the next time it's needed. MEMORY_AND_DISK spills whatever doesn't fit to disk instead of dropping it. The _SER variants store data serialized, a compact binary form, instead of as live objects, which uses noticeably less memory at the cost of CPU time spent serializing and deserializing on every read.

There's no universally right answer here. A dataset you'll touch dozens of times and that comfortably fits in memory wants MEMORY_ONLY. A dataset that's borderline on size and only gets touched two or three times often isn't worth caching at all, since the caching overhead can cost more than just recomputing it.

Without broadcasting, a read-only lookup table referenced inside a task gets serialized and shipped to every task that needs it, which for a large number of tasks means shipping the same data over and over. A broadcast variable ships it once per executor, and every task on that executor reuses the same copy from local memory.

The common case is a broadcast join: when one side of a join is small enough, Spark ships the whole small table to every executor and does the join locally with no shuffle at all, instead of shuffling both sides. spark.sql.autoBroadcastJoinThreshold defaults to 10MB, and Spark decides automatically, though a hint can force it if the automatic estimate is wrong.

from pyspark.sql.functions import broadcast

result = big_orders_df.join(broadcast(small_country_lookup_df), on="country_code")

Look for Exchange nodes, since each one is a shuffle, and count how many stack up. A plan with three or four Exchange nodes for what should be a simple filter-and-aggregate query is worth questioning, it often means a join isn't getting a broadcast when it should, or a repartition got inserted somewhere unnecessary. explain(True) shows all four phases, not just the final physical plan, which helps when the physical plan alone doesn't explain why Spark made a particular choice.

Watermarking tells Spark how long to wait for late or out-of-order data before treating a window as final and dropping the state it was keeping for that window. Without a watermark, a windowed aggregation, counting events per 10-minute window, say, has to keep state for every window forever, since a record with an old timestamp could theoretically still show up next year. That state grows without bound and the job eventually runs out of memory.

200 was a reasonable default years ago, but it has nothing to do with the actual size of your data, which is the real problem. Shuffle 5 GB and 200 partitions might be too many, so each one is a few MB and you pay scheduling overhead for hundreds of tiny tasks with no benefit. Shuffle 500 GB and 200 partitions is way too few, so partitions end up huge and either spill to disk or blow out executor memory.

The fix is setting shuffle.partitions based on actual data volume and cluster size, roughly aiming for 100 to 200 MB per partition after the shuffle. With Adaptive Query Execution turned on, Spark can coalesce these partitions automatically at runtime based on real shuffle output size, which removes most of the guesswork, but it's still worth understanding what the knob does for when AQE isn't available or isn't behaving how you expect.

map applies your function to each record one at a time. mapPartitions hands your function an iterator over an entire partition at once, so it only gets called once per partition instead of once per record.

This matters when your function has real setup cost, like opening a database connection, loading a model, or building an expensive client object. Doing that inside map means paying the cost on every row. With mapPartitions you open the connection once per partition and reuse it across all rows in that partition.

def process_partition(rows):
  conn = create_expensive_connection()
  for row in rows:
    yield transform(row, conn)

df.rdd.mapPartitions(process_partition)

The tradeoff is you have to be careful with memory, since if you materialize the iterator into a list instead of streaming through it, a large partition can blow up executor memory.

A regular Python UDF forces Spark to serialize each row out of the JVM, send it to a separate Python worker process, run your Python code row by row, and serialize the result back. That round trip happens for every single row, and none of it benefits from Catalyst's optimizations or Tungsten's columnar execution, because Catalyst sees your UDF as a black box.

Pandas UDFs, also called vectorized UDFs, use Arrow to move whole batches of rows to Python as a Pandas Series or DataFrame at once, so your function processes a batch instead of paying the serialization tax per row. That's usually an order of magnitude faster than a row-at-a-time UDF, though still slower than an equivalent native Spark SQL expression, since you're still crossing the JVM and Python boundary. The rule of thumb: check if a built-in function can do it first, reach for a Pandas UDF second, and only fall back to a plain UDF for logic that genuinely can't be vectorized.

Caching stores computed data in memory or on disk for reuse, but it still keeps the full lineage behind it. If an executor holding cached data dies, Spark recomputes that partition from the original source using lineage. Checkpointing goes further: it writes the data to reliable storage, usually HDFS or S3, and truncates the lineage, so recovery reads the checkpointed data directly instead of replaying a long chain of transformations.

You need checkpointing mainly in two situations. One is iterative algorithms, like graph algorithms or ML training loops, where the lineage graph grows with every iteration until the DAG itself slows down planning or causes stack overflow errors during recomputation. The other is Structured Streaming, where checkpointing isn't optional, it's how Spark tracks offsets and state so it can resume correctly after a driver restart. The cost is that checkpointing triggers an actual job and a real write to disk, so it's not something to sprinkle in casually the way you might use cache().

With AQE's skew join optimization enabled, Spark checks the actual shuffle output statistics after the map side of a join finishes, since AQE re-optimizes the plan between stages using real runtime sizes instead of only the original query's estimates. If one partition on either side is significantly larger than the median partition size, controlled by a skew factor and a minimum size threshold, Spark splits that oversized partition into several smaller sub-partitions.

It then joins each sub-partition against the matching partition on the other side independently and unions the results back together. This works because a sort-merge join only needs matching keys to land together, so Spark can duplicate the smaller side's matching partition across the split pieces without changing correctness. The result is that instead of one task grinding through a 50 GB partition while every other task finishes in seconds, several tasks each process a manageable chunk, and the stage finishes close to the median partition's time rather than being bottlenecked by the largest one.

Spark compares the estimated size of each side of the join against spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB. If one side is estimated to fall under that threshold, the planner rewrites the join as a broadcast hash join: it ships a full copy of the small side to every executor so each one builds an in-memory hash table and joins against its local partitions of the large side with no shuffle at all.

The catch is that estimate comes from table statistics, or from actual computed stats if you've run ANALYZE TABLE, and it can be wrong, especially after several transformations where Spark can't easily reason about the resulting size. Broadcast something that turns out to be much bigger than estimated and you can OOM an executor. You can override the decision explicitly with a broadcast() hint on the DataFrame.

from pyspark.sql.functions import broadcast

result = large_df.join(broadcast(small_df), "id")

Dynamic partition pruning kicks in when you join a large partitioned fact table against a smaller dimension table, the join key is also the fact table's partition column, and there's a filter applied on the dimension side. Without it, Spark scans every partition of the fact table and filters after the join, since at plan time it doesn't know which partitions actually matter.

With dynamic partition pruning, Spark first evaluates the filtered dimension side, works out the actual set of join key values, and uses that set to prune irrelevant partitions of the fact table before scanning it, effectively pushing the filter through the join. That can turn a full table scan into a scan of a handful of partitions, a massive difference on a fact table partitioned by date with years of history. It only applies to partitioned tables, not to plain filtering on a non-partition column, and needs the optimizer setting enabled, which it is by default in current Spark versions.

The rough guidance is to keep executor-cores around 4 to 5, not the maximum available on a node. Go too high and you get contention on shared resources like HDFS I/O and garbage collection pauses that hurt every task on that executor at once, plus a bigger blast radius if that executor dies. Too few cores and you lose parallelism and pay more JVM overhead per unit of work, since every executor carries its own fixed memory overhead regardless of size.

The other axis is executor-memory versus number of executors on a given node. Fat executors, fewer and larger, reduce per-executor overhead and let you cache more data per JVM, but they take longer to garbage collect and a single OOM kills more concurrent tasks. Thin executors, more and smaller, isolate failures better with shorter GC pauses, but spend more of the total memory on per-JVM overhead. You also have to leave room for the OS, YARN's own daemons, and executor memory overhead, roughly 10 percent of executor memory by default, or containers get killed for exceeding physical memory even though Spark thinks it has room.

Speculative execution launches a duplicate copy of a task that's running much slower than its peers in the same stage, on a different executor, and keeps whichever copy finishes first, killing the other. It's meant to handle stragglers caused by a flaky node, disk contention, or some transient issue on one machine, not a real algorithmic problem with the data.

It doesn't help, and can actively hurt, when the slowness is data skew rather than a bad node, since the duplicate task hitting the same skewed data will be just as slow, and now a second executor's worth of resources has been wasted. It's also risky for tasks with non-idempotent side effects, like writing directly to an external system, since both copies could partially complete their write. Most teams turn it on for read-heavy batch jobs on shared, occasionally flaky infrastructure, and leave it off, or scope it carefully, for jobs doing non-idempotent writes.

The first thing to check is whether it's one executor or many, since that tells you skew versus a genuinely undersized cluster. If one or two executors die while the rest finish fine, it's almost always data skew, one partition holding far more data or far more distinct keys than the others, which shows up in the Spark UI as a task with a dramatically larger input size or shuffle read than the median.

If it's widespread across executors, common causes are too few partitions for the data volume, so each partition is oversized; a broadcast join where the "small" side turned out to be big; caching with the wrong storage level, like MEMORY_ONLY instead of MEMORY_AND_DISK, so instead of spilling it just fails; or a pattern that materializes too much data into a single task's heap, like collecting a large list inside a UDF instead of streaming through it. It's also worth checking executor memory overhead settings, since containers can get killed by YARN for exceeding physical memory even when the JVM heap itself never complained, which usually means the overhead allocation is too tight for off-heap usage like Python workers or native libraries.

repartition(numPartitions) does a full shuffle and spreads rows roughly evenly across that many partitions with no particular column in mind, it just aims for even size. repartition(col) does a hash shuffle keyed on that column's values, so every row with the same key lands in the same partition, with the resulting partition count coming from spark.sql.shuffle.partitions unless you also pass an explicit number.

You'd reach for repartition(col) specifically when you're about to run repeated operations that benefit from co-locating a key, like multiple joins or window functions partitioned by that same column, so you pay the shuffle cost once instead of once per operation. The tradeoff is that if the column's values are skewed, hashing on it produces skewed partition sizes too, so it doesn't guarantee evenness the way a plain numeric repartition does. Plain repartition(numPartitions) is what you use when you just need more parallelism or to fix small-file problems before a write, with no requirement about which rows land together.

In client mode, the driver runs on the machine where you launched spark-submit, outside the cluster, and only the executors run on cluster nodes. That's convenient for interactive work, like running from a notebook or an edge node where you want output in your terminal, but it means the driver's lifetime is tied to that machine staying up and connected. Let your laptop sleep or your SSH session drop, and the driver dies and takes the whole job with it.

In cluster mode, the driver runs inside the cluster as just another managed process, so it survives you disconnecting and can be restarted by the cluster manager on failure depending on configuration. That's what you want for any scheduled production job, since it doesn't depend on a specific client machine staying alive. The operational gotcha is that in cluster mode you don't get driver logs streamed to your terminal, you have to pull them from YARN's log aggregation or the Spark history server instead, which throws people off the first time a job "disappears" after spark-submit returns.

A window function needs all the rows sharing a partition key sorted and available together to compute things like row_number, rank, or a running sum. Write a window spec with no PARTITION BY clause at all, and Spark shuffles the entire dataset into a single partition, since as far as it's concerned every row is in the same window. On a large table that's effectively a full collapse to one task, which either takes forever or blows out that one executor's memory.

The fix is almost always adding a genuine PARTITION BY on the window, even if the business logic doesn't obviously call for one, like partitioning by date or customer segment to at least bound how much data lands in any single partition. The related pitfall is an unbounded window frame, ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING, when only a bounded range is actually needed, which forces Spark to materialize the whole partition's data for every row's computation instead of using a cheaper rolling computation.

By default, Spark reads the schema from one file, or a summary file if present, and applies it to every file in the path. If a later file added a column or changed a type, you either silently lose that column or hit a read failure depending on how incompatible the change is. This bites teams that append new columns to a pipeline over months without going back and reprocessing older data.

Setting mergeSchema to true tells Spark to scan the schema of every file in the dataset and union them into one combined schema, filling in nulls for files missing a given column. That solves the immediate problem but isn't free, since it means an extra pass to read footer metadata from every file before the real job even starts, which gets slow with a large number of small files. Most teams instead standardize on a format like Delta or Iceberg that tracks schema centrally in a transaction log, so schema evolution is handled explicitly and versioned rather than inferred by scanning files at read time.

Not really, and I think that's the most oversimplified thing said about Spark in interview prep. When a Databricks team broke the Daytona GraySort world record in 2014, sorting 100TB in 23 minutes on 206 machines versus the prior Hadoop MapReduce record of 72 minutes on 2,100 machines, they did it entirely on disk, without using Spark's in-memory cache at all. The speedup came from a better DAG scheduler, less serialization overhead, and a more efficient shuffle implementation, not memory.

Memory helps a lot for iterative and interactive work. It isn't the whole story, and reducing Spark to "it's Hadoop with a cache" is the kind of answer that invites a follow-up question you probably don't want.

Twice. Lazy evaluation means nothing is computed until an action fires, and each action re-triggers the whole chain from the source independently, unless something in the middle got cached.

df = spark.read.parquet("s3://data/events/")
filtered = df.filter(df.status == "active")

filtered.count()        # reads the source, runs the filter, once
filtered.write.parquet("out/") # reads the source, runs the filter, AGAIN

Add filtered.cache() right after the filter and the second action reuses the cached result instead of re-reading and re-filtering from scratch. I've watched candidates get this exact scenario in a live-coding round and miss that the fix is one line, not a redesign.

Only the lost partitions, traced back through lineage to the last completed shuffle boundary, not the entire job from the start. Shuffle output written by map tasks on healthy executors persists on local disk and survives a downstream failure, so Spark can usually recompute just the partitions that were lost and re-run the tasks that depended on them instead of restarting every stage before the failure.

I don't have a clean number on how often this actually saves a job in production versus the job just failing anyway when the underlying cause is something like an out-of-memory executor that keeps dying on retry. Recomputation only helps if whatever killed the original task isn't going to kill the retry too.

groupByKey shuffles every single record for a key across the network first, then combines them after the shuffle. reduceByKey combines records locally on each partition first, a map-side combine, functionally similar to a MapReduce combiner, and only shuffles the already-reduced, much smaller intermediate results.

# Shuffles every raw record, then sums after the shuffle
rdd.groupByKey().mapValues(sum)

# Sums locally per partition first, shuffles only the partial sums
rdd.reduceByKey(lambda a, b: a + b)

On a key with a million records spread across partitions, that's the difference between shuffling a million values or shuffling one partial sum per partition. Same output, a very different amount of network traffic.

Skew is when one or a handful of partitions hold dramatically more data than the rest, usually because a join or group-by key is unevenly distributed, one customer ID with 40 million rows sitting next to customer IDs with a few hundred. In the Spark UI it shows up as a stage where 199 tasks finish in a few seconds and one task runs for 40 minutes, dragging the whole stage's wall-clock time up to whatever that one task takes.

Accumulators are write-only counters or sums that tasks running on executors can add to, and only the driver can read the final value, useful for things like counting malformed records while a job runs without pulling all the data back to the driver just to count it.

The catch: accumulator updates are only guaranteed to happen exactly once inside actions. Inside a transformation, if that transformation gets re-executed, a retried task, a recomputed partition after a lost executor, speculative execution running a duplicate copy of a slow task, the accumulator can get updated more than once for the same logical record. Interviewers ask this specifically to see whether you know accumulators aren't a reliable counter unless they're tied to an action, not a transformation.

Catalyst's cost-based decisions happen before the query runs, based on statistics that are often stale, wrong, or missing entirely, especially after several transformations where Spark has to guess at output size. AQE, introduced in Spark 3.0 and on by default since Spark 3.2, re-plans parts of the query at runtime using statistics measured after each shuffle: it can dynamically coalesce a shuffle stage's partitions when the real data turns out much smaller than the static spark.sql.shuffle.partitions setting assumed, switch a sort-merge join to a broadcast join once actual size is known, and split an oversized skewed partition automatically.

My honest opinion is that AQE quietly fixed more real production slowness than any single tuning technique candidates spend interview prep time memorizing. It's also, in my experience, the thing interviewers assume you already know is on by default, and then get visibly surprised when a candidate doesn't mention it at all.

Checkpointing tracks exactly which offsets from the source have been processed, and Spark replays from the last checkpoint after a failure instead of guessing. Combined with a sink that's either natively transactional or idempotent, writes keyed so replaying the same batch twice doesn't duplicate data, that gets exactly-once end to end for supported sinks like Delta Lake or a properly idempotent database write.

It stops applying the moment the sink is something Spark can't make idempotent on your behalf, a plain REST API call inside foreachBatch, an email send, anything without a natural upsert key. Same caveat as Kafka's exactly-once story: it's a strong guarantee for a narrower slice of the pipeline than the name suggests, and you still own idempotency for anything outside it.

First I open the stage's task list, sort by duration, and look at what's different about that one task compared to the fast ones, specifically shuffle read size, input size, and GC time. If that task's input or shuffle read is dramatically larger than the median, it's data skew, one key or one partition is carrying more data, and the fix is on the data side, salting, AQE's skew join optimization, or a different partitioning strategy, not a cluster tuning fix.

If the input size looks similar to the other tasks but GC time is high, something else is going on in that JVM, maybe another process fighting for memory on that node, or a bad executor with a hardware issue. I'd check whether the same physical executor or node shows up as the slow one across multiple stages or multiple jobs, because a consistently slow node is an infrastructure problem worth flagging to the platform team, not a data problem. A different task being slow each time, with similar input sizes, points more toward real skew in the data itself, and speculative execution won't fix that since the duplicate would hit the same skewed data.

This error means the container's total memory usage, not just the JVM heap, exceeded what YARN allocated for it. YARN allocates executor-memory plus spark.executor.memoryOverhead as the container's total budget, and the overhead defaults to roughly 10 percent of executor memory with a floor. If actual off-heap usage exceeds that overhead allocation, YARN kills the container even though the JVM heap itself never threw an OutOfMemoryError.

The usual culprits for exceeding that overhead in production but not dev are Python worker processes for PySpark UDFs, which live outside the JVM heap entirely and scale with data volume and UDF complexity, native libraries doing off-heap allocation, like certain compression codecs or numpy and pandas inside pandas UDFs, or a genuinely larger production dataset pushing more shuffle spill files and bigger direct memory buffers for network transfer. The fix is almost never "add more executor-memory," since that heap increase doesn't touch the actual off-heap problem. Instead you explicitly raise spark.executor.memoryOverhead, or memoryOverheadFactor in newer versions, and if Python UDFs are the cause, separately budget spark.executor.pyspark.memory. I'd also check whether the dev dataset was small enough that off-heap usage never crossed the default overhead floor, which is exactly why this passes locally and only shows up under production data volume.

When a closure passed to map, filter, or similar references a variable or method from the enclosing scope, Spark has to serialize that closure to ship it to executors, and it captures whatever scope is needed to resolve the reference, not just the specific field being used. Reference an instance method or field on a class inside that closure, and Spark ends up trying to serialize the entire outer object, this, and if that object holds something non-serializable, like a database connection, a Spark session reference, or a third-party client, the exception fires even though the actual logic only touches a small piece of it.

The cleanest fix is usually pulling the value you actually need into a local variable before the closure, so only that primitive or small serializable value gets captured instead of the whole enclosing object.

// risky: captures the whole outer object through config
rdd.filter(row => row.value > config.threshold)

// safer: only a primitive gets captured
val threshold = config.threshold
rdd.filter(row => row.value > threshold)

If you genuinely need a larger or expensive-to-construct object, like a client or a lookup table, construct it fresh inside mapPartitions, once per partition rather than once per row, so it never needs to be serialized at all, or wrap it in a broadcast variable if it's read-only data safe to ship once and reuse across tasks. Marking the whole outer class Serializable works as a quick fix but often masks the real issue and can silently ship far more data over the wire than intended, since Java serialization walks every field it can reach.

Inside each executor's JVM heap, Spark reserves a chunk, spark.memory.fraction, defaulting to 0.6 of the heap after subtracting a fixed reserved amount, for what's called unified memory. That pool is split conceptually into execution memory and storage memory. Execution memory backs actual computation: shuffle buffers, sorts, and hash tables used during joins and aggregations. Storage memory backs cached or persisted RDDs and DataFrames, and broadcast variables.

The "unified" part is that this isn't a hard, fixed split the way it was in older Spark versions with separate static regions. Execution and storage can borrow from each other's space depending on current pressure, with spark.memory.storageFraction acting as a soft boundary. If a job needs execution memory and storage is holding cached blocks in the borrowable region, Spark evicts cached blocks, least-recently-used first, since execution has priority: a running task failing is worse than losing a cached block that can be recomputed. If execution memory runs out and there's nothing left to evict or spill, that's when a task actually OOMs, and in practice that usually means an under-provisioned executor for the shuffle, sort, or join workload it's doing, or too few partitions forcing individual tasks to need too much execution memory at once.