Distributed Systems Interview Questions (2026), With Answers

A distributed system is a set of independent machines that communicate over a network and present themselves to users as one coherent system. The hard part isn't the individual machines, it's the network between them. A single-server app either works or crashes, and you find out immediately which one happened. A distributed system can fail in states that don't exist on a single machine: a request that's in flight and might have succeeded or might not have, a node that's alive but too slow to answer, a network that delivers messages out of order or twice.

Every design decision in distributed systems traces back to that one fact: you can't tell the difference between "the other machine is dead" and "the other machine is slow, or the network between us is." Everything else, replication, consensus, timeouts, is built to cope with that uncertainty.

A network partition is a break in connectivity between parts of a system that are both still running fine on their own, they just can't reach each other. From the perspective of a node that stops hearing from a peer, a partition looks identical to that peer crashing: no response comes back either way.

This is the core problem distributed systems have to design around. If you can't distinguish "dead" from "unreachable," you can't safely assume a peer has stopped acting. That's exactly why split-brain scenarios happen when two sides of a partition each assume they're the surviving majority and both keep accepting writes.

CAP theorem says that during a network partition, a distributed data store has to choose between consistency (every read gets the latest write) and availability (every request gets a response, even if it might not be the latest data). You can't have both while the partition is happening, because giving a response with stale data isn't fully consistent, and refusing to respond until you're sure isn't fully available.

The part people gloss over: CAP only matters during an actual partition. When the network is healthy, which is most of the time, you can have both consistency and availability. The tradeoff is specifically about what your system does in the failure case, not its everyday behavior.

Strong consistency guarantees that once a write completes, every subsequent read from any replica returns that value or a newer one, never something older. Eventual consistency only guarantees that if writes stop, all replicas will converge to the same value eventually, with no bound on how long "eventually" takes or what a read might return in the meantime.

The tradeoff is latency and availability versus certainty. Strong consistency usually means coordinating with other replicas on every read or write, which costs time and can block during a partition. Eventual consistency lets every replica answer immediately from local data, at the cost of occasionally serving something stale.

A partial failure is when some but not all parts of a system fail while the rest keep running normally, a single node crashing while its peers stay healthy, one availability zone losing power while others don't. Normal software bugs are usually deterministic and reproducible on one machine. Partial failures are a property of the system as a whole, not any single component, and they're often non-deterministic, the same request might succeed on retry with no code change at all.

This is why distributed systems design spends so much effort on detecting and containing failure rather than preventing it outright. You can't write code that stops a network cable from getting unplugged. You can write code that assumes it might be, and degrades gracefully instead of taking the whole system down with it.

Consensus lets a set of independent machines agree on a single value, or a single ordered sequence of values, even when some of them might crash or the network between them might drop messages. Once a value is agreed on, it's treated as final, and correct nodes never disagree about what was decided.

The reason this is hard rather than obvious: you can't just have every node broadcast its opinion and average it, because messages can be delayed, lost, or arrive out of order, and any single node you'd trust to decide could itself be the one that crashes at the worst moment. Consensus algorithms like Raft and Paxos are specifically built to keep working correctly through those conditions.

Replica lag happens because replication isn't instantaneous, the primary applies a write and then has to transmit it over the network to each replica, which then has to apply it locally. Under normal load that gap is milliseconds, but it grows under heavy write volume, a slow network link, or a replica that's under its own resource pressure and falling behind on applying incoming changes.

The practical problem is stale reads: a user updates something, a request immediately after gets routed to a lagging replica, and it looks like the update never happened. This is the exact gap that read-your-writes consistency and sticky routing are built to paper over, and it's why "just add a read replica" isn't a free win, it trades load off the primary for a consistency tradeoff you now have to design around.

Synchronous replication waits for one or more replicas to confirm they've received a write before telling the client the write succeeded. Asynchronous replication tells the client the write succeeded as soon as the primary has it, and sends it to replicas in the background afterward.

The tradeoff is durability versus latency. Synchronous replication guarantees a confirmed write actually exists on more than one machine, so a single node failure can't lose it, but every write pays the network round trip to at least one replica. Asynchronous replication is faster because the client doesn't wait for that round trip, but if the primary fails before the replica catches up, that last write can simply be gone.

Because clocks on different machines drift relative to each other, even with NTP synchronization running, servers can disagree by tens or hundreds of milliseconds, sometimes more under load or on cheap hardware. If two events happen close together on different machines, comparing their local timestamps might tell you the wrong order, or falsely suggest one caused the other when they were actually unrelated.

This matters most when the gap between two events is smaller than the clock drift between the machines that recorded them. A write that happens a millisecond after another, on a machine whose clock runs ten milliseconds fast, can look like it happened first when it didn't.

A fixed timeout has to work for both the fast path and the slow path, and those two cases usually want opposite settings. Set it too short and you'll time out perfectly healthy requests that are just slower than usual under normal load variance, generating unnecessary retries. Set it too long and, when a downstream service actually is failing, every caller waits the full duration before giving up, which ties up threads and connections across your whole fleet and slows down your own failure detection.

Better approaches size the timeout to the actual p99 latency of the call under normal conditions rather than a round number picked out of habit, and pair it with deadline propagation, passing the remaining time budget down through a chain of calls, so a slow downstream service doesn't cause every layer above it to independently wait its own full timeout on top.

Idempotency means an operation produces the same end result no matter how many times it's applied. It matters for retries because a network failure can happen after a request was actually processed successfully but before the response made it back to the caller, so the caller has no way to know whether the original attempt actually worked or not, and retries anyway to be safe.

Without idempotency, that retry can cause real damage, charging a customer's card twice, sending a duplicate email, incrementing a counter an extra time. With idempotency, retrying a request that already succeeded is harmless, the second attempt just returns the same result the first one already produced, so it's always safe to retry when you're unsure.

Sharding splits a dataset across multiple independent database instances, each holding a subset of the data, so no single machine has to store or serve the entire dataset. Which shard a given row lives on is decided by a shard key, some field or combination of fields hashed or ranged to pick a destination shard.

The common mistake is picking a shard key based on what's convenient to query rather than what actually distributes load evenly. A shard key like signup date or a low-cardinality status field concentrates almost all new writes onto whichever shard owns "today," creating a hot shard while the rest of the cluster sits nearly idle.

A query that needs data spread across multiple shards, joining a user's orders with a product catalog sharded by a different key, for example, can't be answered by a single shard alone. The application has to fan the query out to every relevant shard, wait for all of them to respond, and then merge the results itself, which is slower and more complex than a single-database join and gets worse the more shards are involved.

Common workarounds include denormalizing data so the fields you need together live on the same shard, maintaining a separate read-optimized index or search service that isn't sharded the same way as the primary data, or accepting that certain cross-shard queries simply run less often and asynchronously rather than as part of a real-time request path.

At-most-once means a message is delivered zero or one times, never twice, but it might get silently dropped and never delivered at all if something fails. At-least-once means a message is guaranteed to arrive eventually, but it might arrive more than once if the sender doesn't get confirmation and resends. Exactly-once, in the strict sense, means a message is delivered precisely once, no drops and no duplicates.

Most real message queues, Kafka, SQS, RabbitMQ, default to at-least-once, because guaranteeing delivery with zero possibility of a duplicate requires coordination that's genuinely hard to achieve across a network that can fail in the middle of an acknowledgment. At-most-once is easier to implement but loses messages under failure, which is rarely acceptable for anything important.

A dead letter queue is a separate queue where messages get routed after they've failed processing repeatedly, instead of being retried indefinitely or silently dropped. It exists so a genuinely broken or unprocessable message doesn't block the rest of the queue behind it, and so a human or a separate process can inspect and decide what to do with the failures later.

A message should end up there once it's failed a defined maximum number of processing attempts, not on the first failure, since a single transient error shouldn't immediately dead-letter something that would have succeeded on the very next retry. Getting that retry count right matters: too low and you dead-letter messages that just needed one more attempt, too high and a genuinely broken message clogs the main queue far longer than it should.

Because the choice isn't a global setting, it's a per-operation or per-region decision, and most real systems mix both depending on what's being read or written. A system labeled AP, like DynamoDB in its default mode, chooses availability during a partition: it keeps accepting writes on both sides and reconciles the conflict later. A system labeled CP, like a Raft-backed configuration store, refuses to serve writes on the minority side of a partition rather than risk inconsistency.

In practice, teams pick CP for anything where stale or conflicting data causes real damage, financial ledgers, leader election, and AP for anything where a slightly-stale answer beats no answer at all, a shopping cart, a social feed. The label describes the default behavior, not a hard universal rule the system can never deviate from.

Linearizability is the strongest common consistency guarantee: every operation appears to take effect instantaneously at some point between when it was called and when it returned, and all operations across the whole system agree on a single global order. If a write finishes at 10:00:01, every read that starts after 10:00:01 must see that write or something newer, no exceptions, no stale reads from any replica.

Eventual consistency makes no promise about ordering or timing at all, only that replicas converge given enough time with no new writes. Linearizability is what most people actually picture when they say "strongly consistent," and it's expensive to provide across a wide-area network because it requires real coordination on every operation, not just eventual agreement.

Because retries assume the failure was transient and independent of load, and a lot of real failures are neither. If a backend is failing because it's overloaded, retrying the failed request adds more load to an already-struggling system, which causes more failures, which causes more retries. Google's SRE material documents this pattern directly: a backend rejecting 100 QPS due to overload, retried naively, can turn into 200 failed requests generating retries instead of 100 (Google SRE Book).

Retries also don't help with failures that are deterministic given the current state, a malformed request, a resource that genuinely doesn't exist, a permission you don't have. Blind retrying on those wastes a round trip and delays the point where you actually surface the real error to whoever needs to fix it.

Every node in a Raft cluster starts as a follower. If a follower doesn't hear from a leader within a randomized election timeout, it assumes the leader is gone, becomes a candidate, increments its term number, votes for itself, and requests votes from every other node. A node grants its vote to whichever candidate asks first in a given term, as long as that candidate's log is at least as up to date as its own.

Follower timeout -> becomes Candidate -> requests votes (term N)
 -> if majority votes received -> becomes Leader, starts sending heartbeats
 -> if another node wins first -> reverts to Follower
 -> if no majority (split vote) -> waits, retries with term N+1

Raft's own documentation frames this as decomposing consensus into "relatively independent subproblems," with leader election as one piece and log replication as another, specifically so the algorithm stays understandable rather than a monolithic proof (raft.github.io). Whichever candidate collects votes from a majority of the cluster becomes leader for that term and starts sending periodic heartbeats to reset everyone else's election timers.

Split-brain happens when a network partition splits a cluster into two groups, and each group believes it's the legitimate one and elects its own leader, so you end up with two nodes both accepting writes as if they were the sole authority. Left unchecked, this produces two diverging, conflicting versions of your data.

Quorum-based consensus prevents it by requiring a strict majority to elect a leader or commit a write. If a five-node cluster splits into a group of three and a group of two, only the group of three can reach a majority (three out of five), so only that side can elect a leader or accept writes. The minority side simply can't make progress until the partition heals, which is a deliberate, safe tradeoff, not a bug.

Because a majority is the only quorum size that guarantees any two majorities overlap by at least one node. In a five-node cluster, any two groups of three nodes share at least one node in common. That overlap is what prevents two different leaders from both being elected in the same term, or two conflicting log entries from both being committed, since any new majority has to include at least one node that already knows about the previous decision.

Raft's own material describes tolerating minority failures directly: a five-server cluster keeps operating with two failed nodes, since three out of five is still a majority, but progress halts if a majority of servers fail, because there's no longer any group large enough to safely make a decision (raft.github.io).

Log replication is how a consensus algorithm keeps every node's state machine in sync: the leader appends a new command to its own log, then sends that entry to every follower, and once a majority of nodes have stored it, the leader considers it committed and applies it to its state machine. Raft's own description of this ties it directly to correctness: if any server applies a particular command at a given log position, no other server ever applies a different command at that same position (raft.github.io).

Requiring a majority before committing is what makes the entry durable even if the leader crashes immediately afterward. Since any future leader has to be elected by a majority too, and majorities always overlap, the new leader is guaranteed to already have that committed entry in its own log and won't accidentally lose or overwrite it.

A quorum is the minimum number of replicas that have to participate in an operation for it to be considered successful. In a quorum-replicated system, you configure a write quorum W, the number of replicas that must acknowledge a write, and a read quorum R, the number of replicas a read has to contact before returning a result.

The interesting part is how W and R interact with the total replica count N. If a write only needs to reach W replicas and a read only contacts R replicas, those two sets are guaranteed to overlap by at least one node, meaning the read will see the latest write, as long as W plus R is greater than N.

With N total replicas, W required write acknowledgments, and R required read responses, if W + R is greater than N, any write quorum and any read quorum are mathematically guaranteed to share at least one common replica. That shared replica is guaranteed to have seen the most recent write, so the read is guaranteed to return current data, giving you strong consistency without every single replica needing to participate in every operation.

N = 3 replicas
W = 2, R = 2 -> W + R = 4 > 3, overlap guaranteed, strongly consistent
W = 1, R = 1 -> W + R = 2, no overlap guaranteed, reads may return stale data

Systems tune W and R based on whether they want to optimize for read latency, write latency, or consistency. A low W with a high R makes writes fast and cheap but reads slower and more consistent, and vice versa.

Read-your-writes consistency guarantees that after a user writes some data, that same user's subsequent reads will always reflect that write, even if other users might briefly see stale data from a different replica. It's a weaker, more practical guarantee than full linearizability, aimed specifically at the confusing case where you update your own profile and then reload the page to find your old data still showing.

It's hard to guarantee across replicas because the read that follows a write might land on a different replica than the one the write went to, especially behind a load balancer that doesn't know or care which replica served the previous request. Common fixes include sticky sessions that route a user's requests to the same replica, or tracking a version number per user and routing reads to whichever replica has caught up to that version.

A Lamport clock is a simple counter each node maintains and increments on every local event. When a node sends a message, it attaches its current counter value, and when a node receives a message, it sets its own counter to one more than the max of its current value and the value in the message. This produces a partial order where, if event A happened-before event B, A's counter is guaranteed to be smaller than B's.

Node A: event -> counter=1, sends message with counter=1
Node B: receives message (counter=1), sets local counter = max(0,1)+1 = 2

The limitation is that the reverse doesn't hold: a smaller counter doesn't guarantee happened-before, since two independent events on different nodes can end up with counters in either order purely by coincidence. That's the exact gap vector clocks fill, since they can detect genuine concurrency, while a Lamport clock alone can't distinguish "definitely concurrent" from "just happened to get a similar number."

Not in any meaningful sense, and that's actually fine. If two events are truly concurrent, neither one caused the other, there's no "correct" order to recover because no such order exists in reality. What a well-designed system can and should do is detect that the events are concurrent, using something like a vector clock, rather than silently pretending one came before the other based on an arbitrary tiebreaker like server timestamp or which request happened to arrive at a coordinator first.

Some systems do need a total order even for genuinely concurrent events, a database that has to pick a single winner for two simultaneous writes to the same key, for example. In that case the "order" is a deliberate, arbitrary policy decision, last-write-wins by a coordinator-assigned timestamp is common, not a recovery of some true underlying sequence that never existed.

A cascading failure is a failure that grows through positive feedback: one component's problems increase load or stress on the remaining components, which pushes them toward failing too. Google's SRE material describes a concrete version of this: if a cluster handling part of your traffic goes down, its load shifts to remaining clusters, and if that shifted load pushes them past capacity, they start failing too, which then reduces total capacity even further (Google SRE Book).

Retries make this worse specifically because a naive retry adds load right where the system is already struggling. The same source gives a concrete example: a backend rejecting 100 QPS due to overload, retried once per failure, produces 200 QPS of failed requests generating retries, which can spiral further from there. Fixing this requires retry logic that backs off instead of piling on, which is exactly why exponential backoff and retry budgets exist.

Exponential backoff doubles (or otherwise scales up) the wait time between retry attempts, so the first retry might wait 100ms, the next 200ms, the next 400ms, spreading retries out over time instead of hammering a failing service immediately and repeatedly. Jitter adds randomness on top of that schedule so different clients don't all retry at the exact same moment.

attempt 1: wait 100ms + random(0, 50ms)
attempt 2: wait 200ms + random(0, 100ms)
attempt 3: wait 400ms + random(0, 200ms)

Without jitter, every client that got rejected at the same instant retries on the same schedule, so they all pile back onto the struggling backend in synchronized waves instead of a smooth trickle. Google's SRE guidance groups randomized exponential backoff, per-request retry limits, and retry budgets together as the standard fix for retry-driven overload, precisely because backoff alone doesn't stop the thundering-herd effect, jitter does (Google SRE Book).

A circuit breaker is a piece of client-side logic that tracks failure rates for calls to a downstream dependency, and once failures cross a threshold, it stops sending real requests entirely for a cooldown period, failing fast instead. This protects both sides: the caller stops wasting time and threads on calls that are likely to fail anyway, and the struggling downstream service stops receiving load it can't handle while it recovers.

It should trip open once the failure rate crosses a meaningful threshold over a real sample size, not on the very first error, since a single blip shouldn't take a dependency offline for every caller. After the cooldown, most implementations move to a half-open state, letting a small number of test requests through to check whether the dependency has actually recovered before fully reopening the circuit.

A retry storm is what happens when a large number of clients all start retrying failed requests around the same time, usually because a shared dependency degraded, and the combined retry traffic overwhelms a system that might otherwise have recovered on its own. It's the fleet-wide version of the same feedback loop a single overeager client can cause.

A retry budget caps the total volume of retries a client, or a whole service, is allowed to send as a percentage of its normal request volume, rather than letting every failed call generate its own retry unconditionally. Google's SRE material lists per-request retry limits and service-wide retry budgets alongside backoff and jitter as the standard toolkit for keeping retries from becoming the thing that actually causes the outage (Google SRE Book).

Naive sharding computes hash(key) mod N to pick a shard, which works fine until N changes. Add or remove one shard and nearly every key's assignment shifts, since the modulus itself changed, forcing a near-total data reshuffle just to add capacity. Consistent hashing places both shards and keys onto points on a conceptual ring, and each key is owned by the next shard clockwise from its position on that ring.

Ring positions: Shard A=10, Shard B=50, Shard C=90
key hashes to 65 -> owned by next shard clockwise -> Shard C

Add Shard D at position 70:
key hashing to 65 is now owned by Shard D instead
(only keys between 50 and 70 move, everything else stays put)

Only the keys that fell in the range now owned by the new node need to move. The rest of the ring, and every other shard's assignments, stays exactly where it was, which is the entire advantage over hash-mod-N.

A hot partition is a shard receiving disproportionately more traffic than the others, usually because the shard key doesn't distribute load evenly, a celebrity account on a shard where most keys are ordinary users, or a popular product ID that gets hammered while the rest of the catalog sits quiet. The rest of the cluster can be healthy while that one shard's latency and error rate climb from the concentrated load.

Fixing it after the fact usually means splitting that specific hot key range or hot key onto its own dedicated shard, sometimes with a different, finer-grained key scheme just for that range. A more general long-term fix is redesigning the shard key itself, often by adding a random or hashed suffix to keys that are known to cluster, at the cost of making range queries across that suffix more expensive.

A rebalance moves a subset of keys from an existing shard to a new or under-loaded one, which means copying data across the network while the system is still live and serving both reads and writes for those exact keys. During the move, there's a window where a key might be readable from the old shard, the new shard, or briefly neither, depending on how the migration coordinates the cutover.

It's operationally risky because the migration itself consumes real network and disk I/O on shards that are already serving production traffic, which can degrade latency for unrelated requests during the move. Getting the cutover wrong, routing a write to the old location after a key has already moved, or serving a read from a shard mid-copy, is a common source of silent data loss or split-brain-style duplicate writes if the migration tooling isn't built carefully.

An idempotent consumer tracks which message IDs it has already processed, typically in the same database transaction as the actual side effect, and checks that record before processing a new message. If a message arrives that's already been processed, either because the producer retried after a lost acknowledgment or the broker redelivered it after a rebalance, the consumer recognizes the duplicate and skips reprocessing it instead of applying the effect twice.

This shifts exactly-once behavior from a transport-level guarantee, which is genuinely hard to build, to an application-level property, which is much more tractable: store a processed-message-ID table, check it before acting, and record the new ID atomically with whatever the message actually causes to happen.

Two-phase commit coordinates a transaction across multiple independent nodes in two rounds: in the prepare phase, a coordinator asks every participant whether it can commit, and each one locks its resources and votes yes or no. If every participant votes yes, the coordinator sends a commit message in the second phase, and everyone commits; if any participant votes no, everyone rolls back instead.

The biggest weakness is what happens if the coordinator crashes after participants have voted yes but before it sends the final commit or abort decision. Every participant is now stuck holding its locks, unable to commit or roll back on its own, since it doesn't know what the other participants decided or what the coordinator was about to tell them, and it has to wait, potentially indefinitely, for the coordinator to recover.

A saga breaks a multi-step distributed transaction into a sequence of local transactions, each one committed independently and immediately, rather than holding locks across all participants until a single coordinator gives the final word. If a later step fails, the saga runs compensating actions that undo the effects of the steps that already succeeded, working backward through what's already been committed.

This avoids 2PC's blocking problem because no participant ever holds an open lock waiting on a coordinator's final decision, each step is fully committed the moment it succeeds. The tradeoff is that a saga gives up the atomicity 2PC provides, there's a real window where some steps have committed and others haven't yet, and the system has to be built to tolerate being observed in that partially-completed state.

A compensating transaction is a separate operation designed to semantically reverse the effect of an earlier step in a saga, rather than a literal database rollback. Refunding a payment is the compensating transaction for charging it, cancelling a reservation is the compensating transaction for creating one.

Step 1: reserve inventory    -> compensate: release inventory
Step 2: charge payment     -> compensate: issue refund
Step 3: send confirmation email -> compensate: (can't unsend an email)

You can't always undo a step exactly because some actions have real-world side effects that a database transaction can't roll back. Once a confirmation email has gone out or a shipping label has printed, there's no way to make it as if that never happened, so compensating logic for irreversible steps has to send a correction or cancellation notice instead of a true undo, and saga design has to plan for that difference explicitly rather than assuming every step is symmetric.

A Byzantine fault is when a node doesn't just crash or go silent, but actively behaves incorrectly, sending conflicting information to different peers, or malicious or corrupted data that looks superficially valid. It's named after the Byzantine Generals Problem, where some generals coordinating an attack might be traitors actively trying to sabotage the plan rather than simply being unreachable.

Most production systems don't defend against Byzantine faults because doing so is significantly more expensive, both in the number of extra nodes required and in the algorithm's complexity, and most internal infrastructure runs on hardware and networks the operator controls and trusts. Byzantine fault tolerance matters in contexts where participants genuinely can't trust each other, blockchain consensus is the clearest example, but a company's own internal service mesh generally assumes crash failures, a node stops or lags, rather than a node lying on purpose.

A gossip protocol spreads information through a cluster the way rumors spread through a group of people: each node periodically picks a few random peers and shares what it knows, and information propagates exponentially across the whole cluster within a handful of rounds, with no single coordinator involved at any point.

It's a good fit for cluster membership and failure detection, where you want every node to eventually learn who else is alive without a central coordinator becoming a bottleneck or a single point of failure. Cassandra uses gossip for exactly this, letting nodes discover and track each other's status without a leader. It's a worse fit for anything that needs strong consistency or a strict ordering guarantee, since gossip only promises eventual, unordered propagation.

Both solve the same problem, getting a cluster of machines to agree on a sequence of values despite failures, and both require a majority quorum to make progress. Paxos was published first and is provably correct, but the original paper describes the algorithm in a way that's notoriously difficult to translate into a real implementation, especially the multi-decree version needed for a replicated log rather than a single agreed value.

Raft was designed specifically to be understandable, decomposing the problem into separate, easier-to-reason-about pieces: leader election, log replication, and safety, with a strong invariant that all decisions flow through a single leader at a time (raft.github.io). That single-leader design makes the common path much easier to implement correctly than Paxos's more symmetric peer-to-peer negotiation, which is a big part of why systems like etcd, Consul, and CockroachDB all built on Raft rather than raw Paxos.

Both candidates increment their term and request votes simultaneously. Some nodes will vote for whichever candidate's request arrives first, since Raft only allows one vote per node per term, so the votes end up split between the two candidates and neither reaches a majority. This is called a split vote, and it results in an election with no winner.

Raft resolves this without any special-case logic, purely through randomized election timeouts. Each node picks a random timeout within a range, so after a failed election, nodes time out at different moments and are unlikely to start a new round of competing candidacies at exactly the same instant again. The randomization means split votes happen occasionally but essentially never repeat indefinitely, since the odds of another simultaneous tie shrink fast with each retry.

A vector clock is a data structure that tracks, for a given event, how many events each node in the system has seen, as a vector of counters, one per node. Comparing two vector clocks tells you whether one event causally happened before another, or whether they're concurrent, meaning neither one could have caused the other.

Node A writes: [A:1, B:0, C:0]
Node B, unaware of A's write, writes: [A:0, B:1, C:0]
Neither vector dominates the other -> the writes are concurrent, a genuine conflict

A plain wall-clock timestamp can't express this. Two clocks on different machines are never perfectly synchronized, so comparing timestamps from different nodes can tell you which one has a bigger number, but not which one actually happened first in a causal sense, and it definitely can't tell you two writes were genuinely concurrent rather than one just slightly delayed. Systems like the original Dynamo used vector clocks specifically to detect these real conflicts instead of silently picking a winner based on an unreliable clock.

External consistency is Google Spanner's guarantee that the whole system behaves as though every transaction executed one at a time in a single, global order, even though transactions are actually spread across many servers and datacenters worldwide. Google's own documentation states it directly: if one transaction completes before another starts to commit, clients can never observe a state that includes the second transaction's effects without also including the first's (Google Cloud Spanner documentation, TrueTime and external consistency).

The mechanism behind it is TrueTime, a globally synchronized clock API built on GPS and atomic clock hardware in Google's datacenters, which returns not a single timestamp but a bounded uncertainty interval. Spanner assigns commit timestamps to transactions and, critically, waits out the uncertainty interval before acknowledging a commit, so that any transaction which could observe the first one is guaranteed to get a strictly later timestamp. That's a genuinely unusual design: most systems treat clock uncertainty as noise to tolerate, Spanner treats waiting on it as a correctness mechanism.

Clock skew is the difference between what a machine's local clock reads and the actual, true time, and it grows the longer a clock goes without resynchronizing. Most systems rely on NTP, which corrects skew periodically but still leaves an unbounded, unknown margin of error between syncs, so software built on top of it has no reliable way to know how wrong its own clock currently is.

TrueTime takes a different approach entirely: instead of returning a single timestamp, it returns an explicit interval, "the true time is somewhere between this value and that value," derived from GPS and atomic clock reference hardware distributed across Google's datacenters (Google Cloud Spanner documentation). Because Spanner knows the actual size of its own uncertainty rather than pretending it doesn't exist, it can deliberately wait out that interval before committing a transaction, turning an otherwise-unknown error margin into a bounded, known quantity it can design correctness guarantees around.

Because true exactly-once delivery at the network layer is provably impossible in a system where messages can be lost and machines can crash mid-acknowledgment. If a producer sends a message and the acknowledgment gets lost on the way back, the producer has no way to distinguish "the message never arrived" from "it arrived and only the ack was lost," so it has to either resend, risking a duplicate, or not resend, risking a lost message.

What systems that advertise exactly-once semantics actually provide is at-least-once delivery combined with idempotent processing on the consumer side, so duplicates might still technically arrive on the wire, but they're deduplicated before they cause a visible side effect. The end-to-end behavior looks like exactly-once from the application's point of view, but the underlying transport is still doing at-least-once delivery with deduplication layered on top, not a fundamentally different delivery mechanism.

A distributed lock lets multiple processes across different machines coordinate exclusive access to a shared resource, the same job that a mutex does within a single process, but across a network where any participant might crash or lose connectivity mid-operation. A common pattern is using a key in Redis with a TTL as the lock, acquired with a SET NX (set if not exists) command.

The danger is that a single Redis instance is a single point of failure, and worse, the lock's correctness depends on assumptions about timing that a distributed system can't actually guarantee. A process can acquire the lock, then get paused by a long garbage collection cycle or a slow disk write for longer than the lock's TTL, causing Redis to expire the lock and hand it to a second process, while the first process, unaware it lost the lock, resumes and keeps operating as if it still held exclusive access. Now two processes both believe they hold the lock at the same time, which is exactly the failure the lock was supposed to prevent.

A CRDT, conflict-free replicated data type, is a data structure specifically designed so that concurrent updates made independently on different replicas can always be merged automatically into a consistent result, with no coordination and no conflict resolution logic required at merge time. The mathematical property that makes this work is that the merge operation is commutative, associative, and idempotent, so it doesn't matter what order updates arrive in or whether the same update gets applied more than once.

G-Counter (grow-only counter), one slot per replica:
Replica A increments: [A:1, B:0]
Replica B increments: [A:0, B:1]
Merge: take max per slot -> [A:1, B:1], total = 2 (correct, no conflict)

CRDTs solve the specific class of problem where you want multiple replicas to accept writes independently, without waiting on each other, and still guarantee they'll converge to the same correct state once they sync, without a human or an algorithm having to arbitrate a winner. Collaborative text editors, shopping carts that need to work offline, and distributed counters are common real uses, anywhere availability during a partition matters more than perfectly linearizable ordering.

Start by confirming it's actually one node and not a symptom of something upstream, check whether the spikes correlate with a specific traffic pattern, a specific downstream dependency, or a specific time window, rather than assuming the node itself is the root cause. Google's SRE material makes this exact point about overload symptoms: CPU, memory, and thread exhaustion all produce overlapping secondary symptoms, so what looks like the root cause on one node might actually be a knock-on effect of a different resource being exhausted elsewhere in the request path (Google SRE Book).

From there, check the obvious resource exhaustion candidates on that node specifically: CPU saturation causing queueing, garbage collection pauses, disk I/O contention, or a thread pool that's maxed out and queuing requests behind a slow downstream call. Distributed tracing that follows a single request across every hop it touches is usually the fastest way to see exactly where the extra time is actually being spent, rather than guessing from aggregate metrics alone. If the spike only affects one node out of many identical ones, that asymmetry itself is a clue, it points toward something specific to that node's local state, a bad disk, an uneven cache, a hot key it happens to be handling, rather than a systemic issue every node should be sharing equally.

The client generates a unique idempotency key, typically a UUID, once per logical operation, before making the request, and sends it as a header. The server stores that key alongside the eventual result the first time it processes the request. If a request arrives with a key the server has already seen and completed, it returns the stored result immediately instead of processing the payment again.

Client generates key once: idempotency_key = uuid()
POST /charge Idempotency-Key: <uuid> { amount: 4200 }

Server:
 if key exists in store -> return stored response, don't re-charge
 else -> process charge, store {key: result}, return result

The tricky part is the window while a request is still being processed. A naive implementation that only checks after completion is vulnerable to two nearly-simultaneous requests with the same key both passing the check before either finishes. A correct implementation needs to atomically claim the key, insert a "processing" row with a unique constraint on the key column before doing the actual charge, so a concurrent duplicate fails the insert and can wait for or read the original request's result instead of double-charging.