Backend Developer Interview Questions (2026): APIs, Databases & System Design

A RESTful API treats state as resources addressed by URLs, uses standard HTTP verbs (GET, POST, PUT, PATCH, DELETE) to act on those resources, and keeps each request stateless, meaning the server holds no session context between calls.

HATEOAS, the idea that responses include links to the next valid actions, is the textbook fourth pillar almost nobody actually implements. Interviewers rarely push on it. What they do push on: do you actually understand statelessness in practice (no server-side session tied to one request), and can you explain why PUT should be idempotent while POST usually isn't.

PUT replaces the entire resource with the payload you send. PATCH applies a partial update, changing only the fields included in the request body.

A common trip-up: candidates say PUT is idempotent and PATCH isn't, without explaining why. PUT is idempotent because sending the same full replacement twice leaves the resource in the same state both times. A PATCH that says "increment counter by 1" is not idempotent, since applying it twice changes the result. A PATCH that says "set status to shipped" is idempotent. The verb doesn't guarantee idempotency by itself, the semantics of the operation do.

A clustered index determines the physical order rows are stored on disk, so a table can have only one. A non-clustered index is a separate structure that points back to the row's location, and a table can have several.

Interviewers usually follow up with "so what's the cost of adding a non-clustered index?" Every insert or update now has to update that index too, which is a real write-cost trade-off, not a free performance win. Candidates who only talk about read speed and skip the write cost usually get pushed harder on this.

Redis fits read-heavy paths where the same data gets requested repeatedly and can tolerate a small window of staleness, session data, computed leaderboard rankings, rate-limit counters, or hot catalog rows that get hit far more than average.

The reasoning interviewers want: Redis serves reads from memory in under a millisecond compared to a database round trip, and it takes read pressure off the primary database so it doesn't become the bottleneck for everything else the app needs to do. Candidates who reach for Redis everywhere, including data that changes every request and has no reuse, usually get pushed on why that's actually worth the added complexity.

A race condition happens when a program's correctness depends on the relative timing of operations that can interleave unpredictably.

A classic example: two requests both read a product's inventory count as 1, both decide there's stock available, and both decrement it, overselling the last unit. The fix requires either a database constraint (preventing inventory from going negative, so one write fails) or a lock around the read-check-write sequence so it happens atomically.

Authentication answers "who are you," verifying identity, usually via a password, token, or biometric. Authorization answers "what are you allowed to do," checking permissions after identity is already established.

A quick way interviewers catch confusion here: ask what error code a request should return when authentication fails (401) versus when a correctly authenticated user tries to access something they don't have permission for (403). Mixing those two up is a small tell that the distinction isn't fully internalized.

A unit test isolates a single function or class, mocking out its dependencies, and runs fast with no real database or network call involved. An integration test exercises real collaborators, an actual test database, an actual downstream service or a close stand-in, and runs slower but catches the class of bug unit tests structurally can't see, wrong SQL, serialization mismatches, timeouts against a real connection.

Interviewers usually want to hear that you don't treat these as competing, a healthy backend test suite has far more unit tests than integration tests, but the integration tests are the ones that catch the bugs that actually make it to production.

A stateless service doesn't keep any client-specific data in memory between requests. Everything it needs, user identity, session data, cart contents, is passed in with the request or pulled from a shared store like a database or Redis. Each request is self-contained, so you can drop any incoming request onto any server instance and get the same result.

This matters for scaling because you can add or remove instances behind a load balancer without worrying about sticky sessions or losing in-flight data. If a server crashes mid-request, the client can just retry against a different instance with nothing to recover. The tradeoff is that state has to live somewhere, so you push it into your database or cache instead of your application servers, which is usually the right place for it since those systems are built to handle replication and failover.

A SQL database like Postgres or MySQL enforces a fixed schema, relationships between tables, and strong consistency through transactions. NoSQL is really an umbrella term for several different models: document stores like MongoDB, key-value stores like DynamoDB, wide-column stores like Cassandra, and graph databases like Neo4j. They usually trade some consistency or query flexibility for horizontal scalability and a looser schema.

In practice I reach for SQL by default because most applications need relational integrity, orders belong to users, payments belong to orders, and the tooling for migrations, joins, and reporting is much better. I reach for NoSQL when I have a specific access pattern that doesn't fit rows and joins, like a chat app storing millions of messages keyed by conversation ID, or a catalog with wildly different attributes per category where a rigid schema would mean a lot of null columns.

A transaction groups multiple SQL statements so they either all succeed or all fail together. The classic example is transferring money between two accounts: you debit one row and credit another, and if the process crashes after the debit but before the credit, you've lost money unless both statements were inside a transaction that rolls back cleanly.

Wrapping it in BEGIN, doing both updates, then COMMIT (or ROLLBACK on error) means the database guarantees you never see a half-finished state. Most ORMs expose this as something like db.transaction(async (tx) => { ... }). A common bug I see in review is doing two separate writes without a transaction and assuming it'll probably be fine, which works until a deploy or a network blip happens at exactly the wrong moment in production.

2xx means success. 200 for a normal response with a body, 201 when you've created a resource like a new user or order, 204 when the request succeeded but there's nothing to return, like a DELETE. 3xx is redirection: 301 for a permanent move, 302 or 307 for a temporary one, 304 for not modified when a client's cached copy is still valid.

4xx means the client did something wrong. 400 for a malformed request body, 401 when they're not authenticated at all, 403 when they're authenticated but not allowed to do this specific thing, 404 for a resource that doesn't exist, 429 for rate limiting. 5xx means the server messed up: 500 for an unhandled exception, 502 when a reverse proxy couldn't get a valid response from an upstream service, 503 when the service is temporarily unavailable, often during a deploy or while shedding load on purpose.

Polling means your service repeatedly asks another system whether anything new happened, say every 30 seconds. It's simple to build but wastes resources when nothing has changed, and there's always a delay between when something happens and when you notice it.

A webhook flips that around: the other system calls your endpoint the moment something happens, so you find out immediately instead of waiting for the next poll. The tradeoff is you now need a publicly reachable HTTP endpoint, retries handled if your endpoint is briefly down, verification that the request actually came from who it claims (usually a signature header), and handling for the same event arriving more than once. Stripe and GitHub both sign their webhook payloads for exactly this reason.

If a secret is hardcoded, it ends up in your git history forever, even if you delete it in a later commit, anyone with repo access or a leaked backup can read it. It also means every environment, local, staging, prod, needs its own code branch or file just to use different credentials, which defeats the point of having one codebase.

The standard fix is to read secrets from environment variables at runtime, injected by whatever runs the process: Docker Compose, Kubernetes secrets, a gitignored .env file locally, or a secrets manager like AWS Secrets Manager or Vault in production. The code just reads process.env.DATABASE_URL and doesn't care where the value came from. I've seen real incidents where a hardcoded AWS key sat in a public repo for months before a billing alert caught a crypto-mining bot using it.

Synchronous means the client's HTTP request stays open while your server does the work and only responds once it's done. That's fine for fast operations like fetching a user profile, but a bad idea for anything slow, like generating a PDF report or sending a batch of emails, because the client is stuck waiting and your server has a thread or connection tied up the whole time.

Asynchronous handling means the server accepts the request, hands the work off, usually by pushing a job onto a queue, and immediately responds with something like a 202 Accepted and a job ID. The client then polls a status endpoint or gets notified later, email, websocket, webhook, once it's done. This keeps your API responsive under load and lets you scale the workers doing the heavy lifting independently from the servers handling incoming requests.

A foreign key ties a column in one table to the primary key of another, and the database enforces that you can't insert a row referencing a parent that doesn't exist, and by default won't let you delete a parent row while children still reference it, or it cascades the delete depending on how it's configured.

Skip foreign keys, thinking you'll enforce the relationship in application code instead, and you'll eventually end up with orphaned rows: orders pointing at a user_id that was deleted, or line items pointing at a product that no longer exists. This usually surfaces months later as a mysterious null pointer or a join that silently drops rows, and by then it's a data cleanup project instead of a five-minute schema fix.

The client generates a unique idempotency key and sends it in a header. The server checks whether it has already processed that key. If yes, it returns the cached result instead of creating a second charge. If no, it processes the request and stores the result against that key, usually with a short TTL.

The real test is the follow-up: what happens if two requests with the same key arrive within milliseconds of each other? A naive "check then insert" has a race condition. The fix is a database-level unique constraint on the idempotency key column, so the second concurrent insert fails fast and the handler can look up and return the first result instead of trusting an application-level check alone.

def create_payment(idempotency_key, amount, customer_id):
    existing = db.query(
        "SELECT result FROM payments WHERE idempotency_key = %s",
        (idempotency_key,)
    )
    if existing:
        return existing.result

    try:
        result = charge_provider(amount, customer_id)
        db.execute(
            "INSERT INTO payments (idempotency_key, result) VALUES (%s, %s)",
            (idempotency_key, result)
        )
        return result
    except UniqueViolation:
        # another request won the race, fetch its result
        return db.query(
            "SELECT result FROM payments WHERE idempotency_key = %s",
            (idempotency_key,)
        ).result

Two common approaches: put the version in the URL (/v1/orders, /v2/orders) or in a request header (Accept: application/vnd.company.v2+json). URL versioning is easier to discover, easier to test in a browser, and easier to explain to a partner team. Header versioning is closer to the REST purist's idea of a single stable resource with multiple representations.

Most interviewers accept either answer. What they actually probe is whether you have a deprecation plan, a sunset date, monitoring on old-version traffic, and a communication path to whoever still calls v1, rather than just quietly deleting the old route one day.

N+1 happens when you fetch a list of N records, then loop over them and issue a separate query for each one's related data, one query plus N more instead of one combined query.

The fix is eager loading, joining or batching the related data into the original query (or one follow-up IN query) instead of one query per row. Most ORMs support this but don't do it by default, which is why the bug usually shows up in production long after code review missed it. Interviewers watch for whether you'd catch this in review by spotting a query call inside a loop, or only after a slow-query alert fires.

# N+1: one query per order (bad, 1 + N round trips)
orders = Order.query.all()
for order in orders:
    print(order.customer.name)   # triggers a query every iteration

# fixed: eager load customers in the same round trip
orders = Order.query.options(joinedload(Order.customer)).all()
for order in orders:
    print(order.customer.name)   # no extra query

Atomicity means a transaction either fully commits or fully rolls back. Consistency means the database moves from one valid state to another according to its constraints. Isolation means concurrent transactions don't see each other's uncommitted changes, to whatever degree the isolation level guarantees. Durability means a committed transaction survives a crash.

Isolation is the one most engineers get wrong, because default isolation levels differ by engine (Postgres defaults to read committed, MySQL's InnoDB to repeatable read) and most developers assume stronger guarantees than their ORM actually provides. A good follow-up question is a phantom-read scenario, to see if you know what your default isolation level actually allows through.

Vertical scaling means putting your service on a bigger machine, more CPU, more RAM. It's the simplest option, requires no application changes, but hits a hardware ceiling and leaves you with a single point of failure. Horizontal scaling means running more instances behind a load balancer, which scales further and survives one instance dying, but it requires the application to be mostly stateless, since session data tied to one server's memory won't be visible to the others.

A good answer names the actual precondition: you can't horizontally scale a service that keeps user session state in local memory without first moving that state somewhere shared, like Redis or a database.

Round robin sends requests to servers in rotating order, simple but ignores current load. Least connections routes to whichever server currently has the fewest active requests, which handles uneven request durations better. Consistent hashing routes based on a hash of something like a user ID or session key, so the same user reliably lands on the same backend, useful for sticky sessions or a per-node local cache.

Interviewers usually add a wrinkle: what happens when a server fails a health check? The answer is that it gets pulled out of rotation automatically, and traffic redistributes among the remaining healthy nodes, ideally without the client noticing anything beyond a slightly higher latency during the transition.

TTL-based expiry is simplest: set a time limit, accept some staleness within that window, done. Write-through updates the cache and database together on every write, keeping the cache fresh at the cost of extra write latency. Write-behind writes to the cache first and flushes to the database asynchronously, faster but risking data loss if the cache node dies first. Explicit invalidation deletes or updates the specific key the moment underlying data changes, the most accurate option, but easy to miss one code path and leave a stale entry sitting there indefinitely.

Interviewers check whether you understand that TTL-only strategies quietly serve wrong data for the whole window, and whether you'd keep a TTL as a safety net even on top of explicit invalidation. Most production systems combine both rather than picking one.

A deadlock is when two or more threads each wait on a resource the other holds, and neither can proceed, so both sit frozen. A livelock is when threads keep changing state in response to each other, actively running, but never making progress, like two people repeatedly stepping the same direction trying to get out of each other's way in a hallway.

Interviewers usually ask how you'd prevent deadlocks. The standard answer is a consistent lock acquisition order across the whole codebase (always lock A before B, never the reverse), plus a lock timeout so a stuck thread eventually gives up and retries.

Sessions store state server-side, often in Redis, and are trivially revocable, delete the record and the user is logged out everywhere immediately. JWTs are stateless and self-contained, verified by signature with no round trip to a shared store, which scales cleanly across many app servers. The cost: a JWT can't be revoked before it expires without adding a denylist, which brings back the shared-state problem JWTs were supposed to avoid.

A reasonable middle ground worth mentioning: short-lived JWT access tokens paired with a longer-lived, server-revocable refresh token, limiting the blast radius of a compromised token without a lookup on every request.

The client redirects the user to the authorization server. The user authenticates and approves the requested access. The authorization server redirects back with a one-time authorization code. The client exchanges that code, along with its client secret, for an access token and refresh token, through a server-to-server call that never touches the user's browser.

For public clients that can't safely hold a secret, mobile apps and single-page apps, PKCE adds a client-generated code verifier and challenge, so an attacker who intercepts the authorization code still can't complete the token exchange without the original verifier. Interviewers on mobile or frontend-adjacent teams tend to ask about PKCE specifically.

Route work through a queue when the caller doesn't need the result immediately to keep moving, sending a welcome email, generating a receipt PDF, reindexing a search document, and when you want to absorb a traffic spike without every request blocking on a slow downstream step.

A direct call still wins when the caller genuinely needs the answer synchronously, checking whether a coupon code is valid during checkout, for instance. Interviewers are usually testing whether you default to "everything's async now" without weighing the added complexity: queues introduce ordering questions, retry logic, and a whole new failure mode where a message sits unprocessed and nobody notices.

Inject the clock and the random source as dependencies rather than calling system time or a global RNG directly from inside business logic. In tests, supply a fixed, controllable clock and a seeded random generator, so the same test produces the same result every run.

Candidates who skip this and call datetime.now() directly inside logic usually end up with flaky tests that fail once a day at midnight or once a year on a leap day, and interviewers who've been burned by exactly that bug tend to ask this question specifically to see if you've hit it before.

CAP says that when a network partition happens between nodes in a distributed system, you have to choose between consistency, every node sees the same data, and availability, every request gets a response even if it might be stale. You can't have both during the partition, though once the network heals you can reconcile. CAP only applies during an actual partition, most of the time you get both.

DynamoDB and Cassandra are classic AP systems: if a node can't reach its peers it still accepts writes and reads, and resolves conflicts later with last-write-wins, vector clocks, or application-level merge logic. A traditional single-primary Postgres setup leans CP: if the primary can't confirm a write to enough replicas, it'll refuse the write rather than risk returning inconsistent data. AP makes sense for a shopping cart, where losing an item briefly is annoying but not dangerous. CP makes sense for an account balance, where showing the wrong number is worse than a slow response.

Sharding splits one logical table across multiple physical databases based on a shard key, usually something like user_id or tenant_id, so no single machine has to hold or serve the entire dataset. Each shard handles a subset of rows and traffic, which is how you scale writes past what one database instance can handle.

The hard part isn't picking a shard key, it's everything that assumed a single database afterward. Joins across shards don't work anymore, so you either denormalize or do the join in application code. A query like "all users matching X" now fans out to every shard and merges results in memory. Transactions that used to span two rows on one machine now span two machines and lose easy ACID guarantees. Resharding later, because your key distribution turned out uneven, means moving live data without downtime. I've seen teams shard by user_id and then get stuck when a top-100-users-by-activity report needed data pulled from every shard and aggregated by hand.

Read replicas copy data from a primary database asynchronously, so there's a small delay, usually milliseconds but sometimes seconds under load, between a write landing on the primary and that same row showing up on a replica. Applications route reads to replicas to spread out load, and writes always go to the primary.

The bug pattern I've seen repeatedly: a user submits a form, the write goes to the primary, the app immediately redirects to a page that reads from a replica to show the result, and the replica hasn't caught up yet, so the user sees stale or missing data right after doing the thing that should have created it. Fixes include reading your own writes from the primary for a short window after a write, using a read-your-writes consistency token some databases support, or just polling with a loading state until the data shows up. It's a good reminder that eventually consistent really does mean there's a real window where the data is wrong.

Opening a new database connection is expensive, it involves a TCP handshake, authentication, and on Postgres specifically, forking a whole new backend process. A connection pool keeps a set of already-open connections around and hands them out to requests as needed, so that setup cost gets paid once instead of on every query.

The failure mode under load happens when the pool is smaller than concurrent demand: every connection is checked out to an in-flight request, new requests queue waiting for one to free up, and eventually time out with something like too many clients already or pool exhausted. That's not the database being unhealthy, it's your application, or a slow query holding a connection too long, starving the pool. The fix is usually some combination of increasing pool size, bounded by what the database can actually handle since Postgres has a hard max_connections limit, adding a proxy like PgBouncer that pools connections above your app instances, and fixing whatever slow query or long transaction is holding connections open longer than it should.

Pessimistic locking grabs a lock on a row before you read it, SELECT ... FOR UPDATE, so any other transaction touching that row has to wait until you commit or roll back. It guarantees no interference, but it means real contention, a slow transaction can block a bunch of others, and a deadlock if two transactions lock rows in different orders.

Optimistic locking assumes conflicts are rare. You read a row along with a version number, do your work, and when you write back you check the version hasn't changed, UPDATE ... WHERE id = ? AND version = ?. If zero rows update, someone else beat you to it, and you retry or surface an error. It scales much better under low contention since nothing blocks, but under high contention on the same row you get a lot of wasted retries. I'd use pessimistic locking for something like a seat reservation where correctness under contention really matters, and optimistic locking for something like a user editing their own profile where two people rarely touch the same row at once.

A circuit breaker sits in front of calls to a downstream dependency and tracks failures. Once the failure rate crosses a threshold, it opens and starts failing fast for a cooldown period instead of letting requests through, then periodically lets a small number through to check whether the dependency has recovered before fully closing again.

A plain retry loop actually makes things worse when a downstream service is genuinely down or overloaded: every failed request retries, multiplying traffic hitting an already struggling service, and each retry still burns time, and holds a connection or thread, waiting for a timeout. A circuit breaker stops sending traffic entirely once it's clear the dependency is unhealthy, protecting both the caller with a fast failure instead of a hang, and the downstream service from a pile-on that delays recovery. Libraries like resilience4j for Java or opossum for Node implement this, and it pairs well with a fallback, like cached data or a degraded response instead of a hard error.

Token bucket gives each client a bucket that refills at a fixed rate, say 10 tokens per second, up to some max capacity, and each request consumes a token. It naturally allows short bursts up to the bucket size while still enforcing a long-run average rate, which is why it's the default choice for most public APIs.

Fixed window counters, counting requests per client per minute and resetting at the top of the minute, are simple but have an edge case where a client can send double the limit right at the boundary, once just before reset and once right after. Sliding window approaches fix that by weighting the previous window's count based on how far into the current window you are, giving a smoother rate at the cost of a bit more bookkeeping, usually a sorted set in Redis for the log variant. In practice I've used token bucket via Redis with a Lua script to make the check-and-decrement atomic, since two separate Redis calls introduce a race under concurrent requests from the same client.

When an orchestrator like Kubernetes sends SIGTERM to kill an old instance during a rolling deploy, the default behavior in most frameworks is to just die immediately, dropping any request mid-flight. A graceful shutdown catches that signal, stops accepting new connections, waits for in-flight requests to finish up to a timeout, then exits.

process.on('SIGTERM', () => {
  server.close(() => process.exit(0));
  setTimeout(() => process.exit(1), 10000).unref();
});

You also need the orchestrator side configured correctly. Kubernetes removes a pod from the load balancer's endpoints before sending SIGTERM, but there's a race where a request can still get routed to a pod that's already shutting down, which is why a short preStop sleep hook helps let that routing settle. Skipping this is a common cause of a small spike in 502s or connection resets during every deploy that nobody bothers to investigate.

Backpressure is what happens when consumers can't keep up with producers, messages pile up faster than workers can process them. If nothing pushes back on the producer side, the queue grows unbounded, which either exhausts memory on the broker or, if it's disk-backed, keeps growing until disk fills up, while the age of the oldest unprocessed message keeps climbing.

Handling it means either slowing the producer down when queue depth crosses a threshold, scaling consumers automatically based on queue depth, common with Kubernetes autoscaling a worker pool off a queue-length metric, or shedding load deliberately by dropping or rejecting lower-priority messages rather than trying to process everything. I've seen systems where nobody alerted on queue depth, so a slow downstream API silently backed up a queue for six hours before anyone noticed customers weren't getting notification emails until the next morning.

Read committed, Postgres's default, means each statement in a transaction sees a fresh snapshot of committed data, so running the same SELECT twice in one transaction can return different results if another transaction commits a change in between, a non-repeatable read. Repeatable read fixes that by taking one snapshot at the start of the transaction and using it for every statement in it, so the same query always returns the same rows regardless of what else commits concurrently.

Serializable is the strictest: it behaves as if transactions ran one at a time in some order, even though they're actually concurrent, which prevents phantom reads and write skew, where two transactions each read a set of rows, make decisions based on what they saw, and both commit changes that are individually valid but collectively violate an invariant, like two people booking the last seat because both checked availability before either booked. The cost is that serializable transactions can fail with a serialization error the application has to catch and retry, and both repeatable read and serializable hold onto more resources and increase contention. Most applications run read committed and add explicit row locks or unique constraints where correctness actually matters, rather than pay for serializable everywhere.

A feature flag lets you deploy code with a new path disabled, then turn it on for a small percentage of traffic or a specific set of users without a new deploy, separating deploying code from releasing a feature. If something goes wrong you flip the flag off instantly instead of rolling back a deploy, and you can test in production with real traffic on a small blast radius before going to 100%.

What goes wrong in practice: flags that never get cleaned up pile up until the codebase has dozens of conditional branches nobody's sure are safe to delete, flag evaluation itself becomes a dependency, if the flag service is down does the app fail open or fail closed, and did anyone actually decide that on purpose, and flags gating database schema changes are especially dangerous because both old and new code paths need to work against whatever the schema currently looks like during the rollout window. I've seen an incident where a flag got flipped back off after a bad rollout, but the new code path had already written data in a format the old code path couldn't read, so turning the flag off didn't actually fix anything.

In a monolith, one request produces one contiguous stack trace and one set of log lines you can read top to bottom. Once a request fans out across services, an API gateway to an auth service to an order service to a payment service, each one logs independently, and if something fails three hops in you need a way to tie all those log lines back to the single request that caused them.

The standard approach is to generate a correlation ID at the edge, the first service the request hits, attach it as a header, commonly X-Request-Id or the W3C traceparent header if you're using OpenTelemetry, and every downstream service reads that header, includes it in every log line it writes, and passes it along on any further calls. When something breaks, you query your log aggregator for that one ID and see the full path the request took, including which service it failed in and how long each hop took. Without this, debugging a cross-service failure usually means guessing at timestamps and hoping the clocks are in sync, which they often aren't by more than you'd like.

A liveness probe answers whether this process is stuck and should be killed and restarted. A readiness probe answers whether this instance can currently handle traffic. They sound similar but drive different actions: a failed liveness check gets the container restarted, a failed readiness check just gets the pod pulled out of load balancer rotation without killing it.

Configure them identically, usually a naive check that just returns 200 if the process is running, and you lose the value of having two. A service that's alive but temporarily can't serve traffic, still warming up a cache, or it lost its database connection but will reconnect in a few seconds, should fail readiness and stop getting new traffic without failing liveness, since a restart won't fix a database that's down and a restart loop just adds noise. I've seen a database blip cause a fleet-wide restart storm because readiness and liveness were wired to the same endpoint that checked the database connection, so every pod got killed and restarted at once instead of quietly waiting out the blip.

Cursor-based (keyset) pagination beats offset-limit pagination at this scale. OFFSET 500000 LIMIT 20 forces the database to scan and discard 500,000 rows before returning anything, and that cost grows linearly as users page deeper. Keyset pagination instead references the last seen sort value plus a tiebreaker column and asks for rows after that point, which an index can satisfy directly.

Interviewers push on two follow-ups almost every time. First, what happens when the sort column has duplicate values, answer: you need a tiebreaker, usually the primary key, or pages will skip or repeat rows. Second, what happens when rows get deleted between page loads, answer: keyset pagination handles this gracefully since it never depends on absolute position, unlike offset pagination which can skip a row entirely if something ahead of it was removed mid-scroll.

Start with EXPLAIN ANALYZE to see the actual execution plan, not the guessed one. Look for a sequential scan where you'd expect an index scan.

Common causes, roughly in order of likelihood: a missing index on the filtered or joined column that used to be fast enough on a small table; a function wrapped around the indexed column in the WHERE clause (WHERE LOWER(email) = ...), which blocks a standard btree index unless a matching expression index exists; an implicit type mismatch between the column and the literal you're comparing against; and stale table statistics after a bulk load, fixed with a manual ANALYZE. Interviewers listen for whether you reach for "just add an index" before confirming the actual plan. Indexes aren't free, every one slows down writes and costs disk space.

Almost certainly less than 2,000,000, and the exact number varies between runs. Increment isn't a single hardware instruction in most languages, it's read the current value, add one, write it back. If both threads read the same value before either writes back, one increment gets silently lost.

The fix is making the increment atomic, either with a language-level atomic type, a compare-and-swap loop, or a mutex around the read-modify-write sequence. Interviewers push on why the number isn't simply wrong in a predictable way, it's non-deterministic because it depends on the exact interleaving the scheduler happens to produce that run, which is exactly why race conditions are so miserable to reproduce in a debugger.

# unsafe: read-modify-write is not atomic
counter = 0
def increment():
    global counter
    for _ in range(1_000_000):
        counter += 1   # read, add, write: three separate steps

# two threads running increment() concurrently will
# lose updates whenever their read/write steps interleave

At-least-once means a message might be delivered more than once but is never silently dropped, so the consumer has to handle duplicates. At-most-once means a message might be lost and is never retried, simplest to build, riskiest to run. Exactly-once means each message is processed precisely once, no duplicates, no losses.

Exactly-once is mostly a marketing term. Real systems approximate it by combining at-least-once delivery with an idempotent consumer, deduplicating by a message or event ID. True exactly-once across a network that can partition is provably difficult, and most production queues, including ones with "exactly-once" in their docs, still lean on idempotent consumers to make that guarantee hold up.

Send the same webhook payload, with the same event ID, to the handler twice in the test, and assert the resulting side effect, a ledger credit, an order status change, is recorded exactly once, not twice. Also test the event arriving again after a later, different event has already processed, since providers don't guarantee ordering on redelivery.

The harder version targets the idempotency check itself: fire the same event from two concurrent requests and confirm only one side effect gets recorded. An application-level "look it up, then insert if missing" check has its own race condition, the same one covered in the payment idempotency question earlier on this page. The fix is the same: a database-level unique constraint on the event ID. Application logic alone, promising itself it won't race, doesn't hold up under real concurrency.

The naive approach, write to the database then publish to the broker, has a gap: if the process crashes or the broker call fails after the database commit, downstream consumers never find out the write happened. Flip the order and publish first, and you can publish an event for a write that then fails to commit, so consumers act on something that never actually happened.

The transactional outbox pattern solves this by writing the event into an outbox table in the same database transaction as the actual business data change, so either both happen or neither does, which you get for free from ordinary ACID guarantees on a single database.

CREATE TABLE outbox (
  id BIGSERIAL PRIMARY KEY,
  aggregate_id UUID NOT NULL,
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now(),
  published_at TIMESTAMPTZ
);

A separate process, a polling job, or more commonly a change-data-capture tool like Debezium reading the database's write-ahead log, picks up new rows from the outbox table and publishes them to the broker, marking them sent once the publish succeeds. This gives at-least-once delivery of the event with a guarantee it's never published without the corresponding data actually being committed, at the cost of some latency between commit and publish, and needing to clean up the outbox table so it doesn't grow forever.

Some changes are cheap: adding a nullable column with no default in modern Postgres is a metadata-only change that doesn't rewrite the table, so it's instant even on a huge table. But adding a column with a NOT NULL default, changing a column's type, or splitting one table into two all require rewriting every row, and running that as one blocking ALTER TABLE on a billion-row table would hold a lock for potentially hours, taking writes down the whole time.

The standard pattern is expand-contract, done in stages that each ship independently. Expand: add the new column as nullable, deploy application code that writes to both the old and new column on every write, then backfill the new column for existing rows in small batches, a script updating a few thousand rows at a time with a short pause between batches, so you never hold a long lock or generate a burst of write-ahead log that overwhelms replication. Once the backfill is verified, deploy code that reads from the new column instead of the old one. Finally, contract: drop the old column once nothing depends on it, as its own later deploy so you can back out of any stage cleanly. The mistake I see most often is trying to add the column, backfill, and switch reads all in one deploy, which removes your ability to roll back cleanly if something goes wrong halfway, and on a table that size, something always takes longer than expected.

50 million redirects a day averages out to roughly 580 requests per second, with real traffic spikier than that average. The read path (the redirect) dominates the write path (creating a link) by a wide margin, which changes where you spend engineering effort.

For writes: generate the short code with a dedicated ID-generation service (Snowflake-style, or a counter service handing out ID ranges) and base62-encode it, rather than a single database auto-increment column, which becomes a coordination bottleneck across multiple app servers or regions. For reads: put a cache, Redis or a CDN edge cache, in front of the datastore, since a small set of popular links account for most daily redirects. Only fall back to the primary store on a cache miss. Keep the datastore simple, a key-value store or Postgres with a unique index on the short code, since it's a single indexed lookup, not a complex query. Given the sub-50ms p99 target, cache hit rate is the metric to protect, database throughput mostly stops mattering once the hot set is cached.

An in-memory rate limiter, a counter in each process, doesn't work once you have more than one instance behind a load balancer, because each instance only sees its own slice of traffic and a client could get three times the intended limit just by hitting three different instances. You need shared state, which almost always means Redis or something similarly fast and centralized.

The tricky part isn't the storage, it's making the check-and-increment atomic under concurrency. A GET followed by a conditional INCR as two separate Redis calls has a race: two requests from the same client can both read under limit before either increments, and both proceed. The fix is a Lua script executed atomically on Redis, doing the read, the limit check, and the increment in one round trip Redis guarantees runs without interleaving from another client's script.

local tokens = tonumber(redis.call('HGET', KEYS[1], 'tokens') or capacity)
local last = tonumber(redis.call('HGET', KEYS[1], 'ts') or 0)
local now = tonumber(ARGV[1])
local refilled = math.min(capacity, tokens + (now - last) * rate)
if refilled >= 1 then
  redis.call('HSET', KEYS[1], 'tokens', refilled - 1, 'ts', now)
  return 1
end
redis.call('HSET', KEYS[1], 'tokens', refilled, 'ts', now)
return 0

At high scale Redis itself can become the bottleneck or a single point of failure, which usually means Redis Cluster with rate limit keys sharded by client ID, accepting the limit is enforced per shard rather than globally unless you're willing to pay for cross-shard coordination.

First I'd confirm it's a real leak and not just normal growth. A lot of runtimes, the JVM, Node's V8, let heap usage climb toward a configured max before garbage collecting aggressively, so a sawtooth pattern that resets after GC is healthy, while a line that only ever goes up regardless of GC activity is the actual problem. I'd pull up whatever memory metrics are already being scraped, heap used, RSS, GC pause frequency, to see the shape of the curve before touching anything.

Once confirmed, I want a heap snapshot from a live, leaking instance taken at two points in time far apart, then diff them. In Node that's the inspector plus Chrome DevTools' memory profiler, comparing object counts by constructor between snapshots, the type whose count keeps growing and never drops is usually the leak. In Java it's a heap dump via jmap fed into Eclipse MAT, looking at the dominator tree for what's retaining the most memory and why it's still reachable. Common real causes: an event listener registered on every request but never removed, a cache with no eviction policy or TTL that accumulates keys forever, or a closure that unintentionally captures a large object and keeps it alive through a lingering reference chain nobody realized existed. The fix is almost never adding more memory, that just delays the OOM kill and hides the underlying bug longer.

First I'd check CPU utilization on the box or container. If CPU is pegged near 100%, it's compute-bound, either the request rate genuinely exceeds what this instance can process, scale out, or find and fix an inefficient hot path with a profiler, or something started doing unexpectedly expensive work, like a regex with catastrophic backtracking or serializing a much larger payload than usual. A flame graph from a profiler, pprof for Go, async-profiler for Java, the built-in profiler flag in Node, narrows this to the actual function burning cycles quickly.

If CPU is low but latency is still high, it's more likely IO-bound or blocked, which means checking what the request is actually waiting on: database query time via something like pg_stat_activity for long-running or waiting queries, an outbound call to a dependency that's slow, checked through your own outbound metrics or traces rather than assumptions, or thread pool exhaustion where requests queue waiting for a worker thread tied up elsewhere. Lock contention specifically shows up as low CPU, low apparent IO wait, but high latency, and in Postgres you'd look at pg_locks joined against pg_stat_activity to see what's blocking what, while for an application-level lock you'd want thread dumps taken during the slow period to see multiple threads stuck waiting on the same one. The key discipline is grabbing evidence while the problem is actually happening, because a lot of these symptoms disappear once load drops and you can't reconstruct them from memory afterward.

Active-active means users in the US write to a US region and users in Europe write to a European region, both accepting writes concurrently, instead of every write funneling through one primary region regardless of where the user is, which adds real latency for anyone far from it. The hard problem is what happens when the same piece of data gets modified in two regions before either replication catches up to the other, a genuine conflict to resolve.

Some data doesn't need global ordering and is easy: a user's own profile settings written from wherever they last logged in can use last-write-wins with a timestamp, and the rare case where it's wrong is low stakes. Other data can't tolerate that at all, an account balance or an inventory count can't just pick whichever write came later, since both writes might have been individually valid decrements that together overdraw the balance. For that kind of data you either accept the latency of routing writes to a single authoritative region regardless of user location, giving up the low-latency win for that data type, or use CRDTs, conflict-free replicated data types mathematically designed to merge concurrent updates, which works well for counters and sets but doesn't solve every kind of conflict, like keeping an inventory count from going negative when two regions both accept an order for the last unit. Realistically most systems end up as a mix, most data active-active with simple conflict resolution, a small set of genuinely global invariants routed through one source of truth even at some latency cost.