Microservices Interview Questions (2026): 25 Must-Know Q&A

A monolith deploys as one unit: one codebase, one build, one process (or a cluster of identical processes) that owns the whole application. Microservices split the application into independently deployable services, each owning its own data and its own release cycle, that talk to each other over a network instead of through in-process function calls.

Interviewers use this to check whether a candidate conflates "microservices" with "lots of small files" or "using Docker." The network boundary is the whole point, not the size of the codebase. A 200-line service is still a microservice if it's independently deployed and owns its own data; a 40,000-line module inside one deployable isn't, no matter how cleanly it's organized internally.

REST over JSON is the default when you want human-readable payloads, broad tooling support, and easy debugging with curl or a browser. gRPC fits better for internal, high-throughput service-to-service calls where the binary Protobuf encoding and HTTP/2 multiplexing meaningfully cut latency and payload size, and where both ends of the call are services you control.

The trap: reaching for gRPC because it sounds more serious, then losing the ability to debug an issue quickly because nobody can read a Protobuf-encoded payload off the wire without extra tooling. Public-facing APIs still lean REST almost everywhere for that reason.

A BFF is a thin API layer built specifically for one client type, say a mobile app, that aggregates and reshapes calls to multiple backend services into the exact shape that one client needs. A single generic gateway serving mobile, web, and a partner API at once tends to grow bloated response payloads and conditional logic trying to satisfy all three, and a change for one client risks breaking the others.

Interviewers ask this to see if you understand that a BFF isn't a replacement for the gateway; it usually sits behind it, one BFF per client type, each owned by the team that owns that client, so a mobile-only field change doesn't need sign-off from the web team.

A queue (think SQS, or a RabbitMQ queue) delivers each message to exactly one consumer; once it's picked up and acknowledged, it's gone. A pub/sub topic (Kafka, SNS, or a RabbitMQ exchange fanning out to multiple queues) broadcasts each message to every subscriber, so five different services can each react to the same "order placed" event independently.

Use a queue for work distribution, a pool of workers each picking up one job so no two workers process the same job twice. Use a topic when multiple, unrelated services all need to know the same thing happened, and none of them should have to know the others exist.

A load balancer distributes traffic across identical instances of one service. An API gateway sits in front of many different services and adds cross-cutting concerns, authentication, rate limiting, request routing by path, response aggregation, so individual services don't each reimplement the same plumbing.

Interviewers use this to check whether a candidate has actually operated one versus just heard the name. A gateway that also carries heavy business logic is a smell; it's supposed to stay thin.

A liveness probe answers "is this process still working, or is it stuck." If it fails repeatedly, Kubernetes kills the container and restarts it. A readiness probe answers "should traffic be sent to this instance right now." If it fails, Kubernetes just stops routing traffic to that pod; it doesn't restart anything.

Mixing these up causes real incidents. Point the liveness probe at a downstream dependency, say, checking the database connection, and a slow database now gets every instance of your service restarted in a loop, which is exactly backwards; a database being slow isn't a reason to kill and restart the service that depends on it. The dependency check belongs on readiness, so the pod just gets pulled from rotation until the dependency recovers.

livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

Each service owning its own database, with no other service allowed to query it directly, is what actually makes services independently deployable. Without it, a schema change in one service can silently break another service's queries, and you've got a distributed system with a monolith's coupling hiding inside it.

What you give up is the easy JOIN. Any question that spans two services' data now needs an API call, an aggregation layer, or a data pipeline instead of one SQL query, and that trade-off is the real cost of the pattern, not a footnote.

In a monolith, one request ID tags one process's logs, and that's the whole picture. Across microservices, a single user action can fan out into calls to five or ten services, each logging independently, so a trace ID needs to travel along on every downstream call and show up in every one of those services' logs to reconstruct the full picture.

Without that propagated trace ID, debugging turns into grepping five different log systems by timestamp and hoping the timing lines up, which under load, it often doesn't.

OpenTelemetry (OTel) is a vendor-neutral standard for generating traces, metrics, and logs, one SDK and one wire format that every backend, Jaeger, Datadog, Honeycomb, whatever your company picks next year, can ingest, instead of instrumenting your code against one vendor's proprietary agent and having to re-instrument everything if you switch tools.

Before OTel, switching observability vendors meant ripping out instrumentation code across every service and rewriting it against the new vendor's SDK, which is exactly the kind of lock-in cost that made teams stay on a tool they'd outgrown. Now the instrumentation stays put and only the export destination changes, which is a bigger deal for a company's actual flexibility than it sounds on a whiteboard.

Stateless means the process itself doesn't hold onto anything specific to a request once it's done responding. No request-specific data sitting in local memory, no in-process session object tied to one user. Because of that, any instance of the service can pick up any incoming request, and you can kill an instance, spin up a new one, or scale from three pods to thirty without losing anything a user cares about. That's the whole point: it's what makes horizontal scaling and rolling restarts safe.

The catch is that real systems do need to remember things, a shopping cart, a login session, a multi-step checkout flow. The answer is you push that state somewhere external to the process: a Redis cache keyed by user or session ID, a database row, or a signed token like a JWT that the client carries and sends back on every request so the server doesn't have to store anything at all. A cart service, for example, shouldn't keep the cart in a local hash map, it should write it to Redis on every update so it survives a pod restart and any instance can serve the next request for that user.

The workaround teams sometimes reach for instead is sticky sessions, pinning a user to one specific instance at the load balancer. It works, but it quietly undoes the benefit you split services for in the first place. Now that one instance can't be drained or scaled down without dropping active sessions, and your load balancer has to track affinity instead of just spreading load evenly. It's a crutch, not a fix.

It quietly rebuilds the monolith you were trying to get away from, just at the dependency level instead of the deployment level. If every service pulls in the same internal library for, say, order validation logic or a shared "User" model, then a change to that library doesn't just affect one team, it obligates every service that depends on it to eventually rebuild and redeploy. You've traded a single deployable artifact for a dozen services that all have to move in lockstep whenever that shared code changes, which is most of the coordination cost you split the monolith to avoid.

The more concrete failure mode is version skew. Team A bumps the shared library to pick up a bug fix and redeploys. Team B is mid-sprint and doesn't. Now you've got two services in production serializing the same "domain" object slightly differently, or applying business rules that diverged the moment one side updated and the other didn't. These bugs are nasty because nothing crashes, you just get quietly wrong behavior at the boundary between two services that both think they're using the same code.

The practical line most teams draw is: share generic, infrastructure-level code, things like a logging wrapper, a standard HTTP client with retry and tracing headers baked in, or auth token parsing. Don't share domain models or business logic across service boundaries, that's exactly the coupling that bounded contexts exist to prevent. If two services need the same domain concept, that's usually a sign they shouldn't have been split in the first place, or that one of them needs to own that concept and expose it through an API instead of a shared class.

It pays off when different parts of the system need to scale independently, are owned by different teams who'd otherwise block each other on deploys, or carry genuinely different reliability requirements. It adds overhead, and mostly just overhead, when a small team splits a low-traffic app into a dozen services because that's what big companies do.

The tell interviewers look for: can you name a concrete scaling or ownership problem the split solves, or are you just repeating the Netflix justification. The "start with a monolith and split once boundaries are understood from real usage" argument comes up often enough to be worth having a position on.

Draw the line around a business capability that changes for its own reasons, not around a technical layer. Orders, payments, and inventory usually end up as separate services because they change on separate release schedules and are owned by separate teams; splitting "read" and "write" into different services rarely reflects a real boundary on its own.

Weak answers describe boundaries by data model or by technology ("one service per team's favorite framework"). Interviewers want you reasoning from business change frequency and ownership, the two things that actually predict whether a boundary holds up a year later.

A bounded context is a boundary within which a specific domain model and its vocabulary stay consistent. The word "order" inside the fulfillment service can mean something subtly different than "order" inside the billing service, and that's fine as long as each context defines its own meaning clearly.

It matters because forcing one shared Order object across every service is one of the more common causes of tight coupling in a system that's supposed to be loosely coupled. Each service should own its own model of a concept, translating at the boundary through an API contract, instead of sharing a single canonical class across service lines.

The strangler fig pattern routes traffic for one specific capability, say the returns workflow, to a new microservice while everything else keeps hitting the old monolith through a facade or reverse proxy. Over months, more capabilities get carved out and rerouted until the monolith has nothing left to do and gets switched off, rather than every team stopping feature work for a year to rewrite from scratch.

The proxy layer is the part interviewers actually want you to describe, since routing has to be correct at the URL or feature level from day one, otherwise a customer bounces between old and new systems mid-session and sees stale data on one side. I've watched a migration stall for eight months because nobody had clearly owned that routing layer, not because carving out the first service was hard.

Conway's Law says a system's architecture ends up mirroring the communication structure of the organization that builds it. If two teams sit in separate reporting lines and rarely talk, the software boundary between their pieces hardens along that same line, whether or not it's the right technical boundary.

For microservices, this makes the org chart a design constraint, not a footnote. Teams sometimes draw a boundary that looks clean, then find six months in that two "separate" services need to change together on every release because one team owns both halves of a workflow split across the wrong seam. Reorganizing teams to match the boundary you actually want, the inverse Conway maneuver, is a real lever, not just a line people quote in a talk.

Synchronous calls (REST, gRPC) are simpler to reason about and give an immediate response, but they couple the caller's availability to the callee's: if the downstream service is slow or down, the caller is too, unless timeouts and fallbacks are built in. Asynchronous messaging decouples that availability at the cost of giving up an immediate answer and taking on eventual consistency.

Interviewers push on this by asking you to pick a specific call in a specific flow, not the whole system, and defend it. "Everything should be async" is as wrong an answer as "everything should be sync." A checkout confirmation page probably needs a synchronous inventory check; the email that follows doesn't.

An idempotency key is a client-generated identifier attached to a request so that if the network drops the response and the client retries, the server recognizes "I've already processed this exact request" and returns the original result instead of, say, charging a customer twice.

This matters because retries are unavoidable in a distributed system. Networks partition, pods restart mid-request, and load balancers occasionally route a retry to a different instance with no memory of the first attempt. Without an idempotency key, "just retry on failure" quietly becomes "sometimes double-charge on failure," which is a much worse bug to explain to a customer.

Debuggability. A synchronous call chain shows up as one trace with a clear start and end. An event-driven flow scatters that logic across producers and consumers that never call each other directly, so tracing "why did this order never ship" means hunting through five services' logs with no single request to follow.

My honest take: teams that go all-in on events before they've built the tracing and dead-letter-queue tooling to support it usually regret it within a year. Events fit genuinely decoupled workflows, not every call that feels old-fashioned as a plain request.

Consumer-driven contract testing (Pact is the most common tool) lets a consuming service define, in code, exactly which fields and response shapes it depends on from a provider, then verifies the provider against that contract on every build, without either service standing up the other's full stack. It catches a breaking change to a shared field the moment the provider team makes it, in their own CI pipeline, instead of three sprints later when a shared end-to-end environment finally gets deployed together.

End-to-end tests still matter, but they're slow, flaky across a dozen services, and tell you something broke without pointing at which contract. Contract tests are faster and narrower on purpose; they check the shape of an agreement, not the whole business flow.

{
  "consumer": "checkout-service",
  "provider": "inventory-service",
  "interactions": [
    {
      "description": "get stock for sku CHR-2291",
      "request": { "method": "GET", "path": "/inventory/CHR-2291" },
      "response": {
        "status": 200,
        "body": { "sku": "CHR-2291", "available": 42 }
      }
    }
  ]
}

A dead-letter queue (DLQ) catches messages that failed processing after the configured number of retries, so a poison message, one with a bug-triggering payload that will never succeed no matter how many times it's retried, doesn't block every other message stuck behind it in the same queue.

The mistake teams make is treating the DLQ as a place messages go to disappear quietly. Someone needs to alert on DLQ depth and actually look at what landed there; otherwise a bug that's been silently dropping every fifth order confirmation for two weeks doesn't surface until a customer complains, and by then reprocessing means reconstructing state from logs instead of just replaying the queue.

False, or at least incomplete. Kafka only guarantees order within a single partition, not across an entire topic. Two events for the same order landing on different partitions can be processed out of order relative to each other, even though each partition individually stays strictly ordered.

That's why the partition key matters more than most candidates expect going in: keying by order ID sends every event for that one order to the same partition, preserving the order that actually matters, while spreading load across the topic's other partitions for unrelated orders. Pick the wrong key, say a constant value, and you get perfect ordering with zero parallelism, which defeats the reason you picked Kafka in the first place.

A gateway manages north-south traffic, requests coming into the system from outside. A service mesh (Istio, Linkerd) manages east-west traffic, service-to-service calls inside the cluster, adding retries, mutual TLS, and observability at the network layer through sidecar proxies instead of application code.

Plenty of systems run both, and plenty run just a gateway for years before a mesh is worth the added complexity. The honest answer to "do you need a mesh" is usually not yet, not until you've got enough services that manually wiring retries and mTLS into each one has become its own maintenance burden.

Each instance registers its network address with a discovery service (Consul, Eureka, or Kubernetes' built-in DNS-based discovery) on startup, and deregisters, or gets removed via a health check failure, on shutdown. Callers ask the discovery service, or a client-side load balancer caching its state, for a current instance list instead of hardcoding an address.

Kubernetes mostly hides this behind a Service object and cluster DNS now, part of why fewer candidates hand-roll a Eureka client the way they did five years ago. Interviewers still expect you to explain what's underneath, not just say "Kubernetes handles it."

Server-side load balancing puts a dedicated component (a load balancer, or a gateway) between the caller and the pool of instances; the caller just calls one address and the balancer picks an instance. Client-side load balancing has the caller itself pull the current instance list from a discovery service and pick one directly, no extra hop in between.

Client-side (Netflix's old Ribbon, or a service-mesh sidecar acting on the caller's behalf now) shaves off a network hop and lets each caller make its own retry and failover decisions, but it pushes that logic, and the list-freshness problem, into every single caller instead of one shared place. Most teams running Kubernetes get this for free through kube-proxy and cluster DNS without consciously picking a side, which is part of why fewer candidates can explain the trade-off from first principles now.

A circuit breaker wraps a call to a potentially failing dependency and tracks its failure rate. Closed means calls flow through normally. Open means the breaker has seen enough failures that it stops calling the dependency entirely and fails fast instead, protecting both the caller's threads and the already-struggling dependency from more load. Half-open lets a small number of test calls through after a cooldown to check whether the dependency has recovered.

The part candidates forget: failing fast is the actual point, not a side effect. Without a breaker, a caller's threads pile up waiting on a slow dependency until the caller runs out of capacity too, turning one service's outage into two.

resilience4j:
  circuitbreaker:
    instances:
      inventoryService:
        failureRateThreshold: 50
        slidingWindowSize: 20
        waitDurationInOpenState: 15s
        permittedNumberOfCallsInHalfOpenState: 5

Naive retries turn one slow dependency into a self-inflicted denial-of-service attack on it. If every caller retries immediately, three times, the moment a dependency gets slow, the load on the already-struggling thing has just roughly quadrupled, which is exactly backwards.

The fix is exponential backoff with jitter, so retries spread out instead of arriving in synchronized waves, plus a cap on total attempts and, ideally, a circuit breaker that stops retries altogether once the dependency is clearly down rather than just slow.

int attempt = 0;
long baseDelayMs = 200;
while (attempt < 4) {
    try {
        return callDownstream();
    } catch (TransientException e) {
        long backoff = baseDelayMs * (1L << attempt);
        long jitter = ThreadLocalRandom.current().nextLong(backoff / 2);
        Thread.sleep(backoff + jitter);
        attempt++;
    }
}
throw new RetriesExhaustedException();

Failing fast means the caller gets an error quickly instead of waiting on a dependency that's down. Graceful degradation goes a step further: instead of an error, the caller returns something useful with reduced quality, cached recommendations instead of live ones, a product page without reviews instead of no page at all, so the user barely notices anything's wrong.

Not every feature deserves this treatment, and building a fallback path for everything is its own maintenance cost nobody talks about enough. It's worth it for the features on the critical path to checkout or the homepage; it's usually not worth it for an admin-only reporting screen that three internal people check once a day.

A 2-second timeout, when the dependency's actual p99 is 800ms, means a genuinely stuck call ties up the caller's thread for the full 2 seconds before giving up, which under load is plenty of time for a small number of stuck requests to exhaust the caller's entire thread pool and take down a service that isn't even the one having trouble.

The safer number is closer to the dependency's own p99 or p99.9, plus a small margin, not a round number picked to feel comfortable. Pair it with a circuit breaker so repeated timeouts stop hammering a dependency that's clearly down, and revisit the number when the dependency's latency profile changes, since a timeout set correctly for last year's traffic pattern isn't automatically still correct.

Chaos engineering runs controlled experiments, killing an instance, injecting latency, cutting a service off from a dependency, against a live system to check whether the resilience patterns you built (retries, circuit breakers, fallbacks) actually work under real traffic and real data, not just the traffic pattern a staging environment happens to have.

Staging rarely has production's traffic volume, its noisy-neighbor effects, or its actual data shape, so a fallback path that looks fine in staging can still fail in a way nobody predicted once real load hits it. Netflix's Chaos Monkey popularized this by randomly killing instances during business hours; most teams start smaller, one controlled experiment during a low-traffic window with an easy rollback, not a random killer running unsupervised from day one.

A saga is a sequence of local transactions, each in a different service, where every step has a matching compensating action to undo it if a later step fails. Two-phase commit tries to make the whole sequence atomic across services using a coordinator and a prepare-then-commit protocol; sagas give up atomicity and use compensation instead (Chris Richardson's saga pattern reference).

2PC mostly doesn't survive contact with real microservices because it needs a coordinator holding locks while every participant waits, exactly the cross-service coupling you split the system to avoid. Sagas trade strict atomicity for availability, usually the right trade.

CQRS splits the model used to write data from the model used to read it, letting each be optimized separately: a normalized write model for consistency, a denormalized read model tuned for the exact queries the UI actually runs. It's a real answer for read-heavy systems where the write and read shapes have diverged enough that one shared model serves neither well.

It's overkill for a CRUD service where reads and writes look basically the same. Interviewers want you to name the actual symptom, slow reads from complex joins, or a reporting view fighting the transactional schema, not just say "CQRS" because it sounds advanced.

A shopping cart service during a network partition between two data center regions. Rather than reject every write until the partition heals, and lose sales, the service keeps accepting writes to whichever regional replica the customer's request reaches, then reconciles the two carts once the partition clears, occasionally merging or dropping a duplicate line item.

That's choosing availability and accepting weaker consistency, because the business cost of an unavailable cart page during checkout is worse than the cost of an occasional merge conflict a customer barely notices. Inventory counts usually make the opposite choice; overselling a physical item is expensive enough that some inventory services would rather reject a write than risk a duplicate sale (Eric Brewer's own retrospective on CAP, twelve years later is worth reading if this comes up).

You don't query three services' databases directly from a report; that's the exact coupling database-per-service exists to prevent. Instead, each service publishes the events or a change stream that matters, order placed, inventory adjusted, customer updated, into a data warehouse or a dedicated analytics store, where the join happens on data that's already been copied out and denormalized for reporting.

This means the report is eventually consistent, usually by minutes, not real time, and that's almost always fine for a marketing dashboard. It's a genuinely different problem from an operational query a live checkout page needs answered in milliseconds, and conflating the two, trying to make one system serve both, is where a lot of "just add a read replica" plans quietly fall apart.

A log is a discrete, timestamped event with detail, one line saying exactly what happened. A metric is a number aggregated over time, request count, error rate, p99 latency, cheap to store and good for alerting on trends. A trace follows one request's path across every service it touched, with timing for each hop.

Metrics tell you something's wrong. Traces tell you where. Logs tell you why, once you've narrowed it down. Skipping any one of the three means missing a step in that chain, and interviewers ask this to check whether you've actually used all three together to solve a real incident, not just installed a dashboard.

{
  "timestamp": "2026-03-14T09:12:03.114Z",
  "level": "error",
  "service": "payment-service",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "message": "downstream call to inventory-service timed out after 3000ms"
}

Pull the trace for a slow request and look at each span's own duration, not the total. The service whose own span time, not counting time spent waiting on its children, blew up is the one that regressed; everything upstream of it just looks slow because it's waiting.

Without tracing wired end to end, this turns into checking each service's own dashboard one at a time and guessing at the correlation. That gap, microservices adopted before tracing, is where a lot of 3am incidents drag on far longer than they need to.

An SLO, say 99.9 percent of requests succeed over 30 days, implies an error budget, the 0.1 percent of requests allowed to fail without breaking the promise. Once a team has burned through that budget for the month, the honest answer to "should we ship this risky feature today" changes from "sure" to "no, we're out of room, fix reliability first."

Without an explicit budget, reliability work competes against feature work purely on vibes, and features usually win that argument by default. With a budget, running out of it is a concrete, pre-agreed trigger to freeze risky releases, not a debate that has to be re-litigated every time. Google's SRE book is where most teams pull the framing from, and the actual math, 99.9 percent over 30 days is about 43 minutes of allowed downtime, is worth knowing cold, not just the concept.

Version the API and run both versions side by side for a deprecation window, v1 and v2 as separate routes or a version header, rather than changing v1's behavior under callers who haven't migrated yet. Give consumers a real deadline and a migration guide, then remove the old version once usage telemetry shows nobody's calling it anymore, not on a guess.

The failure mode interviewers probe for is a team that changes a field's meaning in place, "we didn't add a new endpoint, we just made the existing one smarter," which breaks every caller who hasn't read the changelog, and there usually isn't a changelog.

Blue-green keeps two full environments running, "blue" (current) and "green" (new), and cuts over all traffic at once after the new version passes its checks; rolling back means flipping the switch back to blue, fast, but a bug that only shows up under real production traffic still hits 100 percent of users the moment you cut over.

Canary sends a small slice of real traffic, often 5 percent, to the new version first, watches error rates and latency against the baseline, and only ramps up if the metrics hold. It catches a bad release with far fewer affected users, since 5 percent of production traffic saw the bug instead of everyone, but it needs good enough metrics and automation to actually watch that 5 percent and decide, rather than a person eyeballing a dashboard for twenty minutes and guessing it's fine.

Splitting "user profile" and "user preferences" into two services is the classic one. It looks clean on a diagram, but almost every feature that touches a user needs both, so you end up with chatty synchronous calls between the two on nearly every request, plus two places to keep consistent when a user is deleted or merged.

The fix usually isn't recombining them; it's asking whether preferences change often enough, and independently enough, to justify the network hop at all. I've seen teams merge these back within six months more often than I've seen them stay split. Not a universal rule, just a pattern that shows up a lot.

An anti-corruption layer is a translation layer at a service boundary that converts another service's, or a legacy system's, model into your own service's vocabulary, so a messy or unstable upstream model never leaks directly into your domain logic. You write an adapter once, at the edge, instead of scattering "if this field means X over there" checks through every place your service touches that data.

It matters most when integrating with a legacy monolith you don't control, or a third-party API whose data model doesn't map cleanly onto your own bounded context. Skipping it feels like it saves time early, since it's one less class to write, but it's usually the reason a supposedly independent service ends up needing a code change every time the upstream system renames a field.

Deadline propagation. The initial timeout needs to travel with the request as a shrinking budget, not just apply once at the entry point, so each downstream service knows how much time is actually left and can bail out early instead of finishing work nobody's waiting for anymore.

Without it, a caller gives up and returns an error to the user while three more services keep burning CPU and connection-pool slots on a request whose result already got thrown away. gRPC has this built in through context deadlines; plenty of REST-based systems just don't bother, which is part of why a slow edge timeout doesn't always match up with what's actually happening downstream.

ctx, cancel := context.WithTimeout(parentCtx, remainingBudget)
defer cancel()
resp, err := inventoryClient.CheckStock(ctx, req)
if ctx.Err() == context.DeadlineExceeded {
    return nil, ErrUpstreamBudgetExhausted
}

At-most-once means a message might get dropped if anything fails; nobody retries, so no duplicates, but data can silently vanish. At-least-once means the broker retries until it gets an acknowledgment, so a message might arrive twice if the ack itself gets lost, but it never silently disappears. Exactly-once is the one everyone wants, and almost no broker actually gives it to you end to end.

Kafka and most managed queues default to at-least-once, and the "exactly-once" support that does exist, Kafka's transactional producer, for instance, usually only guarantees it within that one system, not across the whole chain once your consumer also writes to its own database. In practice, exactly-once processing gets built at the application layer through idempotency keys and deduplication tables, on top of an at-least-once broker, not by trusting the broker alone.

Event sourcing stores every state change as an immutable, ordered event, "OrderPlaced," "OrderShipped," and rebuilds current state by replaying that event log, instead of storing only the current row and overwriting it on every update. A regular service with CQRS or domain events still keeps a normal table as its source of truth and publishes events as a side effect; event sourcing makes the event log itself the source of truth.

The payoff is a full audit trail for free and the ability to answer "what did this order look like last Tuesday" without a separate history table. The cost is real: replaying thousands of events to rebuild current state gets slow without snapshotting, and a bug in an early event's shape can be painful to fix once years of history depend on reading it a certain way. It's a genuinely good fit for finance and audit-heavy domains, and overkill for a simple CRUD service that just needs an event fired when something changed.

The mesh control plane injects a proxy container (Envoy, in Istio's case) into the same pod as the application container, and reconfigures the pod's networking so all inbound and outbound traffic gets routed through that proxy first, transparently, via iptables rules set at pod startup. The application still just calls "http://inventory-service" like nothing changed; the sidecar intercepts that call, wraps it in mutual TLS using a certificate the control plane issued to that specific workload, and hands it off to the destination pod's sidecar, which decrypts it before the application container ever sees the request.

The certs get rotated automatically, often every 24 hours, by the control plane, which is the actual point: nobody's manually managing a cert store per service. The honest trade-off is added latency per hop, usually single-digit milliseconds, and a genuinely harder debugging story the first time a sidecar itself misbehaves and it's not obvious whether the app or the proxy is the problem.

Bulkheading isolates resources, thread pools, connection pools, or entire instances, per dependency, so a slow or failing dependency can't exhaust the resources every other call in the system also needs. The name comes from ship design: a flooded compartment sinks that compartment, not the whole ship.

Without it, a single slow call can eat every thread in a shared pool, and an unrelated feature that never touches that dependency starts timing out too. I don't have a clean rule for how many pools to carve out; too many and you've fragmented capacity into pools that individually run dry even with headroom system-wide.

Choreography has each service publish an event when it finishes its step, and the next service reacts to that event with no central coordinator. Orchestration has a dedicated orchestrator service that explicitly tells each step what to do next and handles compensation logic centrally.

Choreography scales better with fewer services because there's no single owner to become a bottleneck, but it gets genuinely hard to reason about past four or five steps, since the workflow is implicit, scattered across every service's event handlers instead of written down in one place. Orchestration costs you a new component to build and operate, but it makes the business process readable in one file instead of reverse-engineered from event subscriptions.

Classic version: a service updates its own database and then publishes an event about that update as two separate steps. If the process crashes between the commit and the publish, the event never fires, and every downstream service that should have reacted, sending a confirmation email, updating a search index, silently never does, with no error anywhere to point at.

The outbox pattern fixes it by writing the event into an outbox table in the same local transaction as the actual data change, so both succeed or fail together. A separate poller, or a change-data-capture process reading the database log, reads that table and publishes the events after the fact, guaranteeing the event eventually goes out if the data change committed at all.

{
  "id": "6f2e1c1a-6f0c-4e7c-8a2e-6a8d6f42a3f1",
  "aggregate_type": "Order",
  "aggregate_id": "ord_88213",
  "event_type": "OrderConfirmed",
  "payload": { "order_id": "ord_88213", "total_cents": 4599 },
  "created_at": "2026-03-14T09:12:03Z",
  "published_at": null
}

The outbox pattern still requires the service's own code to write a row into an outbox table inside the same transaction as the data change. Log-based CDC, using something like Debezium, skips that step; it reads the database's own write-ahead log, or binlog for MySQL, directly and turns every row change into an event, without the application ever writing to a dedicated table on purpose.

That means CDC works even against a legacy database whose application code you can't touch, which the outbox pattern can't do, since it needs code changes inside the service. The trade-off is that CDC exposes every row change, including ones that were never meant to be a public event, so teams usually still need a transformation step between the raw CDC stream and whatever a downstream service actually consumes, or they end up leaking internal schema details as an accidental public contract.

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "orders-db.internal",
    "database.include.list": "orders",
    "table.include.list": "orders.order_line_items",
    "topic.prefix": "cdc.orders"
  }
}

A distributed lock is only as good as its expiry and its owner check, and both are easy to get subtly wrong. If a service acquires a lock, gets paused by a garbage collection stop-the-world pause longer than the lock's expiry, and resumes after the lock has already been handed to someone else, it keeps working as if it still holds the lock, because nothing told it otherwise.

That's the classic split-brain scenario, two processes both believing they hold the same lock at the same time and both writing, which is worse than no lock at all since it happens silently. Redlock, Redis's own proposed algorithm for this, has been publicly disputed (Martin Kleppmann's detailed critique of its safety guarantees) precisely because distributed locking looks straightforward until you account for clock drift and process pauses. My honest take: reach for a distributed lock only after checking whether the problem can be solved with a database-level unique constraint or a compare-and-swap instead, since either one is usually safer and a lot less code.

Splitting along technical layers, one service for database access, one for business logic, one for the API, instead of along business capability. It produces a dozen services that still deploy together anyway, one logical unit wearing ten network hops as a costume, all the complexity with none of the independent-deployment benefit.

The second most common mistake, close behind: skipping distributed tracing and centralized logging until after the first painful incident, because it felt like work that could wait. It never should have. Both trace back to the same root cause: splitting before doing the design work that's supposed to justify it.

Expand-contract, sometimes called parallel change. First, expand: add the new column alongside the old one, and have the application write to both on every write path, while still reading from the old one. Deploy that, let it run until you're confident writes are landing correctly on both.

Then switch reads over to the new column, deploy, verify. Only after that, contract: stop writing to the old column, and drop it in a later migration once nothing references it anymore. Skipping straight to a single migration that renames the column in place is the version that goes wrong, because the moment the schema changes, every instance still running the previous deploy, and there always are some during a rolling deployment, starts failing every query that touches that column.

-- Step 1: expand
ALTER TABLE customers ADD COLUMN email_address VARCHAR(255);
UPDATE customers SET email_address = email WHERE email_address IS NULL;
-- app writes to both `email` and `email_address` here

-- Step 2: after reads are verified against email_address, switch reads over

-- Step 3: contract, once nothing references the old column
ALTER TABLE customers DROP COLUMN email;