DevOps Engineer Interview Questions (2026): CI/CD, Kubernetes, IaC & Scenarios

Continuous integration means merging code frequently, with an automated build and test run on every push, so integration problems get caught in minutes instead of at the end of a sprint. Continuous delivery means the codebase is always in a deployable state, but a human still decides when to actually ship it, usually to production. Continuous deployment removes that human gate entirely: anything that passes the pipeline goes live automatically.

Interviewers use this question as a filter early in the loop. Confusing delivery and deployment, or describing CI as "when you use GitHub," is the single most common miss here, and it tells them you've used the tools without understanding what they're actually practices for.

A VM virtualizes hardware, so each one runs a full guest operating system on top of a hypervisor. A container shares the host's kernel and gets isolation from Linux namespaces (separate views of processes, network, and mounts) plus resource limits from cgroups, without running a second OS. That's why containers start in milliseconds to a couple seconds while VMs take tens of seconds to minutes, and why a container's memory footprint is a fraction of a comparable VM's.

Mechanically, not much. A Secret is base64-encoded rather than stored as plaintext, and Kubernetes marks it as a Secret specifically so RBAC rules can restrict who's allowed to read it and so etcd encryption at rest can be applied to that object type. Base64 is encoding, not encryption, anyone with API access to read the Secret can decode it with one command, `echo $VALUE | base64 -d`, so the actual protection comes from RBAC and encrypting etcd, not from the encoding itself.

Teams that assume a Secret is safe because it "looks encoded" are the ones who end up with an API token readable by every service account in the namespace, because nobody ever locked down who could `kubectl get secret` in the first place.

Terraform makes infrastructure changes declarative, versioned, and reviewable. Console changes are undocumented, two engineers can make conflicting manual changes with no record of who did what or why, and there's no way to see what changed between last Tuesday and today. Terraform's plan step shows exactly what will change before it happens, its state file tracks what's actually deployed against what the code says should exist, and the code itself goes through the same git review process as application code.

Terraform provisions infrastructure, the VM, the network, the load balancer, the resources that need to exist in the first place. Ansible configures what's already running: installing packages, editing config files, restarting services on a machine that Terraform (or something else) already stood up. A common real setup uses Terraform to create the EC2 instance and Ansible to configure it once it exists, rather than trying to make one tool do both jobs.

Config drift on long-lived VMs is exactly why plenty of teams have moved toward immutable images instead, baking the configuration into the image with Packer and replacing the whole instance on every change rather than running Ansible against a server that's been patched in place for three years and nobody's totally sure what state it's actually in anymore.

`top` or `htop` for a live view sorted by CPU percentage, or `ps aux --sort=-%cpu` for a snapshot you can grep and pipe into something else. If it's a specific process, `strace -p ` shows the actual syscalls it's making, useful when it's stuck in a loop or blocked on I/O rather than genuinely CPU-bound. `uptime` gives the 1, 5, and 15-minute load averages, which tells you whether this just started or has been climbing for a while.

Cron is simpler and older, one line in a crontab, no dependency management, minimal logging beyond whatever the job itself writes out. A systemd timer pairs with a systemd service unit, so failures show up in `journalctl` alongside every other service on that host, it can require another service be running first, and it supports options like running a missed job on boot if the machine was off when it should have fired, which cron doesn't do at all.

Cron's fine for a basic nightly script on a single box nobody's watching too closely. Systemd timers make more sense once a job's failures actually need to be visible to the same monitoring that's already watching everything else running on that host.

EC2 maps to Azure Virtual Machines, same idea, a provisioned compute instance with a chosen size and OS image. IAM maps loosely to Microsoft Entra ID (formerly Azure Active Directory) combined with Azure RBAC, though the mental model differs some, Azure leans harder on Entra ID as one identity plane shared across the whole Microsoft ecosystem rather than IAM's more account-scoped approach. VPC maps to a Virtual Network, with subnets, route tables, and network security groups playing almost identical roles to their AWS counterparts.

Close enough that an engineer fluent in one cloud can get functional in the other within a day or two, even if the console layout and naming take longer to stop tripping you up.

Git keeps history. Deleting the file in a new commit removes it from the current checkout, not from the repo's history, anyone with access to the repo, or a fork, or an old cached clone, can still find that secret in a previous commit with `git log -p` or by checking out an earlier revision.

The only real fix is rotating the credential immediately, treating it as compromised the moment it hit the remote, and separately cleaning history with something like `git filter-repo` if the old commit genuinely needs to be gone, which is a much bigger operation than most people expect it to be.

A runbook is written before an incident happens, a step-by-step procedure for a known failure mode: if the queue depth alert fires, check X, then Y, then restart Z. A postmortem is written after an incident, documenting what actually happened, why, and what changes as a result.

A good postmortem often produces a new runbook entry, or corrects an existing one that turned out to be wrong or missing a step, so the two documents feed each other over time instead of existing independently. Teams without runbooks tend to relearn the same fix from scratch every time the same alert fires, sometimes with a different engineer each time, which is a slower and more stressful way to solve a problem that's already been solved once already.

CMD sets the default command and arguments for a container, but it's easy to override. If you run `docker run myimage echo hello`, that `echo hello` completely replaces whatever CMD had. ENTRYPOINT defines the executable that always runs, and anything you pass on the command line gets appended as arguments to it instead of replacing it.

The pattern most production images use is ENTRYPOINT for the fixed executable and CMD for its default arguments, so you get sensible defaults that are still overridable. For example:

ENTRYPOINT ["/usr/local/bin/myapp"]
CMD ["--config", "/etc/myapp/default.yaml"]

Running the container with no arguments executes `myapp --config /etc/myapp/default.yaml`. Running `docker run myimage --config /etc/myapp/prod.yaml` swaps out just the CMD portion while keeping the ENTRYPOINT fixed. One gotcha worth knowing: both instructions have a shell form and an exec form. Shell form (`CMD myapp --config foo`) runs through `/bin/sh -c`, which means your process isn't PID 1 and won't receive signals like SIGTERM directly, so `docker stop` ends up waiting out the full timeout and then SIGKILLing it. Exec form, the JSON array syntax shown above, runs the binary directly as PID 1 and forwards signals properly, which matters a lot for graceful shutdown in Kubernetes.

Vertical scaling means giving a single instance more resources, more CPU, more RAM, a faster disk. It's simple: no code changes, no load balancer, no distributed state to worry about. The catch is there's a hard ceiling on instance size, it usually requires a reboot or at least a restart of the process to take effect, and you still have a single point of failure. If that one box dies, everything on it goes down.

Horizontal scaling means running more instances of the same service behind a load balancer instead of making one instance bigger. There's no practical ceiling since you can keep adding nodes, you can add or remove capacity without downtime, and losing one instance just means the load balancer routes around it instead of the whole service going dark.

The tradeoff is that horizontal scaling only works cleanly if the service is stateless, or at least externalizes its state. If a service keeps session data or file uploads on local disk, adding a second instance breaks things unless you move that state to a shared store like Redis, S3, or a database, and add sticky sessions or a proper session store. That's exactly why Kubernetes treats stateless workloads as ordinary Deployments you can scale with a replica count, but gives stateful workloads like databases their own StatefulSet primitive with stable identities and persistent volumes per pod. Cloud-native systems default to horizontal scaling because commodity instances are cheaper per unit of capacity than one giant box, autoscaling groups can react to traffic in minutes instead of requiring a maintenance window, and the failure domain per instance is much smaller.

Rolling deployments replace old pods with new ones gradually and are the Kubernetes default, cheap on infrastructure but rollback isn't instant since old and new versions briefly serve traffic together. Blue-green runs two complete environments and switches all traffic at once, giving you an instant rollback at the cost of running double the infrastructure while both are up. Canary releases the new version to a small slice of traffic first, watches metrics, then ramps up gradually.

Pick blue-green when instant rollback matters more than cost, payment processing is the usual example. Pick canary when you have solid metrics and want statistical confidence before a full rollout. Rolling is fine for everyday, low-risk changes where a few minutes of mixed versions won't hurt anyone.

Trunk-based development keeps everyone merging small changes into main, often behind a feature flag, so the codebase never drifts far from what's about to ship. GitFlow keeps long-lived develop, feature, and release branches that only merge at defined points, which sounded safer a decade ago but creates merge conflicts that get worse the longer a branch lives. Teams doing real continuous deployment almost always land on trunk-based, because a pipeline that only runs a handful of times a week against a stale branch isn't actually continuous.

GitFlow still shows up in places shipping infrequently, embedded firmware, enterprise software with a slow release train, where a release branch buys time to stabilize before a deliberate, scheduled release. GitFlow isn't wrong everywhere, it's built for a release cadence most modern SaaS teams don't have anymore.

Multi-stage builds separate the environment you compile or build in from the environment you actually ship. The final image only contains the runtime and the built artifact, not the compiler, SDK, or dev dependencies used to build it. That shrinks image size dramatically (a compiled Go binary doesn't need the Go toolchain sitting in the shipped image) and shrinks attack surface, since tools nobody needs at runtime aren't sitting in a production container waiting to be exploited.

# build stage
FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN go build -o app .

# final stage
FROM gcr.io/distroless/base-debian12
COPY --from=build /src/app /app
ENTRYPOINT ["/app"]

Check the exit code first, `kubectl describe pod` or `docker inspect` will show it. Exit code 137 means OOMKilled almost every time, which usually means the memory limit was set from a guess rather than real usage under production traffic. After that, check for environment parity gaps: a config value, secret, or env var that exists (or silently defaults) locally but is missing in prod and crashes a code path nobody tested. Last, check whether a liveness probe with too short a timeout is killing a container that's just slow to start, which looks identical to a real crash from the outside.

Bridge is the default: each container gets its own IP on a private, NAT'd network, and ports get explicitly mapped to the host with `-p`. Host mode skips network namespace isolation entirely, the container shares the host's network stack directly, so a service listening on port 8080 inside the container is just listening on port 8080 on the host, no mapping needed. None disables networking altogether, used for jobs that only touch local disk and have no reason to talk to anything.

Host mode buys real performance, no NAT translation overhead, which matters for a high-throughput proxy or something latency-sensitive, but it trades away the port-mapping flexibility and isolation that make bridge mode safe to run untrusted or multi-tenant workloads on the same box.

A container's writable layer disappears the moment the container is removed, so a database's data files or uploaded user content sitting there vanishes on the next `docker rm`, which is rarely what anyone actually wants. A bind mount maps a specific path on the host straight into the container, simple for local development but ties the setup to that exact host's filesystem layout. A named volume is managed by Docker itself, lives outside any single container's lifecycle, and is what Docker actually recommends for production, since it behaves the same regardless of the host underneath and plays nicer with backup tooling than a bind mount pointed at some arbitrary host directory.

A Deployment manages interchangeable, stateless pods, any replica can serve any request, and you scale up or down freely without caring which specific pod handles what. A StatefulSet gives pods stable identities and stable storage (pod-0, pod-1, and so on) that survive rescheduling, needed anywhere ordering or identity matters, databases, Kafka brokers, anything with per-instance state. A DaemonSet runs exactly one pod per node, used for node-level agents like log shippers or CNI plugins that need to exist everywhere, regardless of what else gets scheduled.

A Service gets a stable virtual IP and selects pods by label, populating an Endpoints object with the real pod IPs behind it. kube-proxy on every node watches the API server for changes to Services and Endpoints and programs the actual routing, historically through iptables rules (fine for a few hundred rules, noticeably slower past a few thousand services), or through IPVS or an eBPF-based dataplane like Cilium on larger clusters. When a client hits the Service's virtual IP, the node redirects the packet to one of the healthy pod IPs behind it.

A liveness probe answers "is this container alive, or should Kubernetes restart it." A readiness probe answers "is this container ready for traffic right now," and failing it just pulls the pod out of Service endpoints without restarting anything. A startup probe answers "has this container finished its slow initial boot," and delays the other two until it passes, which matters for JVM apps with long warmup times.

Set a liveness timeout too short on a genuinely slow-but-healthy app and you get a restart loop where the app never finishes starting before it gets killed again. From the outside this looks exactly like a real crash, and it's a self-inflicted outage that a decent number of teams have caused to themselves at least once.

A LoadBalancer Service provisions one cloud load balancer per Service, each with its own IP, workable for a single app but expensive and unwieldy the moment a dozen services all want external access. An Ingress is a single set of host- and path-based routing rules sitting behind one shared load balancer, usually a LoadBalancer Service pointing at an Ingress controller like nginx or Traefik, so ten services can share one external IP and one certificate setup instead of ten.

In practice most clusters run exactly one LoadBalancer Service, for the Ingress controller itself, and everything else routes through Ingress rules behind it. Reaching for a LoadBalancer Service per app is usually a sign nobody's set up an Ingress controller yet, not a deliberate choice.

State is a file, usually JSON, mapping the resources in your code to the real cloud resource IDs they correspond to. It's how Terraform knows what already exists versus what needs creating, updating, or destroying on the next apply. Without locking, two engineers running `terraform apply` at the same time can both read the same state, calculate different plans against it, and write conflicting results back, corrupting the file or silently overwriting each other's changes.

Remote backends like S3 with a DynamoDB lock table, or Terraform Cloud, solve this by making state shared and enforcing a lock during apply so only one apply runs at a time.

terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "prod/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

Running the same playbook twice should produce the same end state both times, not append a duplicate line to a config file or restart a service that's already running the desired configuration. Ansible's built-in modules, `apt`, `copy`, `template`, `service`, are idempotent by design, they check current state before acting. A raw `shell` or `command` task isn't automatically safe though, it just runs the command every single time regardless of whether the work's already been done.

That's the actual reason experienced Ansible users avoid `shell:` wherever a proper module exists. An idempotent playbook can run safely on a schedule or after every code change without an engineer worrying about what running it a second, or fifth, time will do to a box that's already correctly configured.

- name: ensure nginx is installed and running
  apt:
    name: nginx
    state: present
- name: ensure nginx service is running
  service:
    name: nginx
    state: started
    enabled: true

In a normal pipeline, CI pushes changes into the cluster, the pipeline itself holds credentials that can reach production and runs `kubectl apply` or `helm upgrade` directly. In GitOps, a controller running inside the cluster, Argo CD or Flux, continuously watches a git repository and pulls changes toward the cluster from there, nothing outside the cluster ever gets write access to it directly.

Git becomes the single source of truth for what's actually deployed, and drift between the repo and the live cluster gets caught automatically, because the controller is always comparing the two and can revert manual changes that don't match what's committed.

That's Argo CD or Flux doing exactly what it's supposed to do. The controller detected that live state no longer matched the git-defined desired state and reconciled it back, which is the entire point of running GitOps, an emergency `kubectl edit` at 2am shouldn't quietly become the new permanent state that nobody remembers making.

The actual problem here is process, not the tool. The fix belongs in a commit and a pull request, even a rushed one, so it survives the next reconciliation loop instead of getting silently undone an hour later. Engineers new to GitOps sometimes read this as the tool "fighting them," when it's really stopping the exact kind of undocumented manual change Ansible and Terraform were both trying to get rid of already.

Metrics answer "is something wrong right now" with aggregated numbers over time, error rate, CPU, request count, cheap to store and good for dashboards and alerting, but they don't explain why. Logs are discrete, timestamped events, good for "what exactly happened on this one request," expensive at scale and hard to correlate across services without consistent structure. Traces follow a single request across every service it touches, and they're the only one of the three that can answer "which of these 12 microservices added the 800ms."

Alert on symptoms that affect users, error rate, latency, and availability against an SLO, instead of every possible internal cause; CPU at 80% might be fine or might not be, depending entirely on context nobody wants to encode into a threshold. Burn-rate alerting tied to an SLO's error budget handles the urgency question well: a fast burn that would exhaust a month's error budget in 2 hours pages immediately, a slow burn that would take 3 days can wait for a ticket. Anything that fires and gets acknowledged without action more than once or twice should get tuned or deleted. A page that trains people to ignore pages is worse than having no alert at all.

Prometheus scrapes metrics by reaching out to each target's `/metrics` endpoint on a schedule, rather than waiting for applications to push data to it. That pull model means Prometheus decides the scrape interval itself and knows immediately when a target is down, a failed scrape is itself a signal, rather than silently missing data because a service quietly stopped pushing an hour ago and nobody noticed.

It's also why service discovery matters so much in a Kubernetes environment specifically. Pods come and go constantly, so Prometheus needs to know which targets exist right now, through Kubernetes service discovery annotations or a file-based mechanism, because a pull-based system can't scrape a target it's never heard of.

scrape_configs:
  - job_name: 'checkout-service'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: checkout
        action: keep

The `for` clause is the difference between an alert that fires on a genuine problem and one that pages someone over a 30-second blip. Without it, a single scrape interval where error rate happens to tick above the threshold, a brief deploy hiccup, a GC pause, fires the alert instantly. With `for: 5m`, the condition has to stay true across every evaluation in that window before Prometheus actually sends it, which filters out the noise without meaningfully delaying a real, sustained problem.

groups:
  - name: checkout-errors
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="checkout"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Checkout error rate above 5% for 5 minutes"

RED, Rate, Errors, Duration, fits request-driven services: how many requests, how many failed, how long they took. USE, Utilization, Saturation, Errors, fits resources: how busy something is, how much work is queued waiting behind it, and how often it's erroring outright. A dashboard built around either framework, one top row showing the four or five numbers that answer "is this actually broken right now," beats a wall of forty graphs nobody can scan in the first sixty seconds of an incident.

I've opened dashboards with forty panels during a live incident and not one of them answered the single question someone actually needed answered in that moment. More graphs isn't more signal, it's usually just more scrolling while something's still on fire.

A hard link is a second directory entry pointing at the same inode and the same underlying data, so deleting one name doesn't remove the data as long as another link still exists, and it can't cross filesystems. A symlink is a separate small file containing a path to another file, works across filesystems and can point at directories, but goes dangling if the target moves or gets deleted.

This bites people with log rotation and zero-downtime deploys. Tools like logrotate and blue-green deploy scripts rely on atomically swapping a symlink, `current -> release-42`, precisely because that swap is a single atomic operation, and moving actual files around is not.

A Layer 4 load balancer routes based on IP and port without looking inside the packet, so it's fast and protocol-agnostic but can't make any decision based on the request itself. A Layer 7 load balancer terminates the connection and reads the actual HTTP request, path, headers, host, so it can route /api/* to one backend and /static/* to another, or serve multiple domains behind one balancer using the hostname.

You need Layer 7 the moment a routing decision depends on anything above the transport layer, path-based routing, header-based canary routing, or TLS termination with SNI for multiple certificates on one IP.

TCP is connection-oriented, it handshakes, guarantees delivery and ordering, and retransmits lost packets, at the cost of overhead. UDP just fires packets with no handshake, no guaranteed arrival, no guaranteed order, and no automatic retry, which is exactly why it's cheap and used for things like DNS queries or real-time video where a dropped packet isn't worth the cost of resending it.

A health check almost always wants TCP, or HTTP on top of it, because a successful three-way handshake actually proves something is listening and accepting connections. A UDP-based check can't tell the difference between "nothing's listening" and "the packet just got dropped somewhere," which makes it a genuinely bad signal for whether a service is actually healthy.

The core of it is a loop with `curl -s -o /dev/null -w "%{http_code}"` to grab just the status code without downloading the body, compared against 200. Interviewers usually care less about the exact syntax here and more about whether you handle timeouts (a hung request shouldn't block the whole check) and whether you'd actually wire the failure into something, a Slack webhook, an exit code a cron job checks, rather than just printing to a terminal nobody's watching.

#!/usr/bin/env bash
urls=("https://api.example.com/health" "https://app.example.com" "https://cdn.example.com")

for url in "${urls[@]}"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$url")
  if [[ "$code" != "200" ]]; then
    echo "ALERT: $url returned $code"
    curl -s -X POST -H 'Content-type: application/json' 
      --data "{"text":"$url returned $code"}" "$SLACK_WEBHOOK_URL"
  fi
done

The core is parsing `df` output per mount point and comparing used percentage against a threshold, 85% is a common cutoff since disks tend to fail in ugly, inconsistent ways once they're actually full, not gracefully. As with the URL check earlier, the syntax matters less than whether the failure actually goes somewhere someone will see it, and whether the script skips filesystem types like `tmpfs` that aren't real persistent storage and would otherwise generate false alarms.

#!/usr/bin/env bash
threshold=85

df -hP --exclude-type=tmpfs --exclude-type=devtmpfs | awk 'NR>1 {print $5, $6}' | while read -r usage mount; do
  pct=${usage%\%}
  if (( pct >= threshold )); then
    echo "ALERT: $mount is at ${pct}% disk usage"
    curl -s -X POST -H 'Content-type: application/json' \
      --data "{\"text\":\"$mount is at ${pct}% disk usage\"}" "$SLACK_WEBHOOK_URL"
  fi
done

An ALB operates at Layer 7, understands HTTP and HTTPS, and supports path-based and host-based routing, which makes it the right choice for microservices sitting behind one balancer. An NLB operates at Layer 4, handles very high request volume at very low latency, and offers a static IP per availability zone that an ALB doesn't. Reach for an NLB for non-HTTP TCP traffic, extreme throughput needs, or when a client needs a fixed IP to put on an allowlist.

An IAM user has a fixed identity with access keys that don't expire until someone rotates or revokes them. A role has no permanent credentials, anything that assumes it, an EC2 instance, a Lambda, another AWS account, gets short-lived, auto-rotating temporary credentials instead. A leaked access key is a standing liability, it works until someone notices, which could be weeks. A leaked assumed-role credential typically expires within an hour, so the exposure window is far smaller.

A security group is stateful and attaches to an instance or ENI, allow-only, if you allow inbound traffic on a port, the matching outbound response is automatically allowed back out without a separate rule. A network ACL is stateless and attaches to a subnet, evaluates numbered rules in order, and supports explicit deny rules, which security groups don't have at all.

Most setups lean almost entirely on security groups for day-to-day rules and use NACLs sparingly, mainly for a blanket block, an IP range that needs to be denied at the subnet level regardless of what any individual security group says.

An environment variable is static, once it's set, every process reads the same value until someone redeploys with a new one, and if it leaks, it stays valid until a human manually rotates it. A secrets manager issues credentials that can be short-lived and rotated automatically, a database password that changes every few hours without anyone touching it, and applications fetch the current value at runtime through an API call or a sidecar rather than baking it in at deploy time.

That means a leaked credential has a much smaller blast radius, it might already be invalid by the time anyone notices, and rotation stops being a manual, once-a-year event that everyone quietly dreads scheduling.

Tools like Trivy, Grype, or Snyk scan an image's layers against CVE databases and flag known-vulnerable packages baked into the base image or dependencies. It belongs right after the build stage and before the image gets pushed to a registry or deployed anywhere, catching it there is nearly free, catching the same vulnerable base image in production after an audit means rebuilding and redeploying everything that used it.

Most teams gate the build to fail on critical and high severity findings only. Failing on every medium and low finding turns the scanner into noise that gets ignored within a month, which defeats the entire point of running it in the first place.

trivy image --severity CRITICAL,HIGH --exit-code 1 registry.example.com/app:${GIT_SHA}

An SLI is the actual measurement, request success rate over the last 5 minutes, a real number pulled straight from monitoring. An SLO is the internal target for that measurement, 99.9% of requests succeed over a rolling 30 days, something the team holds itself to. An SLA is the external, often contractual promise made to customers, usually looser than the internal SLO on purpose, so there's room to miss internally without breaching what customers were actually promised.

The error budget is just 100% minus the SLO, and it's genuinely useful because it turns "should we ship this risky change" from a gut-feel argument into a number. Budget nearly spent, slow down and stabilize. Budget still has room, ship it and take the calculated risk.

Passing tests measures correctness against known inputs, not production reality, and that gap is exactly what's missing here. Staging rarely matches production's data volume, traffic patterns, or config, so the pipeline needs a stage that watches real production metrics right after the deploy, not just before it. Progressive rollout (canary or a percentage-based flag) plus an automatic rollback trigger tied to error-rate or latency burn is what catches this class of failure before every user hits it.

I think most teams under-invest in the 15 minutes right after a deploy ships. That window is the highest-value place to catch a bad release, and it gets treated as an afterthought compared to the pre-deploy checks everyone obsesses over.

The trick is decoupling the migration from the deploy that uses it, and making every step backward compatible on its own. Add the new column as nullable first, ship code that can write to both the old and new column, backfill existing rows in a batch job separate from the deploy, then only after the backfill finishes ship code that reads from the new column, and drop the old column in a later release entirely.

Going straight from "add column" to "rename column" to "drop column" in one migration works fine on a laptop and takes down the site the moment old pods are still running against the new schema mid-rollout. The migration itself should also run as its own step before the deploy, not embedded inside application startup code, so a slow migration doesn't get bundled with, and mistaken for, a slow or failed deploy.

# run the migration as its own job, separate from the deploy
kubectl apply -f migration-job.yaml
kubectl wait --for=condition=complete job/db-migrate-042 -n prod --timeout=300s
# only deploy the new app version once the migration job succeeds
kubectl set image deploy/app app=registry.example.com/app:v042 -n prod

A request is what the scheduler reserves, it decides which node has room for a pod based on requests, not limits. A limit is the hard ceiling the kernel enforces: cgroups throttle CPU past the limit and the kernel OOM-kills the container past the memory limit. Set requests too low and the scheduler over-packs a node, pods that all technically fit on paper end up fighting for real resources the moment traffic actually hits them. Set memory limits too low and containers land in the OOMKilled loop from earlier in this list; set them too high, or skip them entirely, and one runaway pod can starve every other pod on that node before anything steps in.

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

First, stop anyone else from running apply. Check for a backup: S3 versioning if that's your backend, or state history in Terraform Cloud, most teams don't realize versioning would have saved them until this exact moment. If a clean backup exists, restore it, then reconcile any real changes made during the gap with `terraform import`. With no backup, you're rebuilding state resource by resource, comparing against `terraform plan` after each one until it shows no unexpected changes.

It's slow, and it's the reason every team should turn on remote state locking before the first apply, not after the first incident caused by skipping it.

Cost Explorer first, grouped by service then by linked account, to find which line item moved. Usual suspects, in order of frequency: an oversized RDS or EC2 instance left running in a non-prod account, a Lambda stuck in a retry loop multiplying invocations, NAT Gateway data processing charges from a job moving far more data than expected (a classic silent bill-killer nobody budgets for), or an S3 lifecycle policy that never got applied.

Tag resources by team and environment before this happens, not after. Untagged spend across a 40-plus account org is close to impossible to attribute once the bill has landed.

Acknowledge the page first, so it doesn't escalate to the next person on the rotation. Check the dashboard for what changed, and correlate the spike against the deploy timeline first, most production incidents trace back to a recent change, not a random hardware failure. If something deployed in the last hour, roll it back immediately instead of debugging forward, you can chase root cause once the bleeding stops. If nothing deployed recently, check dependencies, a downstream service, a database, a payment processor, before assuming your own code is at fault. Post a status update in the incident channel within the first few minutes even without an answer yet, silence during an incident is worse than "still investigating."

I think the instinct to keep debugging instead of rolling back is the single most expensive mistake engineers make at 3am. Ego wants you to find the bug. The job just wants the error rate back down.

The stages, in order: lint and format check, unit tests, build a single artifact (a container image, usually), push it to a registry, deploy that same artifact to staging, run integration and smoke tests against staging, pass through a gate (manual approval or automated checks against SLOs), deploy to production, then run a post-deploy verification that checks error rate and latency before calling the deploy done.

The detail interviewers actually listen for is "build once, promote the same artifact." A pipeline that rebuilds the image separately for staging and production breaks the guarantee that what you tested is what you shipped, and that gap is where a lot of "it worked in staging" incidents come from.

name: deploy
on:
  push:
    branches: [main]
jobs:
  build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run lint && npm test
      - name: build image
        run: docker build -t registry.example.com/app:${{ github.sha }} .
      - run: docker push registry.example.com/app:${{ github.sha }}

  deploy-staging:
    needs: build-test
    runs-on: ubuntu-latest
    steps:
      - run: kubectl set image deploy/app app=registry.example.com/app:${{ github.sha }} -n staging
      - run: ./scripts/smoke-test.sh staging

  deploy-prod:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: kubectl set image deploy/app app=registry.example.com/app:${{ github.sha }} -n prod

Start from the client side and work outward. `curl -v` or `telnet ` checks whether the port is reachable at the TCP level at all, if this just hangs, it's a security group, NACL, or routing problem, not the application. Check both sides' security group rules, a surprising number of tickets turn out to be an outbound rule that's fine but an inbound rule on the target that never got updated for the new source. Check DNS resolution separately from connectivity with `dig` or `nslookup`, since a DNS failure and a network failure look almost identical but need completely different fixes. If TCP connects fine and the app still fails, it's not a networking problem anymore, move to application logs.

Start with `kubectl describe pod` for events like OOMKilled, image pull failures, or failed probes, then `kubectl logs --previous` to see what the last container printed before it died (the current container may have no logs yet if it just restarted). The exit code tells you a lot: 137 means OOM or SIGKILL, 1 usually means an application error, 143 means an unhandled SIGTERM. Empty logs and an instant death usually mean a missing mounted config or secret. And check probe configuration, an overly aggressive liveness probe killing a slow-but-healthy pod is the most common "it's not actually broken" case I've seen.

kubectl describe pod my-app-7d9f8c-abcde -n prod
kubectl logs my-app-7d9f8c-abcde -n prod --previous
kubectl get events -n prod --sort-by='.lastTimestamp' | tail -20
kubectl get pod my-app-7d9f8c-abcde -n prod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Blameless is the standard line everyone repeats. The harder part is writing a timeline that's actually specific, real timestamps, exact commands run, exact metric values at each point, instead of something vague like "engineers investigated and found the issue." Root cause needs to go past the first mechanical explanation. "The pod OOM'd" isn't a root cause. "The memory limit was copy-pasted from a different service with roughly 4 times less traffic, and nobody revisited it after launch" is closer to one.

Action items need an owner and a date, not "we should add more monitoring" floating with nobody's name on it, that's the item that never gets done. The postmortems that actually work always include at least one change that makes the failure structurally harder to repeat, a CI check, a required review, an automatic rollback trigger, not just a promise to communicate better next time.