Kubernetes Interview Questions (2026): 39 Most Asked, With Answers

The kube-apiserver is the front door, every read and write to cluster state goes through it, and it's the only component that talks to etcd directly. etcd is the cluster's source of truth, a distributed key-value store holding every object's current desired state. The kube-scheduler watches for unscheduled Pods and assigns them to a node. The kube-controller-manager runs the reconciliation loops (the ones that notice a ReplicaSet has 2 Pods when it should have 3, and creates the third) (Kubernetes docs, Components).

Interviewers use this question to check whether "control plane" is a real mental model or just a phrase you've heard. A candidate who lumps etcd, the apiserver, and the scheduler into one vague "the master" is going to struggle two questions later when asked what actually breaks if only one of them goes down.

A Pod is a shared context for one or more containers: they share a network namespace (same IP, can reach each other over localhost) and can share storage volumes. Kubernetes models "one instance of my application" as a Pod because real applications sometimes need a tightly coupled helper running alongside the main process, a log shipper, a proxy, something that only makes sense co-located.

Scheduling, scaling, and networking all happen at the Pod level, never per-container. Two containers in one Pod always land on the same node, always scale together, and always share the same IP address.

A Deployment manages ReplicaSets, and a ReplicaSet manages Pods. The ReplicaSet's whole job is keeping N identical Pods running; it doesn't know anything about rollout strategy, revision history, or rollback.

A Deployment adds all of that on top: rolling updates, a revision history you can roll back to, and pause/resume for a rollout. Creating a bare ReplicaSet directly means giving up rollout and rollback entirely, so almost nobody does it outside of a lab exercise.

A StatefulSet is for workloads that need a stable, unique identity per replica, most commonly a database, a message queue, or anything where "which specific instance am I talking to" matters. Deployment replicas are interchangeable and disposable, Pod names carry a random suffix, and any replica can stand in for any other. StatefulSet replicas get a predictable name instead, web-0, web-1, web-2, and that identity survives a restart.

The practical trigger for picking one is whether replica 2 needs to come back as replica 2 with the same disk it had before, or whether it genuinely doesn't matter which replica handles a given request. Stateless web servers almost never need this. A 3-node Postgres cluster or a Kafka broker almost always does.

A DaemonSet guarantees exactly one Pod per node, or per node matching a selector, and it automatically adds a Pod when a new node joins the cluster and removes it when a node leaves. A Deployment has no concept of "one per node" at all, its replica count is a fixed number you set, unrelated to how many nodes exist.

Log collectors, node-level monitoring agents, and CNI plugins themselves are the classic DaemonSet use cases, anything that has to run on every single node regardless of what else gets scheduled there. Faking that with a Deployment means manually tracking node count and updating replicas every time the cluster scales, which nobody actually does in practice.

A Job runs a Pod, or several, to completion and then stops, unlike a Deployment, which expects its Pods to run forever and restarts them if they exit. Set completions and parallelism to control how many successful runs are needed and how many can run at once, and a Job's restartPolicy has to be Never or OnFailure, never Always.

A CronJob is a Job template instantiated on a schedule, standard cron syntax, and Kubernetes creates a fresh Job object every time the schedule fires. The field worth knowing is concurrencyPolicy: Allow lets overlapping runs pile up if one's still going when the next is due, Forbid skips the new one entirely, and Replace kills the old run and starts the new one. Forgetting to set this to Forbid on a job that isn't safe to run twice at once is a common source of very confusing double-writes.

ClusterIP is the default: a stable virtual IP reachable only from inside the cluster. NodePort opens the same port on every node in the cluster and forwards to the Service, giving external access without a cloud load balancer, useful for bare-metal or local testing. LoadBalancer provisions an actual cloud load balancer (an ELB, say) that routes to the Service, the standard way to expose something to the public internet on a managed cloud. ExternalName is different from the other three entirely, it's just a DNS alias to an external hostname, no proxying or load balancing involved at all.

Structurally, almost nothing, both store key-value data and both can be mounted as environment variables or a volume. A Secret's values are base64-encoded rather than plain text.

Base64 is encoding, not encryption. Anyone with API access to read the Secret object can decode it in one line. Without encryption at rest enabled on etcd, a Secret sitting in etcd is genuinely no more protected than a ConfigMap, it's just less readable at a glance. Real protection comes from RBAC restricting who can read Secrets, encryption at rest on etcd, and for anything genuinely sensitive, an external secrets manager like Vault instead of relying on the built-in object alone.

A plain Volume (like emptyDir) is tied to a Pod's lifecycle, gone when the Pod is. A PersistentVolume (PV) is a cluster-level resource representing actual durable storage, an EBS volume, an NFS share, whatever the underlying infrastructure provides, independent of any specific Pod. A PersistentVolumeClaim (PVC) is a request: "I need 10Gi of storage with these access modes," which Kubernetes binds to a matching PV.

Pods never reference a PV directly. They reference a PVC, and the PVC is what's bound to a PV behind the scenes. That indirection is what lets storage survive a Pod being deleted and recreated, as long as the PVC itself isn't deleted too.

Not by default, and this catches people out. A namespace isolates names (two Deployments called api can coexist in different namespaces without conflicting) and gives you a scope for RBAC rules and ResourceQuotas. It does not isolate network traffic on its own, a Pod in namespace A can reach a Pod in namespace B over the network unless a NetworkPolicy explicitly blocks it.

Real isolation between namespaces takes RBAC (who can do what, scoped per namespace) plus NetworkPolicies (what traffic is allowed between namespaces) working together. Neither alone gets you there.

HPA adjusts the replica count on a Deployment, ReplicaSet, or StatefulSet based on observed metrics, CPU and memory by default, or custom and external metrics through the metrics API. It needs the metrics-server (or a custom metrics adapter for anything beyond CPU and memory) actually running in the cluster, without it, HPA has nothing to read and won't scale at all (Kubernetes docs, Horizontal Pod Autoscaling).

A Role grants permissions scoped to a single namespace. A ClusterRole grants permissions across the whole cluster, or for cluster-scoped resources like Nodes that don't belong to any namespace at all. A RoleBinding attaches a Role (or even a ClusterRole) to a subject within one specific namespace. A ClusterRoleBinding attaches a ClusterRole to a subject across every namespace in the cluster.

The trap: a ClusterRole bound with a RoleBinding only grants those permissions in the RoleBinding's namespace, not cluster-wide, which is a legitimately useful pattern (reuse one ClusterRole definition, bind it narrowly per namespace) but confuses people who assume ClusterRole always means cluster-wide access.

A liveness probe answers "is this container alive, or should Kubernetes restart it." A readiness probe answers "is this container ready for traffic right now," and failing it just pulls the Pod out of the Service's Endpoints without restarting anything. A startup probe answers "has this container finished its slow initial boot," and it delays the other two probes until it passes, which matters for JVM apps or anything with a genuinely long warmup.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30
  periodSeconds: 2

Three mechanisms exist beyond httpGet. exec runs a command inside the container and treats exit code 0 as success and anything else as failure. tcpSocket just checks whether a TCP connection to a given port succeeds, no HTTP involved at all. grpc, stable since 1.27, calls the standard gRPC health checking protocol directly.

livenessProbe:
  exec:
    command:
      - cat
      - /tmp/healthy
readinessProbe:
  tcpSocket:
    port: 5432

tcpSocket is the common choice for anything that isn't HTTP: a database, a message broker, a raw TCP service with no health endpoint at all. exec is the fallback when nothing else fits, checking for a file's existence or running a small script, but it's the slowest of the three since it forks a process inside the container on every single check.

A typo in the image name or tag is the most common cause by a wide margin. Second: missing or wrong imagePullSecrets for a private registry. Third: the image genuinely doesn't exist at that tag, someone forgot to push it, or a CI job tagged the wrong build. Fourth, and rarer: registry rate limiting, which shows up as intermittent pull failures rather than a consistent one.

CrashLoopBackOff means the container keeps exiting and Kubernetes keeps restarting it with an increasing backoff delay between attempts, up to 5 minutes. The container itself, not the Pod, is what's crashing, so the cause is almost always inside it: a bad command or entrypoint, a missing environment variable the app requires at startup, a dependency such as a database or config file that isn't reachable yet, or the app hitting an unhandled exception seconds after boot.

kubectl logs <pod-name> --previous
kubectl describe pod <pod-name>

--previous is the detail people forget. By the time you notice the crash loop, the current container attempt might not have logged anything useful yet, but the one before it usually did. If logs are empty even with --previous, the crash is happening before the app can log anything at all, which usually points at the entrypoint or command being wrong, not the application code.

The kube-apiserver is centralized, one logical instance (usually run 3-way for high availability) that every other component talks to. The kubelet is per-node: an agent that watches the API server for Pods assigned to its own node and makes sure the containers described in those PodSpecs are actually running and healthy.

Kubelets never gossip with each other directly. There's no peer-to-peer layer. Everything routes through the apiserver, which is what lets Kubernetes enforce RBAC and admission control consistently instead of trusting whatever a node claims about itself.

Worker nodes run application Pods. Control plane nodes run the apiserver, scheduler, controller-manager, and usually etcd (in a "stacked" topology, though etcd can also run externally). Most production clusters run 3 or 5 control plane nodes specifically so no single failure takes the whole thing down.

If you're running a single control plane node and it dies, already-running Pods on worker nodes keep serving traffic, since kube-proxy rules and running containers don't depend on the control plane staying up. What stops working: no new scheduling, no self-healing (a crashed Pod won't get replaced), no kubectl access, and no rollouts. It's degraded, not dead, and that distinction trips up candidates who assume "control plane down" means "cluster down."

A static Pod is defined by a manifest file sitting directly on a node's filesystem, usually under /etc/kubernetes/manifests, and the kubelet on that node watches the directory itself, not the API server, to decide what should be running. The kubelet still creates a mirror Pod object in the API server so kubectl get pods shows it, but that mirror is read-only. Deleting it through kubectl accomplishes nothing, the kubelet just recreates it because the real source of truth is the file on disk.

This is exactly how kubeadm bootstraps a cluster: kube-apiserver, kube-scheduler, kube-controller-manager, and often etcd itself run as static Pods on the control plane nodes. It's a neat fix for a chicken-and-egg problem, the apiserver needs to be running before normal Pods can get scheduled, but the apiserver can't be a normal Pod if nothing's running yet to schedule it.

A sidecar is a second container in the same Pod that supports the main application container, a service mesh proxy, a log forwarder, a config-reloader, without being bundled into the same container image. It has its own lifecycle, its own image, its own restart behavior, but it shares the Pod's network and can share volumes with the main container.

As of Kubernetes 1.29 (default-enabled) and stable since 1.33, sidecars can be declared as init containers with restartPolicy: Always. That fixes a real problem the old pattern had: native sidecars start before the main container, and Kubernetes waits for a sidecar's own startup probe to pass before moving on, instead of racing both containers to start at the same time and hoping the proxy comes up first (Kubernetes docs, Sidecar Containers).

apiVersion: v1
kind: Pod
metadata:
  name: app-with-sidecar
spec:
  initContainers:
    - name: log-shipper
      image: fluent-bit:2.2
      restartPolicy: Always
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
  containers:
    - name: app
      image: myapp:1.4
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
  volumes:
    - name: logs
      emptyDir: {}

Pending, Running, Succeeded, Failed, and Unknown. Pending means the Pod's been accepted by the cluster but at least one container image is still being pulled, or the Pod hasn't been scheduled yet at all.

Running only means at least one container is running, not that the application inside is healthy or serving traffic. That's the gap readiness probes exist to close. A Pod can sit in Running for hours while its actual app is deadlocked, and kubectl get pods will show it as if nothing's wrong unless something's actually checking.

The Deployment controller creates a new ReplicaSet with the updated Pod template and revision 0 replicas. It then scales the new ReplicaSet up and the old one down incrementally, bounded by maxSurge (how many extra Pods beyond the target count it can create at once) and maxUnavailable (how many Pods below the target count it can tolerate at once).

Each new Pod has to pass its readiness probe before the controller scales the old ReplicaSet down further, that's the mechanism that actually prevents a bad rollout from taking down all your capacity at once.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: checkout:2.6.1
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5

Each StatefulSet Pod gets a DNS name in the form <pod-name>.<service-name>.<namespace>.svc.cluster.local, and that name is stable across restarts because the Pod name itself is stable. A headless Service, one created with clusterIP: None, is what makes this resolvable. Instead of load-balancing across every Pod behind one virtual IP the way a normal Service would, a headless Service returns the individual Pod IPs directly through DNS.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: postgres:16

That's the mechanism a client uses to reach web-1 specifically instead of whichever Pod answers first, which matters for a primary-replica database setup where writes have to land on one particular Pod.

The Service gets a stable virtual IP and selects Pods by label, which populates an Endpoints object (or EndpointSlice, the newer, more scalable version) with the real Pod IPs currently behind it. kube-proxy on every node watches the API server for changes to Services and Endpoints and programs actual packet routing, historically through iptables rules, though IPVS or an eBPF dataplane like Cilium scale better once a cluster has thousands of Services.

When a client hits the Service's virtual IP, the node intercepts the packet and rewrites the destination to one of the healthy Pod IPs behind it. The client never talks to a Pod IP directly, which is exactly what lets Pods die and get replaced without the client noticing anything changed.

CoreDNS runs as a cluster-internal DNS server, and every Pod's /etc/resolv.conf is configured by the kubelet to send DNS queries there by default. When a Pod looks up checkout.prod.svc.cluster.local, CoreDNS resolves it to the Service's stable ClusterIP, not to any specific Pod. Resolution to an actual Pod happens later, at the kube-proxy layer, once a packet actually gets sent to that ClusterIP.

The short form, just checkout from within the same namespace, works because of the search domains Kubernetes automatically appends to resolv.conf. Cross-namespace calls need at least checkout.prod. Dropping the namespace is the single most common reason "it works from this namespace but not that one" bugs show up, and it's rarely a networking problem at all, it's just an incomplete DNS name.

A Service load-balances traffic once it's already inside the cluster and operates at Layer 4, it doesn't understand HTTP paths, hosts, or headers. An Ingress adds Layer 7 routing on top: host-based and path-based rules, and usually TLS termination, so one entry point can route api.example.com and app.example.com to two completely different backend Services.

An Ingress resource by itself does nothing. It's just a set of rules. You need an Ingress Controller actually running in the cluster (nginx-ingress and Traefik are the common ones) to read those rules and configure the real proxy that does the routing. A fresh cluster with an Ingress manifest applied and no controller installed will just sit there, rules defined, nothing happening.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-routes
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80

The most specific path match wins, not first-defined-in-the-file order. An exact pathType: Exact match on /api/health beats a pathType: Prefix match on /api, and a longer prefix beats a shorter one.

This is a real source of confusion when two teams add rules to the same Ingress resource (or two separate Ingress resources for the same host) without checking what's already there. Most Ingress Controllers log a warning about conflicting rules, but not all of them do, and a silent conflict routes traffic somewhere nobody expected.

Usually yes, eventually. Volume-mounted ConfigMaps sync on an interval (the kubelet's sync period, typically under a minute) so the file inside the container updates without restarting anything. Environment-variable-based ConfigMap values behave completely differently: they're injected once at container start and never update until the Pod is recreated.

That's the trap. A team that switches from env vars to a mounted volume assuming "same behavior, just a different mechanism" gets surprised the first time they expect a restart-free config change and it doesn't apply, because they never actually changed how they were consuming it.

Setting immutable: true on a ConfigMap or Secret does exactly what it sounds like, any attempt to edit its data after creation gets rejected by the API server. To change a value, you create a new object, commonly with a content hash in the name, and update whatever references it, rather than editing in place.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-a1b2c3
data:
  LOG_LEVEL: "info"
immutable: true

Two real reasons to do this, not just caution. First, it stops an accidental edit to a shared ConfigMap from silently affecting every Deployment that references it. Second, and this one surprises people, the kubelet doesn't need to keep watching an immutable object for changes, which measurably cuts load on the API server in clusters with thousands of ConfigMaps mounted across thousands of Pods (Kubernetes docs, Immutable ConfigMaps). It's a performance feature wearing a safety feature's clothes.

A StorageClass is a template for dynamically provisioning PVs on demand, instead of an admin pre-creating a pool of PVs by hand and hoping the sizes match what gets requested. It defines the provisioner (which storage backend to use), parameters like disk type, and a reclaim policy for what happens to the underlying storage once the PVC is deleted.

Immediate binding provisions the volume as soon as the PVC is created, before any Pod using it is even scheduled. WaitForFirstConsumer delays provisioning until a Pod actually references the PVC, which matters in multi-zone clusters, an Immediate-bound volume can end up in a different availability zone than the Pod that needs it, and the Pod will sit unschedulable forever because storage and compute can't cross zones.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

A ResourceQuota caps the aggregate resource consumption across an entire namespace, total CPU requests can't exceed 20 cores, total Pod count can't exceed 50. A LimitRange sets defaults, minimums, and maximums for individual containers, so a container that doesn't specify its own requests or limits gets a sane default instead of none at all.

You need both together the moment a namespace has a ResourceQuota that requires every Pod to specify requests and limits (which most quotas do), because without a LimitRange providing defaults, any Pod submitted without explicit requests gets rejected outright rather than falling back to something reasonable.

A request is what the scheduler uses to decide which node has room for a Pod, it's a reservation, not a cap. A limit is the real ceiling: exceed it on memory and the container gets OOMKilled, exceed it on CPU and the container gets throttled, not killed. Kubernetes derives a QoS class from the relationship between the two. Guaranteed applies when every container's requests equal its limits for both CPU and memory. Burstable applies when at least one request is set but doesn't equal its limit. BestEffort applies when neither is set at all.

QoS class decides eviction order under node memory pressure, and it's not close: BestEffort Pods get evicted first, then Burstable ones exceeding their requests, and Guaranteed Pods last, in practice almost never. A database Pod without explicit requests and limits is BestEffort by default, which means it's first in line to get killed the moment a noisy neighbor Pod on the same node starts eating memory, regardless of how important that database actually is.

A nodeSelector is opt-in from the Pod's side: it says "only schedule me on nodes with this label," and Pods that don't specify it can still land anywhere. A taint is opt-out from the node's side: it repels every Pod by default, and only Pods with a matching toleration are allowed to schedule there.

That's the whole reason taints exist. A nodeSelector can steer Pods toward a node, but it can't stop a Pod that doesn't care about labels at all from landing there. Taints are how you dedicate a node (a GPU node, say) to only the workloads that explicitly tolerate being scheduled on it.

NoSchedule blocks new Pods from being scheduled there unless they tolerate it, but Pods already running on the node when the taint is added stay put. PreferNoSchedule is the soft version, the scheduler tries to avoid the node but will still place a Pod there if nothing else fits.

NoExecute is the one people forget: it evicts Pods that are already running and don't tolerate the taint, not just new ones. Add a NoExecute taint to a node that's about to be drained for maintenance, and every non-tolerating Pod already on it gets kicked off immediately, which is exactly the behavior you want for planned node decommissioning.

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300

A PodDisruptionBudget (PDB) sets a floor, a minimum number or percentage of Pods from a given selector that must stay available, that voluntary disruptions aren't allowed to cross. kubectl drain, cluster autoscaler scale-downs, and node upgrades all check the PDB before evicting a Pod. If evicting one would violate the budget, the eviction is refused, not queued, refused, until enough replacement Pods come up elsewhere.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout

PDBs only cover voluntary disruptions, things an operator or controller chooses to do. They do nothing for involuntary ones, a node hardware failure or a kernel panic doesn't check a PDB first, it just happens. That distinction is worth stating explicitly if asked, because "does a PDB protect me during an outage" and "does a PDB protect me during a planned drain" have different answers.

HPA changes replica count in response to CPU usage. VPA changes each container's CPU request and limit in response to the same observed usage. Point both at CPU for the same workload and they can fight: VPA resizes a Pod's request upward, which changes the per-Pod CPU utilization percentage HPA is watching, which triggers HPA to scale differently, which changes aggregate usage per Pod again.

The common fix is splitting responsibilities: VPA in "Off" or recommendation-only mode just to size requests sensibly, HPA doing the actual live scaling. Running both in fully automatic mode on the same metric is the kind of setup that looks fine in a demo and gets weird two weeks into real traffic.

Kubernetes assumes a flat network model: every Pod gets its own IP, and every Pod can reach every other Pod's IP directly, no NAT, regardless of which node they're on. Docker's default bridge networking doesn't give you that across multiple hosts out of the box.

A CNI plugin (Calico, Cilium, and Flannel are the common ones) is what actually implements that flat network across nodes, wiring up routes or an overlay so cross-node Pod-to-Pod traffic just works. Different CNIs make different tradeoffs, Cilium's eBPF dataplane, for instance, scales noticeably better than iptables-based routing once a cluster has thousands of Services, but the core contract, every Pod reachable from every other Pod, is the same regardless of which one you pick.

Whether your CNI plugin actually enforces NetworkPolicy at all. This is the single most common cause of a "policy that does nothing." Flannel, for example, doesn't enforce NetworkPolicies by default, it just handles the networking, not policy enforcement, so a NetworkPolicy object sits there parsed and accepted by the API server but never actually applied to traffic.

Calico, Cilium, and most enterprise-grade CNIs do enforce it. If you're on one of those and it's still not working, the next check is whether the policy's selector actually matches the Pods you think it does, a label typo in the selector is a close second on the list of causes.

No, it's not a default, Kubernetes ships with fairly restrictive out-of-the-box RBAC. Cluster-wide Secret read access almost always traces back to someone binding a permissive ClusterRole (or worse, cluster-admin) to a ServiceAccount because a narrower Role would've taken more time to write correctly.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: payments
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secret-reader-binding
  namespace: payments
subjects:
  - kind: ServiceAccount
    name: payments-api
    namespace: payments
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Replace the ClusterRoleBinding with a namespaced Role and RoleBinding scoped to exactly the Secrets that ServiceAccount needs, and audit what's currently bound with kubectl get clusterrolebindings -o wide before assuming the fix worked.

Start with kubectl describe pod, the readiness probe's failure reason is usually right there in the events section. If the probe's an HTTP check, confirm the path and port actually match what the app listens on, a surprising number of these bugs are just a probe pointed at the wrong port after a config change.

kubectl describe pod checkout-7f8b9c-xk2q9 -n prod
kubectl logs checkout-7f8b9c-xk2q9 -n prod
kubectl exec -it checkout-7f8b9c-xk2q9 -n prod -- curl -s localhost:8080/ready
kubectl get endpoints checkout -n prod

If the app looks healthy from inside the container but the Pod's still not in the Service's Endpoints, check the label selector on the Service against the Pod's actual labels next, a mismatched label is invisible in most dashboards but obvious the moment you diff the two.

etcd uses the Raft consensus algorithm, and Raft requires a strict majority of members to agree before a write is considered committed. For a 5-node cluster, quorum is 3. Losing exactly 2 members still leaves 3 alive, which is a majority, so the cluster keeps accepting writes.

Lose 3 at once and you're down to 2 members, below the majority of 3. The cluster can't elect a leader or commit new writes at all. Existing Pods that are already scheduled keep running (kubelets don't need etcd to keep containers alive), but nothing new gets scheduled, no Deployment can roll out, and the API server itself starts rejecting most write requests until quorum is restored.

The new ReplicaSet's Pods keep crashing before passing their readiness probe, so the Deployment controller never advances past maxUnavailable, it's protecting you from a bad rollout, not stuck by accident. kubectl rollout status will sit there waiting forever because "done" never arrives.

kubectl rollout status deployment/checkout -n prod
kubectl get pods -n prod -l app=checkout
kubectl logs <pod-name> -n prod --previous
kubectl rollout undo deployment/checkout -n prod

Check kubectl logs --previous on one of the crashing Pods first, most of the time it's a config problem or a bad dependency in the new image. If it's not a quick fix, kubectl rollout undo rolls back to the last good revision immediately. Debug the actual cause after the bleeding's stopped, not during.

By default, nothing. The Pods terminate, but their PersistentVolumeClaims stay behind untouched. Scale back up to 5 and the new web-3 and web-4 Pods reattach to the exact same PVCs they had before, same data, not a fresh volume. That's deliberate. A StatefulSet assumes scaling down is often temporary, not a signal to throw the data away.

It surprises people in the other direction too: deleting a StatefulSet entirely, not just scaling it down, still leaves the PVCs behind unless you delete them separately, which is a safety net right up until it's the reason a namespace teardown didn't actually free the storage everyone assumed it would. Kubernetes 1.27 added an optional persistentVolumeClaimRetentionPolicy field if you want PVCs deleted automatically on scale-down or StatefulSet deletion instead of the conservative default.

spec:
  persistentVolumeClaimRetentionPolicy:
    whenScaled: Delete
    whenDeleted: Retain

Start with kubectl describe pvc, the events section almost always names the actual problem: no StorageClass matches what was requested, the requested size exceeds what any available PV or the provisioner's quota allows, or (with WaitForFirstConsumer) there's simply no Pod referencing it yet, which looks identical to a real problem but isn't one.

If a provisioner is involved, check its controller Pod logs next, a misconfigured cloud IAM permission preventing the CSI driver from actually creating the underlying disk is a common cause that describe pvc alone won't always surface clearly.

Node affinity steers Pods toward (or away from) nodes based on node labels, like preferring nodes in a specific availability zone or with an SSD-labeled node pool. Pod anti-affinity steers Pods away from other Pods based on their labels, most commonly used to spread replicas of the same Deployment across different nodes or zones so a single node failure doesn't take out every replica at once.

Both come in requiredDuringSchedulingIgnoredDuringExecution (a hard rule, the scheduler won't place the Pod if it can't satisfy this) and preferredDuringSchedulingIgnoredDuringExecution (best-effort) flavors. I've seen more outages caused by making pod anti-affinity required, and then the cluster running out of eligible nodes during a scale-up, than by any anti-affinity misconfiguration in the other direction. Preferred is usually the safer default unless you genuinely need the hard guarantee.

Yes, and it's one of the more counterintuitive things about how the scheduler bin-packs. It sums up the requests of every Pod already placed on a node, not their actual live usage, and compares that sum against the node's allocatable capacity. A node can be scheduling-full on paper, every Pod's requested memory adding up to the node's capacity, while every Pod is actually using a fraction of what it requested, leaving real RAM sitting unused that the scheduler will never place a new Pod into.

This is exactly why teams that set requests well above real usage "just to be safe" end up with clusters that look 90% allocated in a dashboard but run at 30% actual utilization, and adding nodes doesn't fix it, tightening requests to match real usage does. It's also why VPA in recommendation mode is worth running even if you never turn on its automatic resizing, seeing the gap between requested and actual usage is often the first time a team notices how much headroom it's been paying for.

Most likely, the stabilization window is too short, or missing entirely, so a brief spike scales up, the spike passes, it scales back down, and the next spike starts the cycle over. HPA's default scale-down stabilization window is 5 minutes, but if that's been overridden to something small, or the metrics being sampled are genuinely noisy, this exact flapping pattern shows up.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
    scaleUp:
      stabilizationWindowSeconds: 60

Widening the stabilization window, and checking whether the workload's CPU usage is genuinely spiky versus an artifact of too few replicas averaging out the noise, are the two things to check before assuming the HPA config itself is broken.

A ServiceAccount dedicated to the pipeline, never a shared one. A Role scoped to that single namespace with only the verbs it actually uses, typically get, list, update, and patch on Deployments (for the rollout), plus maybe get on Pods for status checks. No delete unless the pipeline genuinely tears things down. No access to Secrets or RBAC objects themselves.

I've seen more CI pipelines granted cluster-admin "temporarily, to get it working" than pipelines that ever got that permission scoped back down afterward. The honest answer to "why is this pipeline so over-permissioned" is almost always deadline pressure the first time it was set up, not a deliberate decision.

RBAC answers "is this user allowed to make this request at all." Admission control runs after that check passes and before the object is persisted to etcd, and it answers a different question: even though you're allowed to submit this, does it meet the cluster's policy. A mutating admission webhook can change the object on the way in, injecting a sidecar, setting a default resource limit a team forgot. A validating webhook can only accept or reject it, no edits, and validating webhooks always run after every mutating webhook has already had a chance to modify the object.

Kyverno and OPA Gatekeeper are the two tools most commonly built on this. Rules like "every Deployment must set resource limits" or "no image from an untrusted registry" get enforced here, not in RBAC, because RBAC has no concept of what a Pod spec actually contains (Kubernetes docs, Admission Controllers). I've watched candidates confidently answer the RBAC question and then go quiet the moment "how do you stop someone from deploying a container running as root" comes up, because the real answer lives outside RBAC entirely.

Pod Security Admission (PSA), a built-in admission controller, replaced PodSecurityPolicy, and it's applied per namespace with a label rather than a separate cluster-wide policy object. Privileged enforces nothing, it exists mostly so infrastructure namespaces running genuinely privileged workloads, a CNI plugin, say, have somewhere to opt out cleanly. Baseline blocks the most obviously dangerous settings, running as a privileged container, using the host's network or PID namespace, while still allowing a fairly permissive default Pod spec otherwise. Restricted is the hardened tier: no root user allowed, no privilege escalation, a required seccomp profile, and dropped Linux capabilities by default.

The label goes directly on the namespace, and you can set warn or audit modes alongside enforce, which is the practical rollout path, run Restricted in warn mode against real traffic for a couple weeks, see what actually breaks, before flipping it to enforce and finding out the hard way (Kubernetes docs, Pod Security Admission). Most teams I've seen skip the warn phase entirely and pay for it with a production incident the day they turn Restricted on for real.

apiVersion: v1
kind: Namespace
metadata:
  name: prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

The probe timeout is shorter than the app's actual response time under real load, so Kubernetes concludes the container is dead and restarts it, and the restart itself adds more load to the remaining Pods, which makes their probes slower too. It's a self-inflicted outage that looks identical to a genuine crash from the outside.

Fix: raise the liveness timeout and failure threshold to something realistic under actual production load, not the load from a quiet staging environment, and separate the liveness check (something cheap and internal, "is my process responding at all") from a readiness or deep health check that might touch a database or downstream dependency. A liveness probe that checks database connectivity is a common mistake, a slow database now restarts your application containers too.

137 is 128 plus signal 9 (SIGKILL), almost always OOMKilled in a Kubernetes context, the container exceeded its memory limit and the kernel killed it with no chance to clean up. 143 is 128 plus signal 15 (SIGTERM), a graceful shutdown request, usually from a Pod deletion or rolling update, that the app either handled and exited, or ignored until the grace period ran out and got a 137 anyway.

kubectl get pod checkout-7f8b9c-xk2q9 -n prod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
kubectl get pod checkout-7f8b9c-xk2q9 -n prod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

kubectl describe pod usually names the reason directly (OOMKilled shows up as text, not just a number), but the exit code confirms it when the reason field is empty or ambiguous. A 137 means go raise the memory limit or find the leak. A 143 that keeps recurring on every deploy usually means the app isn't handling SIGTERM at all, and is just getting force-killed once its grace period expires.

Averages hide spikes. cgroups enforce the memory limit instantly, the moment usage crosses it even for a fraction of a second, the kernel kills the container. It doesn't care what the average looked like over the last hour. A JVM doing a large garbage collection pass, a request handler loading an oversized payload into memory, or a batch job processing one unusually large record can all spike past the limit briefly and get killed, while every dashboard sampled at 30-second intervals shows a flat, comfortable line.

The fix isn't raising the limit blindly, it's checking the actual memory ceiling the process needs during its worst moment, not its typical moment. kubectl top pod polled frequently, or better, the container's own memory metrics scraped at a few-second interval, will usually show the spike a dashboard built for capacity planning was never sampling fast enough to catch.

Ready doesn't mean correct. All three Pods can be passing a shallow readiness probe (a plain /healthz that just checks the process is up) while a downstream dependency, a database, a config value, a feature flag, is actually broken for all three at once. Since every replica shares the same bad state, this looks nothing like a typical single-Pod failure.

Check application logs across all three Pods for the actual error, not just their health status. Then check what changed recently that all three would share: a ConfigMap update, a Secret rotation, a downstream service deploy, a database migration. The bug almost never lives in "which Pod," it lives in whatever all three Pods have in common.

An emptyDir volume, declared once at the Pod level and mounted into both containers, is the standard fix. It's created fresh when the Pod is scheduled to a node and deleted when the Pod is removed, so it's not durable storage, just a shared scratch space for the Pod's lifetime.

apiVersion: v1
kind: Pod
metadata:
  name: shared-scratch
spec:
  containers:
    - name: writer
      image: writer:1.0
      volumeMounts:
        - name: shared-data
          mountPath: /data
    - name: reader
      image: reader:1.0
      volumeMounts:
        - name: shared-data
          mountPath: /data
          readOnly: true
  volumes:
    - name: shared-data
      emptyDir: {}

The detail interviewers listen for: emptyDir survives a container restart within the same Pod (Kubernetes doesn't delete it just because one container crashed and got restarted), but it's gone the moment the whole Pod is rescheduled or deleted. Candidates who assume it's persistent storage are going to have a bad time in production.