Kubernetes Admin Interview Questions: The Governance Layer Most Candidates Skip

An application developer describes desired state, image, replica count, resource requests, and trusts the platform underneath it. An administrator owns that platform: control plane availability, etcd health, upgrade cadence, RBAC boundaries, capacity planning, and what happens when any one of those breaks at an inconvenient hour.

Interviewers use this question mostly to calibrate scope before the rest of the loop. A candidate who answers with object definitions instead of operational ownership usually gets steered toward the application-developer track for the remainder of the conversation.

A static Pod is defined by a manifest file sitting on a node's own filesystem and watched directly by the kubelet, not scheduled through the API server (though a read-only mirror Pod object gets created so kubectl get pods can still see it).

The control plane runs this way on kubeadm-based clusters for a chicken-and-egg reason: the apiserver, scheduler, and controller-manager all need something to start them before the API server itself exists to do the scheduling. Static Pods sidestep that entirely, the kubelet just starts whatever manifest is sitting in /etc/kubernetes/manifests.

The kubelet watches node-level resource pressure signals, available memory, available disk on the node's filesystem, available disk for images. Cross a configured hard eviction threshold and the kubelet starts evicting Pods on its own, ranked by QoS class and how far over their requested usage they've drifted, to reclaim the resource before the kernel's OOM killer has to step in and pick something at random.

Shared cluster capacity is finite, and one team's runaway Deployment can starve every other namespace of schedulable room. A ResourceQuota caps aggregate CPU, memory, and object counts per namespace, so a mistake stays contained to the team that made it instead of becoming everyone's incident.

A DaemonSet guarantees exactly one Pod per matching node, used for node-level agents: log shippers, CNI agents, monitoring exporters, security scanners. A Deployment has no concept of "one per node" at all, it just keeps a replica count running wherever the scheduler decides.

ResourceQuota caps the namespace's aggregate consumption. LimitRange sets defaults, minimums, and maximums per container. You need both the moment a ResourceQuota requires every Pod to specify requests and limits, because without a LimitRange supplying defaults, an unspecified Pod gets rejected outright instead of falling back to something reasonable.

A Service load-balances at Layer 4 inside the cluster and has no concept of HTTP paths or hosts. An admin installs and actually operates an Ingress Controller (nginx, Traefik, or a cloud-managed one) that reads Ingress objects and configures the real Layer 7 routing, TLS termination, and often rate limiting, none of which exists until that controller is running somewhere.

A ClusterRole with get, list, and watch on nodes, Pods, Services, and endpoints, plus non-resource access to metrics paths, bound via ClusterRoleBinding to Prometheus's own ServiceAccount. Not cluster-admin. Read-only and metrics-scoped, with no ability to write anything or read Secrets it has no business reading.

Authentication answers "who are you," through client certificates, tokens, or OIDC integration. Authorization answers "what are you allowed to do," almost entirely through RBAC (Role, ClusterRole, RoleBinding, ClusterRoleBinding), though ABAC and webhook authorization exist for edge cases. Admins configure authentication mostly at the API server's flags and identity provider integration, and authorization almost entirely through RBAC objects.

The kubeconfig context first, wrong cluster selected, an expired client certificate, or a load balancer VIP that moved. Then whether the API server process or Pods are actually up on the control plane nodes. Then the network path, a security group or firewall rule that changed recently. Most "kubectl is broken" incidents turn out to be kubeconfig or network path problems, not the API server itself being down.

A Namespace partitions one physical cluster into multiple virtual clusters for the purposes of naming, RBAC, and resource quotas. It doesn't give you network isolation on its own, that's what NetworkPolicy is for, and it doesn't give you compute isolation either, a noisy Pod in namespace A can still starve a node that a Pod in namespace B happens to land on. What a Namespace actually buys you is a scope for object names (two Deployments named "api" can coexist as long as they're in different namespaces), a scope for RoleBindings, and a scope for ResourceQuota and LimitRange objects.

In practice, most teams split namespaces by team or by environment (dev, staging, prod) rather than by individual service, because the RBAC and quota boundaries map more naturally onto "which humans can touch this" than onto "which microservice is this." Running everything in default is fine for a single-person demo cluster, but once more than one team shares the cluster, you lose the ability to reason about ownership, and you can't apply a quota without it silently applying to everyone in it.

Both objects are just key-value stores you can mount into Pods as environment variables or files, and by default a Secret's values are only base64-encoded, not encrypted. Base64 is an encoding, not encryption, so anyone with API access to read Secrets, or read the raw object through kubectl get secret -o yaml, can decode the value in one command. So the difference between the two isn't really about protection out of the box, it's about intent and about what the platform does around the object.

Kubernetes and its ecosystem treat Secrets differently in a few concrete ways: they get their own RBAC resource type, so you can grant a team read access to ConfigMaps without granting read access to Secrets in the same namespace. They also integrate with encryption-at-rest providers and external secret backends like Vault or a cloud provider's secret manager through the CSI Secrets Store driver, in a way ConfigMaps never need to. As an admin, the actual security comes from enabling encryption at rest for the secrets resource in etcd, restricting which ServiceAccounts and users can read secrets, and, where possible, keeping the source of truth in an external secrets manager instead of raw Kubernetes Secret objects.

kube-proxy is the component that makes a Service's virtual IP actually route to real Pod IPs. It runs as a DaemonSet on every node, watches the API server for Service and EndpointSlice objects, and programs the node's networking layer, iptables rules in the classic mode or IPVS rules in ipvs mode, so traffic sent to a Service's ClusterIP gets translated to one of the backing Pods.

If kube-proxy stops running or crashes on a node, that node loses its ability to translate Service IPs into Pod IPs for anything originating from it. Pods already scheduled there can still be reached directly by their Pod IP from elsewhere in the cluster, since that path is handled by the CNI, but they can't reliably call other Services by ClusterIP themselves, and NodePort traffic that lands on that specific node won't get forwarded correctly. This is a common root cause behind "some Pods can reach the database and others can't" reports that turn out to be one node with a dead kube-proxy Pod, not an application bug at all.

All three Service types do the same core job, give a stable virtual IP and DNS name that load-balances across a set of Pods, they just differ in how that IP gets exposed outside the cluster. ClusterIP is internal only. NodePort opens a fixed port, in the 30000 to 32767 range by default, on every node and forwards it to the Service, reachable from outside only if you can already reach a node's IP directly, and that port range runs out fast once you have more than a few dozen exposed Services. LoadBalancer asks the cloud provider, or a controller like MetalLB on bare metal, to provision an actual external load balancer sitting in front of the NodePort underneath.

A developer often reaches for LoadBalancer because it's the one that just works and hands them a public IP, without realizing that on a cloud provider each LoadBalancer Service typically provisions a real, separately billed load balancer resource. An admin's job is usually to route external traffic through a single Ingress controller backed by one or two LoadBalancer Services, rather than letting every team spin up their own, both to control cost and because managing TLS certs, routing rules, and rate limits across dozens of separate cloud load balancers doesn't scale operationally.

They solve opposite problems. A taint sits on a node and repels Pods by default, and a toleration sits on a Pod and says it's allowed to ignore that particular taint. Taints are how you keep general workloads off nodes reserved for something specific, GPU nodes, nodes mid-drain, or control plane nodes, without having to edit every other Deployment's spec to avoid them.

Node affinity works the other direction: it's a rule on the Pod saying only schedule me on nodes matching this label, or preferring nodes matching this label, and it doesn't stop any other Pod from landing on those same nodes unless you also add anti-affinity or a taint. In practice you use both together for something like a GPU pool: taint the GPU nodes so ordinary workloads can't accidentally consume expensive capacity, then give the training Pods both a toleration for that taint and a node affinity rule targeting the GPU label, so they're the only Pods that both can and want to land there. Using either alone usually gets you half of what you need, taints alone don't guarantee the right Pods find those nodes, and affinity alone doesn't stop the wrong Pods from landing there too.

An etcd snapshot (etcdctl snapshot save) captures every Kubernetes object definition currently stored in etcd's key-value store: every Deployment, Secret, ConfigMap, and RBAC binding, as desired state. It does not contain running container images, process memory, or the actual data sitting inside a PersistentVolume.

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db 
 --endpoints=https://127.0.0.1:2379 
 --cacert=/etc/kubernetes/pki/etcd/ca.crt 
 --cert=/etc/kubernetes/pki/etcd/server.crt 
 --key=/etc/kubernetes/pki/etcd/server.key

etcdctl snapshot status /backup/etcd-snapshot-20260718.db --write-out=table

That gap is why an etcd backup alone never counts as a full disaster recovery plan. You need PV data backed up separately, through Velero or a CSI snapshot, or a restore just gives you a cluster that remembers what should exist with nothing actually in the databases (Kubernetes docs, Operating etcd clusters).

Upgrade the first control plane node completely before touching anything else: kubeadm upgrade plan, then kubeadm upgrade apply vX.Y.Z, then drain that node, upgrade its kubelet and kubectl packages, and uncordon it. Every additional control plane node after the first uses kubeadm upgrade node instead of upgrade apply, since the cluster-wide upgrade only happens once.

kubectl drain cp-node-1 --ignore-daemonsets
kubeadm upgrade apply v1.30.2
apt-get update && apt-get install -y kubelet=1.30.2-* kubectl=1.30.2-*
systemctl restart kubelet
kubectl uncordon cp-node-1

Worker nodes come last, one at a time: drain, upgrade the kubelet package, restart it, uncordon. Doing them all at once instead of one at a time is the fastest way to turn a routine upgrade into an actual incident (Kubernetes docs, kubeadm upgrade).

Cordon marks a node unschedulable for new Pods but leaves whatever's already running untouched. Drain does that too, and then evicts the existing Pods, respecting PodDisruptionBudgets along the way.

You'd cordon alone when you want to freeze a node for inspection, checking disk pressure, reading kubelet logs, without disturbing the workloads currently on it. Drain is for the moment you actually intend to take the node away.

A genuinely different hardware need (GPU nodes, high-memory instances), a workload isolation requirement that compliance or a customer contract actually demands, a different OS or kernel version requirement, or workloads that need different scaling economics entirely (spot instances for batch jobs versus on-demand for anything user-facing).

Taints paired with tolerations are what actually keep general workloads off a dedicated pool once it exists. A node pool without a taint is just an expensive suggestion.

Every kubelet, the apiserver, and etcd's peer and client connections all run on certificates with an expiry, kubeadm defaults most of them to a year. If a kubelet's client certificate expires and rotation wasn't enabled, that node can't renew its lease or report status anymore and quietly drops out of the cluster, no crash, no obvious error, just a node that stops updating.

kubeadm automates renewal during an upgrade. The actual risk sits with clusters that go a year or more between version bumps and never touch certificates in between on purpose.

If a ResourceQuota constrains compute resources in that namespace, every Pod submitted there must specify requests and limits or the object gets rejected at admission entirely, not capped, rejected outright. Without a LimitRange supplying sane defaults, a Pod that doesn't specify its own requests fails for a reason that has nothing to do with hitting a ceiling.

A PDB caps how many Pods from a set can be voluntarily disrupted at once, expressed as minAvailable or maxUnavailable. Without one, draining several nodes back to back during a maintenance window can evict every replica of a Deployment nearly simultaneously if they happen to land on those nodes, and nothing stops that from happening.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
 name: checkout-pdb
spec:
 minAvailable: 2
 selector:
  matchLabels:
   app: checkout

The scheduler looks for a node where evicting some set of lower-priority Pods would free enough room, and if the math works, it evicts them (with a normal graceful termination, not a hard kill) to make space.

It's not instant and it's not guaranteed. It can genuinely take down lower-priority workloads that were doing something important, just less officially important than whatever preempted them, which is a real trade-off worth naming out loud rather than glossing over (Kubernetes docs, Pod Priority and Preemption).

concurrencyPolicy: Forbid skips a new run entirely if the previous one is still active. A job that runs longer than its own schedule interval, with Forbid set, silently skips every trigger after the first until it finally finishes, no error, no alert, unless something's actually watching for it.

kubectl get cronjob nightly-report -n batch
kubectl get jobs -n batch --selector=job-name=nightly-report
kubectl logs -n batch -l job-name=nightly-report-28457200 --previous

Stable, predictable Pod identity (app-0, app-1, app-2) with per-Pod persistent storage through volumeClaimTemplates, plus ordered, one-at-a-time startup and scaling. It matters anywhere identity itself is meaningful, a database replica genuinely needs to know whether it's replica 0 or replica 2, not just for any workload that happens to hold data.

Whether it actually enforces NetworkPolicy (Flannel doesn't, Calico and Cilium do), how its dataplane performs at scale (iptables versus IPVS versus an eBPF approach), the IP address management model at your expected node and Pod count, and whether it exposes the network observability your security team is eventually going to ask for.

Yes, on a cron job or a managed backup tool, never as a manual step someone remembers to run. The part people get wrong: storing the snapshot on the same node, or even the same region, as the cluster it's backing up. A snapshot sitting on the control plane node that just went down with the rest of the region is not a backup, it's a coincidence.

Check CoreDNS Pod health and restart counts first, OOMKilled CoreDNS Pods under load is a common cause at scale. Then confirm the kube-dns Service's Endpoints are actually populated. Then check for node-level UDP conntrack issues, a known kernel-level race under high DNS query volume.

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl top pods -n kube-system -l k8s-app=kube-dns
kubectl get endpoints kube-dns -n kube-system

If it's intermittent and scales with cluster size rather than any one Pod's health, NodeLocal DNSCache is usually the actual fix, not a CoreDNS resource bump.

The bound PersistentVolume and its underlying storage (an EBS volume, say) get deleted along with the PVC. The data is gone unless a separate snapshot or backup exists, Kubernetes does not soft-delete storage on your behalf.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: db-storage
provisioner: ebs.csi.aws.com
parameters:
 type: gp3
reclaimPolicy: Retain

A Retain policy instead leaves the PV (and the disk) around in a Released state for manual recovery, at the cost of having to clean it up yourself later instead of it happening automatically.

Velero backs up Kubernetes object manifests by default. Actual PV data needs a CSI volume snapshot plugin configured explicitly, or file-level backup enabled through restic or Kopia. A team that only enabled Velero's default manifest backup gets every object back on restore except the data that was sitting inside the volumes.

An etcd snapshot covers Kubernetes' own object state, every Deployment, Service, Secret, and ConfigMap definition currently in the cluster. Velero, with a snapshot plugin configured, covers application data sitting in PVs plus namespace-scoped selective restores that etcd alone can't do.

Restoring only etcd gets you a cluster that knows what should exist with empty databases behind it. Restoring only Velero into a cluster that already has conflicting object state gets messy fast, the two backups solve different halves of the same problem.

Immediate binding provisions a volume the moment the PVC is created, before any Pod using it is scheduled, which can land the volume in a different availability zone than wherever the scheduler eventually places the Pod. WaitForFirstConsumer delays provisioning until a Pod actually references the PVC, so the volume gets created in the same zone the scheduler already picked.

It runs a DNS caching agent as a DaemonSet on every node, so most lookups resolve locally instead of hitting CoreDNS over the network for every single query. Worth it once cluster size and query volume are high enough that CoreDNS itself, or a node's UDP conntrack table, becomes a genuine bottleneck, not something to reach for preemptively on a 10-node cluster that isn't showing any symptoms yet.

Pull every ClusterRoleBinding and check which ones reference the cluster-admin ClusterRole, then trace each subject, user, group, or ServiceAccount, back to whether it genuinely needs that level of access.

kubectl get clusterrolebindings -o json | 
 jq -r '.items[] | select(.roleRef.name=="cluster-admin") |.metadata.name'
kubectl get clusterrolebinding <name> -o jsonpath='{.subjects}'

The uncomfortable finding is almost always a CI/CD ServiceAccount bound to cluster-admin "temporarily" during initial setup, and never scoped back down once things worked.

Don't loosen the cluster-wide default. Pod Security Admission is enforced per namespace through labels, so the actual fix is moving that one workload's namespace to a looser level, baseline, or privileged only if truly justified, while every other namespace stays at restricted.

apiVersion: v1
kind: Namespace
metadata:
 name: legacy-storage-agent
 labels:
  pod-security.kubernetes.io/enforce: privileged
  pod-security.kubernetes.io/warn: restricted

Loosening the whole cluster to fix one workload defeats the entire point of setting the policy in the first place (Kubernetes docs, Pod Security Admission).

API server CPU and disk overhead, and if you log everything at the RequestResponse level, a genuinely large volume of data fast. What's actually worth the cost: writes to RBAC objects at RequestResponse level, so you can reconstruct exactly what permission changed and who changed it, reads and writes to Secrets at Metadata level (who touched it, not the payload), and everything else at Metadata level as a baseline.

Check the webhook's failurePolicy first. Fail (the safer default) blocks every request if the webhook is unreachable. Ignore lets requests through unvalidated if it's down. If it's set to Fail and the webhook Pod itself is unresponsive, that's your actual root cause.

kubectl get validatingwebhookconfigurations -o yaml
kubectl get pods -n webhook-system

Fixing the webhook is the real answer. Patching failurePolicy to Ignore as an emergency, temporary measure gets the cluster functional again while that happens, but it's a stopgap, not a fix.

Base64 in a Secret object is encoding, not encryption. Anyone with direct read access to etcd, a stolen snapshot, a compromised node with etcd access, can decode every Secret in the cluster in one command. Encryption at rest through an EncryptionConfiguration means an attacker with just the raw etcd data still needs the encryption key, which should live somewhere etcd itself can't reach directly, a KMS provider, not a file sitting next to the data it's supposed to protect.

Kubelet certificate rotation is usually automatic if --rotate-certificates is enabled, but clusters that disabled it, or nodes offline during the rotation window, are the real failure case. Catching it early means alerting on certificate expiry dates directly, exporters exist that surface this as a Prometheus metric, rather than waiting to notice a node quietly went NotReady.

First I check the event stream with kubectl describe pod, since it usually names the real failure, not found, unauthorized, or a timeout, and that changes the direction of the investigation entirely. "Manifest not found" or "repository does not exist" usually is a typo or a tag that's been deleted from the registry, which happens constantly right after a CI job prunes old images. "Unauthorized" or "authentication required" means the imagePullSecret is missing, wrong, or scoped to the wrong namespace, since Secrets don't cross namespace boundaries, and a very common mistake is creating the pull secret once and forgetting every namespace needs its own copy, or a ServiceAccount patched to reference it.

If the error is a timeout rather than an explicit rejection, I'm checking whether the node can reach the registry at all. This shows up a lot with private registries sitting behind a firewall rule or VPN that only some node pools have network access through, or with a public registry rate-limiting anonymous pulls, Docker Hub's pull limits catch teams constantly once they're relying on public base images at any real scale, and the fix is authenticating pulls or mirroring images into your own registry. I'd also check for a stale cached credential on that node's container runtime after a registry token rotation, since kubelet doesn't always pick up a refreshed imagePullSecret cleanly without the Pod being recreated. And if only one node in the cluster fails to pull an image that every other node pulls fine, I'd suspect that node's containerd config or local disk pressure before I'd suspect anything about the image itself.

Stop the kube-apiserver static pod first (moving its manifest out of the kubelet's watched directory works fine), restore the snapshot into a fresh data directory, then point etcd's static pod manifest at that new directory and bring it back up as a fresh single-member cluster before restoring the apiserver manifest.

etcdctl snapshot restore /backup/etcd-snapshot-20260718.db 
 --name etcd-restore 
 --data-dir /var/lib/etcd-restored 
 --initial-cluster etcd-restore=https://10.0.0.5:2380 
 --initial-advertise-peer-urls https://10.0.0.5:2380

If you're restoring into an HA cluster, every remaining etcd member needs to point at the same restored data directory and the same fresh cluster ID, mismatched cluster IDs across members is the most common way this goes sideways. I don't have a clean number for how long a real restore should take on a busy production etcd, it depends heavily on cluster size and how much desired state you're carrying, but a runbook with no timing expectation written into it is worth fixing before it's ever needed for real.

A handful of usual suspects, roughly in order of how often each one is the actual cause: a Pod with the cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation, a Pod using local storage (an emptyDir) without being marked evictable, a bare Pod not owned by any controller (Cluster Autoscaler won't touch those), a kube-system Pod with no PodDisruptionBudget, or simply no room anywhere else in the cluster for the Pods that would need to be rescheduled.

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml
kubectl describe node <candidate-node-name>

The autoscaler-status ConfigMap actually names the blocking reason per node, most candidates don't know that exists and go straight to guessing instead of just reading it (Kubernetes Autoscaler FAQ).

Run it for real. Kill the etcd leader on a non-production cluster on purpose. Restore an actual snapshot into a scratch environment and time how long it takes. A runbook that's never been executed almost always has one wrong flag or one outdated command hiding in it somewhere, and the only way to find that out is running it before an actual outage forces the question.

I'd rather see a candidate say "we've never actually tested our restore" honestly than describe a theoretically perfect process nobody's run. The first answer is a real gap you can plan around. The second one usually falls apart under a single follow-up question.

Spread replicas across more nodes first so fewer land on the specific ones you're draining, scale the Deployment up temporarily during the drain and back down after, or just go slower, node by node, with patience instead of forcing it. kubectl drain --disable-eviction exists and skips PDB checking entirely, but reaching for it defeats the entire reason the PDB was there in the first place.

VPA in Auto mode evicts and recreates a Pod to apply a new resource request or limit. For a stateless web Pod that's a non-event. For a StatefulSet Pod carrying a database replica, an unplanned eviction in the middle of an incident is exactly the kind of automation that ends up in a postmortem.

Common practice: VPA in Off or recommendation-only mode for anything stateful, and reserve fully automatic resizing for stateless workloads that genuinely tolerate being killed and restarted without ceremony.

Whether the CNI actually enforces NetworkPolicy at all. Flannel accepts and stores the object without ever applying it to real traffic, which looks identical to "policy applied" in most dashboards. If the CNI does enforce it, check next whether the policy's label selector actually matches namespace X's Pods, a mismatched selector is a silent no-op that's invisible until you diff it directly.

Almost always a finalizer, typically kubernetes.io/pv-protection, still attached because a PVC (or a Pod still referencing that PVC) technically exists somewhere, or the underlying cloud volume's detach never actually completed and the CSI driver hasn't reported success back to the API server.

kubectl describe pv pv-checkout-data
kubectl get pvc --all-namespaces -o wide | grep pv-checkout-data

Manually removing the finalizer should be a last resort. It's there specifically to stop you from losing data you didn't mean to lose, and stripping it doesn't fix whatever the underlying detach problem actually is.

Define a small, reusable set of ClusterRoles for the actual permission tiers you need (viewer, deployer, namespace-admin), then bind them per team's namespace through RoleBindings tied to a group from your identity provider, never to individual users directly.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: payments-deployer
 namespace: payments
subjects:
 - kind: Group
  name: payments-team
  apiGroup: rbac.authorization.k8s.io
roleRef:
 kind: ClusterRole
 name: namespace-deployer
 apiGroup: rbac.authorization.k8s.io

Group-based bindings scale with your identity provider's group membership, not with headcount, which is the actual reason this approach survives 6 teams instead of collapsing into 240 hand-maintained bindings.

Exactly whatever that ServiceAccount's Role or ClusterRole grants, which is the whole reason scoping matters more than nearly anything else in this section. Containment: delete and regenerate the token if it's a legacy long-lived Secret-based token, or rely on the short TTL already limiting the damage window if it's a projected bound service account token.

Then audit everything that ServiceAccount actually touched through the audit log. The token being valid tells you nothing about what it was used for while it was live, that's a separate question you still have to answer.

Not all 40 equally. Sort by actual exploitability given your cluster's real exposure. An anonymous-auth flag on an API server that's only reachable from inside a private VPC is a lower real-world priority than a ClusterRoleBinding that grants cluster-admin to a default ServiceAccount every Pod in that namespace inherits automatically.

Red Hat's 2024 research puts misconfigurations like that second one near the top of what actually causes incidents in practice, not theoretical CIS line items with no realistic attack path behind them (Red Hat, 2024).

Check the join token first, kubeadm tokens expire after 24 hours by default. Then verify the discovery-token CA cert hash actually matches the running cluster's CA. Then check kubelet logs on the failing node for the real error, a version skew beyond the supported range is a common culprit: Kubernetes only supports the kubelet being up to 3 minor versions older than the control plane, never the reverse.

kubeadm token list
journalctl -u kubelet -f --no-pager
kubectl version --short

An etcd snapshot restore procedure, with the snapshots themselves stored off-region (see the point above about coincidences). PV data backed up separately through Velero or CSI snapshots, also off-region. A documented and genuinely tested DNS cutover if there's a secondary cluster standing by. An explicit ordering for bringing services back, databases before the apps that depend on them, not the reverse. And a way to communicate status to the rest of the company that doesn't depend on tools running inside the cluster that just went down.