MLOps interview questions that separate pipeline owners from textbook answers

Weak answers describe the happy path: ingest, transform, train, evaluate, deploy, done. Strong answers name a specific failure, a source table's schema changed upstream and nobody updated the feature code, a join key got duplicated after a vendor changed their export format, a training job silently used stale data for three weeks because a partition never landed. Interviewers aren't grading whether your pipeline is impressive. They're checking whether you've actually operated one long enough to have scar tissue.

Unit tests for transformation logic, same as any codebase. On top of that: data validation tests (schema and distribution checks against a fixture), model quality gate tests (does a newly trained model clear the minimum bar on a fixed eval set), serving smoke tests (does the deployed endpoint return a sane prediction shape for a known input), and a training-serving skew check (does the feature pipeline produce the same output in the training path and the serving path for identical raw input). Miss that last one and you'll find out about skew in production instead of in CI.

Level 0 is fully manual, a data scientist trains a model in a notebook and hands a file to someone else to deploy by hand. Level 1 automates the training pipeline itself, so retraining on new data doesn't require a human rerunning notebook cells. Level 2 goes further and automates CI/CD around the pipeline, so changes to the pipeline code itself get tested and deployed automatically, not just changes to the model (Google Cloud's MLOps maturity framework). Entry and mid-level roles mostly operate at Level 0 or 1. Level 2 tends to be a senior or staff-level responsibility, and naming that distinction unprompted is a decent signal to an interviewer that you know where you'd actually sit on a team.

Pin four things together: the exact code version (a commit hash, not "main"), the exact data version (a dataset snapshot ID, not "the latest table"), the exact environment (a locked dependency file or container image, not "whatever's installed"), and the random seed. Miss any one of the four and "reproduce this training run" becomes "approximately reproduce this training run," which is fine for exploration and not fine when someone asks you to explain, precisely, why a model behaved a certain way six months after it shipped.

Batch scoring, running predictions for a whole population on a schedule and storing the results, is simpler, cheaper, and easier to debug than real-time serving, and it's the right default whenever a decision doesn't need to reflect something that just happened. Real-time earns its cost when the input itself only exists at request time, a fraud model scoring a transaction as it happens, a recommendation reacting to a click three seconds ago, something batch scoring structurally can't do because the input didn't exist yet at the last batch run.

Serverless, a managed endpoint that scales to zero and back, is less operational overhead and cheaper for spiky or low-volume traffic, at the cost of cold-start latency when it scales up from zero and less control over the exact runtime. A Kubernetes deployment gives you full control over resource allocation, GPU scheduling, and warm-pool behavior, worth the extra operational cost once traffic is steady enough that scaling to zero isn't actually saving you money anyway.

Business metric first, the thing the model exists to move, conversion rate, fraud caught, click-through, because that's what everyone actually cares about and the one metric that can't be gamed by a model that looks technically fine on paper. Prediction distribution second, has the shape of what the model outputs shifted in a way that suggests something's off even before the business metric moves. Input data quality third, null rates, schema violations, distribution shift on the features themselves. Infrastructure last, latency, error rate, throughput, important, but if this is all you're watching you'll catch server problems and miss the model quietly getting worse at its actual job.

Data drift is a change in the input distribution, users from a new region start showing up, average order size shifts seasonally, while the underlying relationship between features and target stays the same. Concept drift is a change in that relationship itself, the same input now means something different, a fraud pattern that used to be reliable stops being predictive because fraudsters adapted. Data drift often just needs the model retrained on newer data. Concept drift can mean the model's whole feature set is no longer the right one, which is a much bigger conversation than a retrain.

Driving false positives to zero usually means setting the threshold so loose that real drift slips through too, precision and recall trade off against each other here same as anywhere else. A reasonable target is whatever rate the team can actually investigate without alert fatigue setting in, which in my experience is closer to one or two actionable alerts a week per model than one an hour. If your detector fires daily and gets dismissed daily, the threshold is wrong, not the concept of monitoring for drift itself.

A feature store is a system that computes, stores, and serves features consistently to both training and serving, so the same feature definition produces the same value whether it's being pulled for a batch training job or a live prediction request. The problem it solves is training-serving consistency at the source. Instead of trying to catch skew after the fact with tests, you remove the second implementation that could drift in the first place by having one place features get defined and computed.

Some features genuinely can be computed live cheaply, a request's own timestamp or device type. Most useful features aren't that simple, a 30-day rolling aggregate needs 30 days of history available fast, and recomputing that from raw event data on every single request is both far too slow for a latency budget measured in milliseconds and needlessly expensive at any real request volume. Precomputing it and storing the current value in a low-latency store is what makes a feature that took a batch job minutes to compute available for lookup in single-digit milliseconds at serving time.

Comparability across teams and searchability over time. If team A logs experiments in MLflow and team B logs them in a spreadsheet, nobody can answer "has anyone already tried this approach" across the org, which means duplicated work that a shared, searchable log would have caught immediately. The counterargument worth knowing too: forcing a tool nobody on a given team likes tends to produce experiments that get logged inconsistently or not at all, which defeats the point just as thoroughly as having no standard. I don't think there's a clean universal answer, only that the tradeoff is real and worth naming out loud if an interviewer pushes on it.

Quantization reduces the precision of a model's weights and activations, usually from 32-bit floating point down to 16-bit float, 8-bit integer, or in some cases 4-bit. The model gets smaller on disk, needs less memory bandwidth, and runs faster on both CPU and GPU because low-precision math is cheaper and more of the model fits in cache at once. On a GPU with Tensor Cores, INT8 inference can run two to four times faster than FP32 for the same architecture, which is usually the whole reason teams bother.

The cost is accuracy, and it isn't distributed evenly across the network. The first and last layers tend to be more sensitive to rounding error than the middle ones, so naive quantization can quietly wreck a model that looked fine on paper. That's why post-training quantization needs a calibration step, running a few hundred representative inputs through the model to pick the right scale and zero-point per tensor, and why quantization-aware training, which simulates the rounding during training itself, holds up better than quantizing after the fact. You also lose portability: an INT8 export tuned for one accelerator doesn't transfer cleanly to a different one, so the calibration and export step is usually specific to whatever hardware you're actually deploying on.

A model card is a short, structured document that ships alongside a trained model and answers the questions someone will ask six months from now, after everyone who built the thing has moved on to something else. At minimum it should cover what data the model was trained on and over what date range, what the intended use case is and just as importantly what it shouldn't be used for, what metrics it hit on which test set, and any known failure modes or subgroups where it underperforms.

The sections people skip are usually the useful ones: training data provenance, meaning where the labels actually came from and whether they were human-annotated or heuristic, evaluation broken out by the segments that matter rather than one overall number, and a plain description of what changed since the last version. A card that just says "AUC 0.91, trained Tuesday" isn't much better than no card at all. The real payoff shows up during an incident, when someone has ten minutes to decide whether a model is safe to keep serving traffic and the card is the fastest way to answer that without pulling in whoever built it.

An SLO is a target on a measurable signal, something like "99.5 percent of prediction requests return in under 200ms" or "precision on flagged transactions stays above 92 percent over a rolling 7-day window." It's different from just watching a dashboard because it gives you a threshold to compare against and a clear line for when to act versus when to leave things alone.

The error budget is what falls out of that target. A 99.5 percent SLO means you're allowed to spend 0.5 percent of requests on things like deploys, traffic spikes, or a shaky model version before you're actually in violation. That budget changes behavior in a concrete way. If a rocky rollout already burned most of the month's budget, that's the signal to slow down, extend canary time, or hold the next retrain, rather than push it out on schedule anyway. It turns "is this model healthy" from a gut feeling into a number, and it gives an engineer a defensible reason to say no to a risky deploy without it being a personal judgment call against someone else's timeline.

Schema checks (column types, required fields, allowed value ranges), distribution checks (has the mean, the null rate, or the cardinality of a categorical field moved outside a tolerance band), and freshness checks (is this partition actually today's data or a stale copy) all need to run as gates that can fail the pipeline, not just log a warning somewhere nobody reads. Google's TensorFlow Data Validation work formalized a lot of this thinking, comparing incoming data against a schema and a set of expected statistics computed from a trusted baseline, and blocking the pipeline when the comparison fails (Breck et al., "Data Validation for Machine Learning," 2019).

def validate_batch(df, schema):
  errors = []
  for col, rules in schema.items():
    if col not in df.columns:
      errors.append(f"missing column: {col}")
      continue
    null_rate = df[col].isna().mean()
    if null_rate > rules.get("max_null_rate", 0.02):
      errors.append(f"{col}: null rate {null_rate:.3f} exceeds threshold")
    if "allowed_range" in rules:
      lo, hi = rules["allowed_range"]
      if not df[col].between(lo, hi).all():
        errors.append(f"{col}: values outside [{lo}, {hi}]")
  if errors:
    raise ValueError("data validation failed:n" + "n".join(errors))

Two gates, not one. Offline evaluation against a fixed, held-out test set with a minimum-bar threshold the new model has to clear before it's even eligible to deploy, and a champion-challenger comparison against the model currently in production, not just against an arbitrary baseline. A model can clear an absolute bar and still be worse than what's live today. The failure mode candidates miss most: evaluating on a test set that's gone stale itself, so the "bar" it's clearing no longer reflects current production traffic.

Tie it to a latency requirement and a cost number, not a preference. If predictions can tolerate being a few hours or a day stale, batch is cheaper to build, cheaper to run, and easier to debug, you can rerun a batch job and diff the output against yesterday's. Streaming earns its complexity when a decision has to reflect something that happened in the last few seconds, fraud scoring on a transaction in flight, not fraud scoring on yesterday's transactions. I'd default to batch unless someone can name the specific decision that needs freshness measured in seconds, because streaming infrastructure carries ongoing operational cost that a lot of teams underestimate going in.

Idempotent steps first, so a retry after a partial failure doesn't double-process or corrupt state, this matters more than almost anything else on this list. Graceful degradation second: if today's features can't be computed, does the system serve yesterday's cached predictions, or does it serve nothing? Both are defensible, but you need to have picked one on purpose, in advance. Alerting last, and it needs to be actionable, "pipeline failed" at 2am that requires someone to open five dashboards before they know what to do is an alert that trains people to snooze it.

Git versions code well and data terribly. A training set can be gigabytes to terabytes, and committing it directly bloats a repo into something unusable within weeks. Tools like DVC or lakeFS version data the way git versions code, pointing a lightweight metadata file at immutable storage (S3, GCS) instead of committing the actual bytes. What you actually want reproducible together is code, data, and environment as one unit, a specific commit hash paired with a specific dataset version and a specific dependency lockfile, so "what exactly produced this model" has one unambiguous answer six months later instead of three people's competing guesses.

Nothing's wrong with it existing in two execution contexts, a batch training job and a low-latency serving path genuinely have different performance constraints. What's wrong is when the transformation logic itself gets written twice, once for the batch path and reimplemented separately for low-latency serving, because the two implementations drift apart the moment someone fixes a bug in only one of them. The fix is a shared feature definition, ideally one library both paths call, not two independently maintained copies of "the same" logic. This is the root cause behind most training-serving skew incidents, more on that in the deployment section below.

Shadow deployment runs the new model alongside the current one on real production traffic, but its predictions never reach a user, they just get logged and compared. Zero user risk, which makes it the right first step whenever you're not confident the new model is actually better, not just different. Canary routes a small percentage of real traffic to the new model and watches both business metrics and prediction-quality metrics before ramping up. Blue-green switches all traffic at once with an instant rollback path, useful when a slow ramp isn't worth the operational overhead and you trust your offline evaluation enough to go all-in.

A model registry tracks every trained model version with its metadata, training data version, evaluation metrics, who trained it, what stage it's in (staging, production, archived), and gives you an API to promote or roll back a version without anyone touching files by hand. A folder of pickle files works fine for one person on one project. It stops working the moment two people are training models for the same use case, because nothing tells you which file is actually live in production right now, what data trained it, or whether the file currently in the folder even matches the metrics someone reported in a spreadsheet three weeks ago.

Not automatically, and this is a good one to answer with a question of your own if an interviewer lets you: differ how? If the differences cluster in cases where the new model is plausibly more correct, better calibrated on a segment the old model handled poorly, that's the whole point of running the exercise. If the differences look random or correlate with something suspicious, a particular input range, a particular time of day, that's worth investigating before you'd trust a canary ramp. Raw disagreement rate alone doesn't tell you which situation you're in.

Guardrail metrics matter more, and the measurement window is usually longer. A product feature test can often show a clear signal in days. A model change can look neutral, or even slightly negative, on a short-term metric while a slower-moving downstream metric, retention, complaint rate, a fairness metric across a specific segment, moves in a way the short test never had time to catch. I'd want at least one guardrail metric defined before the test starts, not picked afterward to explain whatever happened, and I'd rather run it a week too long than call it a week too early.

An alert that fires constantly and gets ignored is worse than no alert at all, it trains people to tune out the channel entirely. Tie the alert to an actual outcome, did the business metric or a downstream label move, rather than any input distribution shift at all, because plenty of input drift is genuinely harmless, a marketing campaign shifted traffic mix for a week and nothing downstream cares. Add severity tiers so a small, likely-benign shift logs quietly while a large shift correlated with a metric drop pages someone. I'd rather under-alert for a month while tuning thresholds than keep a noisy alert running and lose the team's trust in monitoring altogether.

Check for an upstream change that individually looked harmless but compounded: a feature pipeline update that shifted a distribution just under your alert threshold, seasonality your training data didn't capture a full cycle of, or label distribution drift, the base rate of the thing you're predicting genuinely changed, that a static accuracy threshold doesn't catch. Slow, sub-threshold drift is the hardest kind to catch by design, since each individual check passes. A rolling comparison against a baseline from several months back, not just yesterday, catches this where a day-over-day check won't.

Because which features actually drive predictions can shift even when the model itself hasn't been retrained, if the input distribution changes enough that the model starts relying more heavily on a feature that used to matter less. A sudden shift in feature importance, without a corresponding model update, is often the earliest signal that something changed in the data before it ever shows up in an accuracy metric. I'd treat a sudden importance shift the same way I'd treat a drift alert, worth a look, not automatically an emergency.

At minimum: the business metric the model exists to move, a guardrail metric that shouldn't get worse even if the primary metric improves, and a prediction-distribution comparison against the model it's replacing. How long depends on the traffic pattern and the metric's natural variance, a weekday-only business metric needs to see at least one full week to avoid a false read from day-of-week effects. I'd rather watch a canary for an extra few days than ramp early and find out the metric only looked good because Tuesday happened to be an unusually easy day.

Offline storage (a data warehouse or lake) is optimized for large batch reads, computing features across millions of historical rows for training, and it's cheap at that scale. Online storage (usually a low-latency key-value store, Redis or a DynamoDB-style store) is optimized for a single-row lookup in milliseconds, exactly what a live prediction request needs. Neither is built to do the other's job well, batch-querying a low-latency key-value store for millions of rows is slow and expensive, and a data warehouse can't return a single row in the few milliseconds a serving path has to spare.

A spreadsheet works fine for the first ten runs. It stops working the moment three people are training variants of the same model and nobody remembers which hyperparameter combination produced which metrics, or whether the number in the spreadsheet came from a run that used yesterday's data version or last month's. Tools like MLflow or Weights and Biases log parameters, metrics, and artifacts automatically for every run, so "which config got us 0.91 F1 three weeks ago" is a query, not an archaeology project through someone's Slack history.

An experiment tracker's job is comparison, it logs every run, including the hundred failed ones, so you can compare metrics across configurations and pick a winner. A model registry's job is lifecycle management for the winners, tracking which specific model version is staged, which is in production, and giving you a controlled promotion and rollback path. Some tools do both under one roof now, but the responsibilities are genuinely different, and conflating them is how teams end up trying to "deploy" straight from an experiment log with no promotion gate in between.

Worth it whenever the job checkpoints frequently enough that losing an instance costs you minutes, not hours, of recomputed work, spot instances can run at a fraction of on-demand cost but come with no guarantee they won't get reclaimed mid-run. Not worth it for a job with no checkpointing that takes six hours to complete, where a reclaim at hour five means starting over from scratch. The actual engineering work here is making the training loop resilient to interruption, checkpoint every N steps, resume cleanly from the last one, not just hoping the instance survives.

Break down spend by team and job type first, cost tools that only report a total cluster bill are close to useless for this. Usual suspects, roughly in order of how often I've actually seen each one: a hyperparameter sweep that fans out into far more parallel runs than anyone intended, GPU instances left running after a job finished because cleanup wasn't automated, and an oversized instance type picked once for a memory-hungry job and left as the team's copy-paste default for everything since. Tag every job by team and purpose before this happens, not after, an untagged multi-team GPU bill is close to impossible to attribute once it's already landed.

Data parallelism means every worker holds a full copy of the model and processes a different shard of the batch, then gradients get synchronized across workers, usually with an all-reduce, before the weights update. It's the default choice, it's simple to reason about, and frameworks like PyTorch's DistributedDataParallel handle most of the wiring for you. The limit is that the model has to fit in a single GPU's memory, since every worker needs the whole thing.

Model parallelism splits the model itself across devices, either by layer, pipeline parallelism, where GPU 1 holds the first several layers and GPU 2 holds the next set, or by splitting individual layers across devices, tensor parallelism, common for the large matrix multiplies inside transformer attention and feed-forward blocks. You reach for this once the model doesn't fit on one GPU, which is the normal situation past a certain parameter count. The tradeoff is communication overhead: pipeline parallelism creates idle bubbles where later stages wait on earlier ones unless you overlap microbatches carefully, and tensor parallelism needs fast interconnects like NVLink rather than plain PCIe, because it's synchronizing on every forward and backward pass, not once per step. Large training runs usually combine all three, data parallelism across node groups, pipeline parallelism across GPUs within a group, tensor parallelism within a pod, because none of them alone scales cleanly past a certain size.

A lot of the fundamentals carry over. You still need versioning, monitoring, and a rollback path, but the unit you're versioning and the failure modes both change. Instead of retraining on a schedule, you're mostly changing prompts, few-shot examples, retrieval configuration, or which base model you're calling, and each of those needs the same discipline as a model version bump: tracked in source control, tested against a fixed eval set before shipping, rolled back the same way if a system prompt change tanks quality on some slice of input.

Evaluation is the part that's genuinely harder. A traditional model has one number on a held-out set, accuracy or AUC. An LLM feature usually needs a rubric-based or model-as-judge eval, because "is this summary good" doesn't reduce to a single ground-truth label, and those judge-based evals need their own calibration against human review or they start drifting on their own. Cost and latency also work differently, you're paying per token and watching context length, so a lot of engineering effort goes into trimming prompt size and picking the cheapest model that still clears the quality bar rather than tuning GPU utilization for one fixed model. And if there's a retrieval step, you now have a second system that can go stale independently of the model weights, the embedding index can be perfectly healthy while the documents it was built from are three months out of date.

Active learning is a loop where the model picks which unlabeled examples get sent to a human for labeling, instead of labeling a random sample or labeling everything that comes in. The usual selection strategy is uncertainty sampling, send the examples where the model's confidence is lowest or where an ensemble disagrees most, since those are the ones most likely to teach it something it doesn't already know. Some setups use a proxy like expected model change instead, but uncertainty sampling shows up most often in production because it's cheap to compute from a model you're already running.

It earns its keep when labeling is genuinely expensive, medical imaging, legal document review, anything that needs a domain expert rather than a crowdworker, because the win is fewer labels needed to hit the same accuracy. It's usually not worth the engineering effort when labels are cheap and plentiful, you're better off just labeling more random data. It can also backfire: if the model is systematically bad at a whole subgroup for a reason that has nothing to do with sample size, a sensor reporting garbage values for one device type, say, active learning keeps sending you examples from that subgroup over and over, and you burn your labeling budget on noise instead of on genuinely hard cases. Most setups keep a floor of random sampling mixed alongside the uncertainty-driven picks specifically to catch that failure mode and keep the training distribution from drifting away from what production actually looks like.

A data contract is an explicit, versioned agreement about the shape and semantics of a dataset or event stream, schema, types, allowed null rate, expected value ranges, update cadence, signed off by the team producing the data and the teams consuming it. The document itself isn't the point. The point is that it becomes something enforceable with a schema validator, Avro, Protobuf, or a JSON schema check, sitting between the producer's write path and the consumer's read path, so a breaking change gets caught at write time instead of surfacing three weeks later as an unexplained drop in model accuracy.

The part people underestimate is that a contract needs a deprecation process, not just a schema. If the upstream team wants to rename a field or change currency from cents to dollars, the contract should force a dual-write or dual-read period with a fixed deadline, rather than one cutover that breaks every downstream consumer at once. Without that, an upstream engineer ships a schema change on a Friday, nothing breaks in their own tests since nothing type-checks against a consumer they don't own, and the ML team finds out on Monday when a feature that used to be a float is suddenly a string. Contracts also need a named owner and a notice period written into the agreement itself, because a contract nobody's on the hook for renegotiating is just documentation nobody reads.

The baseline is that nothing sensitive goes into source control or a notebook cell, full stop, and that includes a credential committed "temporarily" and removed later, because it's still sitting in git history and needs an actual rotation, not just a revert. The standard pattern is a secrets manager, Vault, AWS Secrets Manager, or a cloud KMS-backed store, that the pipeline authenticates to at runtime using a short-lived identity, an IAM role attached to the training job's service account, rather than a static key baked into an environment variable or config file.

The part that actually trips teams up is interactive notebooks, because the whole point of a notebook is fast iteration, and someone pastes a key in to get unblocked and forgets to strip it before the notebook gets checked into a shared repo. The practical fix is making the secure path the easy path: give data scientists a helper library that pulls credentials from the secrets manager and injects them into the environment at kernel startup, so there's never a reason to type a key by hand. On the infra side you want secrets scanning in CI as a backstop, gitleaks or truffleHog catching anything that slips through, short expiry windows on any credential that could end up in a log by accident, and audit logging on the secrets manager itself, so if something does leak, you can see exactly which jobs pulled that credential and in what window, which matters a lot when you're figuring out how far the blast radius goes.

Airflow is a general-purpose DAG scheduler that happens to run ML pipelines fine but wasn't built for ML specifically, no native model versioning or experiment tracking baked in. Kubeflow Pipelines is Kubernetes-native and ML-aware, containerized steps, native integration with model registries and metadata tracking, at the cost of a steeper operational learning curve if your team doesn't already run Kubernetes comfortably. Prefect and Dagster are newer, developer-experience-focused options with better local testing and clearer data lineage out of the box than classic Airflow, though the ML-specific tooling around them is thinner than Kubeflow's.

I don't think there's a universally correct answer here, and I'd be a little suspicious of a candidate who claims one. If your team already lives on Kubernetes, Kubeflow's integration story is hard to beat. If you're a five-person data science team without platform engineers, Airflow or Prefect will get you shipping faster this quarter.

Immutable artifacts. Once a model version is registered, it never changes, a "fix" is a new version, not an edit to an existing one. Pin the model artifact together with the exact feature definitions and preprocessing logic that were live at training time, because rolling back the model while the feature pipeline has since changed just trades one incompatibility for another. A rollback that's boring is one where "go back to version 47" is a single command that restores model, feature logic, and preprocessing together as one unit, not three separate systems you have to sync by hand under pressure.

Almost always feature logic differences, the training pipeline computed a feature one way (say, a 30-day rolling average computed in a batch job against clean historical data) and the serving path computed "the same" feature slightly differently (a live average with an edge case around missing days the batch job never hit). Prevention is a shared feature definition both paths call, ideally the same code, not a reimplementation, backed by a feature store that guarantees the online and offline values were computed from the same logic. Testing for skew directly, running both paths against identical raw input and diffing the output, catches this in CI instead of three weeks into production when someone finally notices accuracy quietly dropped.

This one trips people who've only ever rolled back stateless software. If a recommendation model already influenced what got cached, what got pre-fetched, or what a downstream system logged as "shown to the user," rolling the model back doesn't undo those side effects, it just changes what happens next. Depending on the system, you may need a separate remediation step, invalidating a cache keyed on the bad model's outputs, on top of the model rollback itself. I don't have a universal answer here, it depends entirely on what the model's outputs actually touched downstream, which is exactly what the interviewer is checking whether you've thought about.

Proxy monitoring while you wait: track the input feature distribution and the prediction distribution in near real time, since both are available immediately, and treat a shift in either as an early warning even without label confirmation yet. When labels do arrive, backfill the actual accuracy metric and reconcile it against what the proxy signals predicted, this tells you over time how reliable your proxy actually is as an early-warning system for this specific model. A model with a two-week label lag that only checks accuracy once labels land is flying two weeks blind on anything that changed in between.

import numpy as np

def population_stability_index(expected, actual, bins=10):
  breakpoints = np.percentile(expected, np.linspace(0, 100, bins + 1))
  breakpoints[-1] += 1e-6 # include max value in last bin
  e_counts, _ = np.histogram(expected, bins=breakpoints)
  a_counts, _ = np.histogram(actual, bins=breakpoints)
  e_pct = np.clip(e_counts / len(expected), 1e-6, None)
  a_pct = np.clip(a_counts / len(actual), 1e-6, None)
  return float(np.sum((a_pct - e_pct) * np.log(a_pct / e_pct)))

# PSI < 0.1: no significant shift. 0.1-0.25: moderate. > 0.25: investigate.

First, scope the damage: which predictions during those nine days used the degraded feature, and does the model's behavior when that feature is null look meaningfully different from normal (some models silently fall back to a default value that quietly changes the prediction without erroring at all). Communicate what happened to whoever consumes the model's output before they find it themselves. Then the actual fix: a null-rate check on every feature, per pipeline run, with a threshold that fails the run rather than logging a warning, exactly the kind of gate that should have caught this on day one instead of day nine.

Nine days is a long time for a null-rate spike to go unnoticed, and honestly, it usually means nobody was actually watching that specific metric at all, not that an alert fired quietly and got ignored.

Label leakage, and it's a subtle kind that doesn't show up as an obvious bug. If a feature for a training example gets computed using data that would only have been available after the label was actually observed, "average purchase amount over the last 30 days" pulled using data through today instead of data through the moment the label happened, the model trains on information it would never have access to at real prediction time. It looks like a great model in offline evaluation and quietly falls apart in production, because production can never provide that future information the training set accidentally leaked in.

# WRONG: joins on customer_id only, pulls the feature as of "now"
features = feature_table[feature_table.customer_id == cid]

# RIGHT: point-in-time join, feature must be computed as of the label's timestamp
features = feature_table[
  (feature_table.customer_id == cid) &
  (feature_table.feature_timestamp <= label_timestamp)
].sort_values("feature_timestamp").tail(1)

Fixing it today probably means killing or throttling the offending job and having an uncomfortable conversation. Preventing it going forward means the platform enforces hard per-team quotas instead of relying on people to self-regulate, plus visibility, a dashboard everyone can see showing current usage by team, so the next runaway job gets caught by a teammate noticing, not by another team complaining a week later. I'd also add an automatic alert when any single job's runtime or resource usage crosses a few standard deviations from that job type's historical norm, which catches a runaway sweep before it's had a week to do damage.

The fact that p50 is flat and only the tail moved is the first clue. This isn't the model getting slower on average, it's a subset of requests hitting something expensive that the average hides completely. The first thing I'd check is whether the spike correlates with scale-up events specifically, because a classic cause is cold starts: a new pod comes up, the container starts, but the model weights still have to load from disk or object storage into GPU memory, the CUDA context has to initialize, and if the framework does any JIT compilation or cudnn autotuning, the first several inferences on that pod are dramatically slower while it warms up. If autoscaling is reactive on CPU or request count, new pods can come up mid-load-spike and start receiving full traffic before they're actually warm, and every request routed to a cold pod becomes a tail-latency outlier.

Average GPU utilization looking fine doesn't rule this out, it can even point toward it. A pod stuck loading weights or compiling a graph shows low compute utilization, not high, so if the fleet is mostly steady-state with a couple of pods mid-warmup, the aggregate number stays unremarkable while a handful of requests take seconds instead of milliseconds. I'd pull per-pod latency instead of fleet-averaged and check whether the p99 offenders cluster on pod age rather than being spread evenly across the fleet.

If that's not it, the other usual suspects are request batching and queueing, a burst during a scale-up window can back up requests waiting for a batch to fill, and that queueing time lands entirely in the tail without ever touching the average, or noisy-neighbor GPU memory pressure if new pods land on a node that's already tight and the model falls back to a slower memory path. The general fix for autoscaled GPU inference regardless of which cause it turns out to be: keep a warm pool of pods pre-loaded and idling below the scale threshold so new capacity is already warm before traffic hits it, scale on a leading indicator like queue depth rather than a lagging one like CPU, and gate new pods out of the load balancer until they've served a few warmup requests successfully.

Model-side: quantization (INT8 instead of FP32 cuts memory bandwidth and often latency, at some accuracy cost worth measuring, not assuming), distillation into a smaller model, and batching requests where the use case tolerates a few milliseconds of queuing to fill a batch. System-side: caching predictions for repeat inputs, and colocating feature computation with the model server so a network round-trip to a separate feature store isn't sitting on the critical path of every request. Most latency problems I've seen traced back to the feature lookup, not the model's forward pass.

This is a senior-level question and interviewers expect you to name what you're deliberately not building yet. Start with resource isolation, one team's runaway hyperparameter sweep shouldn't starve another team's production retraining job, which usually means namespace-level quotas and a scheduler that understands priority, not just first-come-first-served. Layer in guardrails: a shared data validation library every pipeline has to call, a required model registry step before anything reaches serving, and templated pipeline scaffolding so teams aren't each reinventing retry logic from scratch.

I'd start with a thin templating layer over an existing orchestrator, Kubeflow Pipelines or Airflow, not a custom scheduler built in-house, and add the guardrails as required steps baked into the template, not as optional documentation nobody reads.

Route on a request attribute, tenant ID, model version header, or a feature-flag-style config, and keep the routing logic outside the model-serving code itself so adding a model doesn't require redeploying the router. Cold models, ones not resident in memory, add real latency on first request, so a warm pool or predictive pre-loading based on traffic patterns matters once you're past a handful of models. Watch resource contention specifically: one model getting a traffic spike shouldn't degrade latency for every other model sharing the same compute pool, which usually means per-model resource limits, not a shared free-for-all.

Start with quotas per team, not a shared free-for-all, so one team's overnight hyperparameter sweep can't starve another team's scheduled retraining job. A priority queue with aging, jobs waiting long enough gradually gain priority, prevents a low-priority job from waiting forever behind a constant stream of higher-priority ones. Preemption policy matters too: can a lower-priority job get checkpointed and paused to free up GPUs for something urgent, or does it just get killed and restarted from scratch? Checkpoint-and-resume is more complex to build but far less wasteful of compute than kill-and-restart once jobs run for hours.