Data Science Interview Questions (2026): Stats, SQL, and Product Sense

Mean is the arithmetic average, sensitive to outliers, a single $2 million salary in a room of ten $60k earners drags the mean somewhere nobody in the room actually earns. Median is the middle value once sorted, far less swayed by outliers, better for skewed distributions like income or session length. Mode is the most frequent value, useful for categorical data where mean and median don't even apply. The mislead case that actually comes up: reporting mean revenue per user on a freemium product where 95 percent of users pay $0 and a handful of whales pay $400 a month. The mean looks healthy. The median is zero, and it's the more honest number to lead with.

It's the probability of seeing a result at least as extreme as what you observed, assuming the null hypothesis (usually "no real difference") is true. A p-value of 0.03 doesn't mean there's a 97 percent chance your change works, and it doesn't mean there's a 3 percent chance the null hypothesis is true. It means: if nothing had actually changed, you'd see a result this extreme or more extreme about 3 percent of the time by chance alone. That distinction trips up more candidates than any other single stats concept on this page.

The sampling distribution of the mean approaches a normal distribution as sample size grows, regardless of the shape of the underlying population distribution. It matters for A/B testing because most significance tests assume normally distributed sample means, and CLT is the reason that assumption holds even when the underlying metric itself, session length, say, which is heavily right-skewed, isn't normal at all, as long as your sample size is reasonably large.

Alpha is the false positive rate you're willing to tolerate, usually set at 0.05 before the test runs. Power is 1 minus beta, the probability you correctly detect a real effect if one exists, usually targeted at 80 percent. Alpha protects you from crying wolf. Power protects you from missing a real signal. Underpowered tests are the more common failure in practice, teams run a test for the standard two weeks without checking whether that sample size can even detect the effect size they care about, then wrongly conclude "no effect" when the honest answer is "inconclusive."

Discrete distributions describe outcomes you can count, a finite or countably infinite set of possible values. Binomial (number of conversions out of N visitors, each independently converting or not) is the one that shows up constantly in A/B testing. Continuous distributions describe outcomes measured on a scale, any value in a range is possible. Normal distribution, approximated by session length or revenue per user after a log transform, is the default assumption underlying most significance tests, which is exactly why checking that assumption before trusting a p-value matters.

Define the hypothesis and the primary metric before writing any code. Calculate the required sample size given your baseline conversion rate, minimum detectable effect, alpha, and power. Randomly assign users to control and treatment, ideally with a randomization method that's actually random and consistently sticky per user across sessions. Run the test for the pre-calculated duration, resist the urge to peek and stop early. Analyze with the pre-registered statistical test. Ship, iterate, or kill based on the result, and document the decision either way so the next person doesn't quietly re-run the same test in eight months.

Two-tailed, as the default, almost always. A one-tailed test only checks for an effect in one direction (the new design is better) and is blind to the change being worse, a real possibility you generally want to catch, not assume away. One-tailed tests are easier to reach significance with at the same alpha, which is exactly why reviewers get suspicious when someone reaches for one after the fact. Use a one-tailed test only when there's a strong theoretical reason the effect literally cannot go the other direction, decided before running the test, not after seeing which direction looks more favorable.

The only real gotcha is the COUNT(DISTINCT user_id). Forgetting DISTINCT counts events, not users, and inflates the number for anyone who fires more than one event a day, which is nearly everyone.

SELECT
 DATE(event_timestamp) AS activity_date,
 COUNT(DISTINCT user_id) AS dau
FROM events
GROUP BY DATE(event_timestamp)
ORDER BY activity_date;

First-touch credits the very first marketing channel a user interacted with before converting. Last-touch credits whichever channel touched them immediately before the conversion event, often overweighting channels like branded search or retargeting that catch people late in their decision, after an earlier channel actually did the harder work of creating awareness. Computing either is a MIN or MAX over touch timestamps per user, joined back to the channel of that specific touch: MIN(touch_timestamp) for first-touch, MAX(touch_timestamp) for last-touch, both filtered to before the conversion event and grouped by user.

When the rule is genuinely simple and stable, "flag any transaction over $10,000 for manual review" doesn't need a model, a threshold does the job with zero training data and zero drift risk. Also when there isn't enough labeled data yet to train anything reliable, a heuristic built on domain knowledge can ship this week and start generating the labels a future model will actually need. The honest interview answer isn't "always use a model," it's recognizing that a model adds real cost, a training pipeline, monitoring, a retraining cadence, and that cost needs to buy something a simpler rule genuinely can't.

The simplest thing that could possibly work: predict the mean for regression, predict the majority class for classification, or a single-feature logistic regression, trained and evaluated before touching a gradient-boosted anything. It sets the floor. If a fancier model only beats the baseline by half a percent after three weeks of feature engineering, that's a signal the extra complexity isn't earning its keep, not a reason to keep tuning harder. Skipping the baseline is how teams end up unable to answer "is this model actually good, or just better than nothing," a question stakeholders ask more often than "what's your AUC."

Start by asking what action actually signals value delivered, not just any logged event. Opening the app isn't the same as an action that indicates the product did its job that day. For a messaging app, sending or reading a message is a stronger definition than "app opened." For a note-taking app, creating or editing a note beats opening the app to briefly glance at it. The honest answer to give an interviewer who pushes further: "I'd want to look at retention curves cut by a few candidate definitions before committing to one," because the "right" definition is an empirical question, not something you can reason your way to from the product description alone.

Ask what decision each analysis actually feeds. An analysis nobody will act on regardless of the result isn't worth doing first, no matter how interesting the question is. Of the ones tied to a real decision, prioritize by which decision is closest to being made, an analysis for a roadmap review happening in three days outranks one for a strategy discussion that's still six weeks out and could easily slip further. And ask, bluntly, whether any of the five can be answered in an hour with data that already exists in a dashboard, versus which genuinely need a week of new query work, since fast and good enough sometimes beats thorough and late when the decision has a real deadline attached.

Precision is TP over TP plus FP, the fraction of your positive predictions that are actually correct. Recall is TP over TP plus FN, the fraction of the real positives your model actually catches. Moving the classification threshold trades one for the other: raise the threshold and precision goes up because you're only calling something positive when you're confident, but recall drops because you miss more of the real positives sitting just below that bar.

Which one you optimize for comes down to which mistake costs more. In fraud detection, missing real fraud (a false negative) is usually far more expensive than flagging a legitimate transaction for review, so you accept lower precision for higher recall and let a human layer absorb the false alarms. In a personal spam filter, a false positive that buries a real email is the mistake users actually complain about, so you tune toward precision even if some spam gets through. When there's no clean argument either way, F1 or a full precision-recall curve gives a more honest picture than committing to a single number.

Bias is the error you get from a model that's too simple to represent the real relationship, it makes the same kind of mistake no matter what training data you give it. Variance is the error from a model that's too sensitive to its particular training sample, it fits noise as if it were signal, so a slightly different training set produces a meaningfully different model. Expected error roughly decomposes into bias squared plus variance plus irreducible noise, and the two terms usually pull in opposite directions as you change model complexity.

A linear regression on a genuinely nonlinear relationship has high bias, no amount of extra data fixes it because the model form itself can't represent the pattern. An unpruned decision tree trained on a small dataset has high variance, it memorizes quirks specific to that sample and generalizes poorly. Regularization and ensembling reduce variance at the cost of a small amount of bias, while just adding more training data reduces variance without touching bias at all.

Not automatically, and saying "yes because it's below 0.05" is the answer that costs you the round. Check three things first. Was 0.05 the pre-registered threshold, or did you pick it after seeing the number? Is the effect size actually meaningful, a p-value of 0.04 on a 0.1 percent lift might be statistically real and business-irrelevant. And did any guardrail metric move in the wrong direction? A significant primary metric next to a tanking guardrail metric is a real trade-off decision, not a green light.

Type I error (false positive): you conclude a new checkout flow increased conversion when it actually didn't, and you ship it, permanently complicating the codebase for a lift that was noise. Type II error (false negative): the new flow genuinely increased conversion by 2 percent, your test didn't have enough power to detect it, you conclude "no effect" and kill a change that would have made money. Alpha (usually 0.05) controls how often you make the first mistake. Power (usually targeted at 80 percent) controls how often you avoid the second one, and it's the piece most candidates forget to mention unprompted.

Correlation means two variables move together. Causation means one variable changing directly produces a change in the other. The classic textbook burn: ice cream sales and drowning deaths correlate, both rise in summer, neither causes the other, heat is the confounder. The version that actually shows up on a data team: power users use a feature more and also retain better. Concluding the feature causes retention and pouring engineering time into pushing everyone toward it, when the real driver is that engaged users adopt more features generally, is an expensive mistake teams make constantly.

Common detection methods: values beyond 1.5 times the interquartile range (IQR) from Q1 or Q3, values beyond 2 or 3 standard deviations from the mean (assumes rough normality), or isolation forests and local outlier factor for multivariate cases where a point looks normal on every individual feature but weird in combination. Don't remove outliers automatically. A $40,000 transaction in a dataset averaging $50 orders might be fraud, worth flagging, or it might be your single biggest enterprise customer, worth protecting in every future model you build. Removing it because it hurts your R-squared is optimizing the wrong thing.

A 95 percent confidence interval means: if you repeated this exact sampling process many times, 95 percent of the intervals constructed this way would contain the true parameter. It says nothing about the probability that this specific interval contains the true value, which is a common misinterpretation. A credible interval, the Bayesian equivalent, directly states: given the data and your prior, there's a 95 percent probability the true parameter falls in this range. The credible interval answers the question most people actually want answered when they ask about a confidence interval, part of why Bayesian methods have gained ground in industry experimentation.

Users who see a push notification and users who make a purchase that day might look independent of each other's underlying probability, until you notice both are driven by a third variable: whether the user opened the app that day at all. App-open correlates with both getting a notification interaction logged and purchasing, so notification exposure and purchase look statistically dependent even if the notification itself did nothing. The catch is conditioning: check whether the apparent dependence survives once you control for the shared driver. If it vanishes, you found a confounder, not a real relationship.

The entity you randomly assign to control or treatment, usually the user, but sometimes the session, the device, or a geographic region. Pick the wrong unit and you get contamination: randomize by session on a feature the same logged-in user encounters across multiple sessions, and that user might see both control and treatment, which pollutes their behavior in both arms and makes your effect estimate meaningless. Rule of thumb: randomize at the level where you expect spillover. If a user could plausibly experience the feature more than once, randomize by user, not by session or page view.

You need four inputs: your baseline metric value (say, a 5 percent conversion rate), the minimum detectable effect you actually care about (a lift smaller than this isn't worth acting on even if real), alpha (usually 0.05), and desired power (usually 80 percent). Plug those into a standard sample size calculator for a two-proportion z-test, and it tells you how many users per arm you need. Skip this step and you get one of two failure modes: an underpowered test that runs the standard two weeks and "finds nothing" when a real effect existed but the sample was too small to detect it, or an overpowered test that runs so long it detects a statistically significant but practically meaningless 0.02 percent lift.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect_size = proportion_effectsize(0.05, 0.055) # baseline 5%, target 5.5%
analysis = NormalIndPower()
n_per_arm = analysis.solve_power(effect_size, alpha=0.05, power=0.8, alternative="two-sided")
print(n_per_arm)

The obvious version: running dozens of tests and cherry-picking the one that came back significant. The subtler, far more common version: peeking at results daily and stopping the test the moment it crosses significance, instead of running it for the pre-calculated duration. Every peek is effectively another chance for a false positive to appear, even with no ill intent at all, checking the dashboard every morning out of genuine curiosity has the same statistical effect as deliberately hunting for significance. Sequential testing methods that adjust the significance threshold for repeated looks exist specifically to fix this, but most teams just don't use them, and stop early anyway.

Novelty effect: users engage more with a change simply because it's new and different, not because it's actually better, and the lift decays as the novelty wears off. Primacy effect, roughly the opposite pattern: existing users initially resist a change out of habit, showing a temporary dip that recovers as they adjust. Both show up as the same red flag, an effect size that trends over the course of the test instead of staying roughly flat. The fix is to plot the daily effect size, not just the final aggregate number, and to segment new users against existing users separately, since the two effects often point in opposite directions within the same test.

A metric you're not trying to improve but committing to not hurt, page load time, unsubscribe rate, support ticket volume, app crash rate. A checkout redesign might lift conversion 4 percent, great, ship it, while quietly doubling the support ticket rate because the new flow confused a meaningful minority of users. Not great, don't ship it without a fix. Primary metrics tell you if something worked. Guardrail metrics tell you what it cost. Teams that only track the primary metric routinely ship changes that win on paper and generate a customer-facing mess nobody caught until the complaints rolled in.

Frequentist testing gives you a p-value and a binary reject-or-fail-to-reject decision at a fixed threshold. Bayesian testing gives you a full posterior distribution and lets you directly state something like "there's an 87 percent probability the treatment beats control by at least 1 percent," which maps much more naturally onto the actual business question of whether to ship. Bayesian methods also handle continuous monitoring more gracefully, since the interpretation doesn't degrade the same way frequentist p-values do under repeated peeking. The trade-off: you need a prior, and a badly chosen one can bias results toward whatever you already believed, exactly the kind of thing a good interviewer will push on if you claim Bayesian methods are strictly better.

Join signups to activity on user_id, then bucket each activity row by how many days after signup it happened.

WITH signups AS (
 SELECT user_id, DATE(created_at) AS signup_date
 FROM users
),
activity AS (
 SELECT user_id, DATE(event_timestamp) AS activity_date
 FROM events
 GROUP BY user_id, DATE(event_timestamp)
)
SELECT
 s.signup_date,
 COUNT(DISTINCT s.user_id) AS cohort_size,
 COUNT(DISTINCT CASE WHEN a.activity_date = s.signup_date + 1 THEN s.user_id END) AS day1_retained,
 COUNT(DISTINCT CASE WHEN a.activity_date = s.signup_date + 7 THEN s.user_id END) AS day7_retained
FROM signups s
LEFT JOIN activity a ON a.user_id = s.user_id
GROUP BY s.signup_date
ORDER BY s.signup_date;

Divide day1_retained and day7_retained by cohort_size in the application layer or a wrapping query, and watch the trailing edge: a cohort that signed up 3 days ago doesn't have Day-7 data yet, so it needs to be excluded from the Day-7 average, not counted as zero.

SELECT
 COUNT(DISTINCT CASE WHEN event_name = 'view' THEN user_id END) AS viewed,
 COUNT(DISTINCT CASE WHEN event_name = 'add_to_cart' THEN user_id END) AS added_to_cart,
 COUNT(DISTINCT CASE WHEN event_name = 'checkout' THEN user_id END) AS checked_out,
 COUNT(DISTINCT CASE WHEN event_name = 'purchase' THEN user_id END) AS purchased
FROM events
WHERE event_timestamp >= CURRENT_DATE - INTERVAL '7 days';

That version is fine for a rough funnel, but it silently counts anyone who purchased even if they skipped add-to-cart entirely, a direct-buy button, say. If the interviewer pushes on "strict" funnel logic, where each step requires the previous step from the same user, you need a self-join or window function checking that each later event's timestamp comes after an earlier step's timestamp for that same user, not just that both events exist somewhere in the table.

AVG() in standard SQL silently ignores NULL rows rather than treating them as zero, which sounds helpful until you realize those NULLs might represent sessions that never closed properly, a crashed app, a lost connection, rather than missing data at random. Silently dropping them can bias the average, since sessions that crash are disproportionately either the long, engaged ones hitting an edge case or the short ones that bounced before the close event fired, and you genuinely don't know which without checking. The fix isn't a SQL trick, it's deciding explicitly: COALESCE(session_length, 0) if a NULL means the session effectively didn't happen, or filter NULLs out and report what fraction of rows got excluded, so whoever reads the number knows it isn't the whole picture.

Conditional aggregation does this without a dedicated PIVOT clause, which not every SQL dialect supports anyway. For each feature, wrap the aggregation in a CASE WHEN that only counts the relevant event type, then group by user.

SELECT
 user_id,
 COUNT(CASE WHEN event_name = 'login' THEN 1 END) AS login_count,
 COUNT(CASE WHEN event_name = 'purchase' THEN 1 END) AS purchase_count,
 MAX(CASE WHEN event_name = 'purchase' THEN event_timestamp END) AS last_purchase_at,
 SUM(CASE WHEN event_name = 'purchase' THEN amount ELSE 0 END) AS total_spend
FROM events
GROUP BY user_id;

This is also the point where a data scientist and a data engineer start talking about a feature store instead of re-running this query fresh for every training run, but that's usually a separate conversation from the SQL round itself.

Order each user's events by timestamp, use LAG to pull the two events before whatever you're defining as the churn event, then concatenate the three event names into a single path string and group by that path to count frequency.

SELECT
 event_2_back || ' -> ' || event_1_back || ' -> ' || last_event AS path,
 COUNT(*) AS occurrences
FROM (
 SELECT
  user_id,
  LAG(event_name, 2) OVER (PARTITION BY user_id ORDER BY event_timestamp) AS event_2_back,
  LAG(event_name, 1) OVER (PARTITION BY user_id ORDER BY event_timestamp) AS event_1_back,
  event_name AS last_event
 FROM events
) t
WHERE event_2_back IS NOT NULL
GROUP BY path
ORDER BY occurrences DESC;

The genuinely hard part isn't the SQL, it's defining "churn" as an event in the first place. Most products don't have a clean cancellation click, churn is usually an absence of activity past some threshold, which means this query needs a churn-labeling step upstream before the sequence analysis even starts.

Skip the coefficient entirely and lead with the decision it changes. Instead of "the coefficient on tenure is 0.4," say "customers who've been with us longer than 18 months are about twice as likely to renew, holding everything else equal, which is why the model flags newer accounts for the retention team's outreach." Tools like SHAP values translate well here too, not as a technical artifact to show, but as the underlying reason you can point to a specific customer and say which three factors pushed their score up or down, in plain terms, instead of "the model said so."

Whether the model's output actually reaches the point where a human or a system acts on it. A shockingly common failure is a model with 92 percent accuracy sitting in a dashboard nobody checks, changing nothing downstream. Second, whether the offline evaluation distribution matches production traffic, a model validated on last year's data can look great offline while quietly underperforming on a shifted user base this year. Third, whether the metric being tracked is even the right one. A churn model can correctly flag at-risk customers while the retention team's outreach script does nothing to actually retain them, a real gap, but not the model's fault.

Find the closest historical analog, even an imperfect one, a similar feature shipped before, or a competitor's version with public usage estimates, and use it to anchor a range rather than a single number. Break the estimate into a funnel: how many users will discover the feature, times what fraction will try it, times what fraction of those convert to the revenue event, times the average value per conversion. Each step carries real uncertainty, so present a range with the assumptions stated explicitly rather than one confident number, and flag which assumption the whole estimate is most sensitive to. That's usually the one worth validating with a small test before committing engineering time to the full build.

Whether "neutral" means the confidence interval is tight around zero, genuinely no effect, or wide and centered near zero, underpowered, could plausibly be positive or negative, you just can't tell from this sample. Those look identical on a summary slide and mean completely different things. If it's genuinely neutral, ask what it costs to maintain, engineering complexity, support burden, before shipping something with no measured upside. If underpowered, ask whether it's worth running longer or with more traffic before deciding anything, rather than shipping or killing based on a test that couldn't have detected the effect either way.

A model that predicts the majority class for every single row hits 98 percent accuracy while catching zero of the positives you actually care about, which is exactly the failure mode you're trying to avoid. Accuracy on an imbalanced dataset doesn't distinguish a useful model from a model that's done nothing at all, so reporting it alone is close to meaningless.

I'd report precision, recall, and F1 on the positive class, and look at a precision-recall curve rather than ROC, since ROC's false positive rate is diluted by the huge number of true negatives and looks deceptively good even on a weak model. If the positive class is rare enough that PR curves get noisy, I'd anchor to an operating point that matches how the model will actually be used, something like "at 80 percent recall, what's our precision," since that maps more directly to a real decision than an abstract summary metric. Resampling or class weighting during training can help the model learn the minority class better, but they don't change what you should be judging it on at evaluation time.

Standard k-fold shuffles rows randomly into folds, which for time series means your training fold can end up containing data from after the point you're predicting in the validation fold. That's leakage from the future, and it makes offline validation numbers look better than what you'll actually get once the model is running on real, unseen future data.

The fix is a rolling or expanding window split: train on everything up to time t, validate on t through t+k, then slide the window forward and repeat, never letting a training fold contain anything later than its validation fold.

Two things still trip people up after switching to this. Feature transformations fit on the full dataset before splitting (normalizing by a global mean, say) leak the same way, so any scaling or encoding needs to be fit only on each fold's training portion. And if the target has a horizon, like predicting churn 7 days out, you need a buffer gap between the end of training and the start of validation, otherwise features in the validation window can be influenced by information that only existed because of what happens inside that gap.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_idx, val_idx in tscv.split(X):
  X_train, X_val = X[train_idx], X[val_idx]
  y_train, y_val = y[train_idx], y[val_idx]

Both add a penalty to the loss to discourage large coefficients, but the shape of the penalty differs. L2, ridge, penalizes the sum of squared coefficients, which shrinks everything toward zero smoothly but rarely pushes any single coefficient all the way to exactly zero. L1, lasso, penalizes the sum of absolute values, and because of the geometry of that penalty it tends to zero out some coefficients entirely, effectively doing feature selection as a side effect of fitting the model.

I reach for L1 when I have a large feature set and suspect only a subset genuinely matters, since the sparsity helps with interpretability and can trim a feature list before something latency-sensitive goes to production. I reach for L2 when I think most features carry some signal and I mainly want to control overfitting without deleting anything, especially when features are correlated, since L1 tends to arbitrarily pick one feature from a correlated group and zero out the rest, which makes the model unstable across retrains. Elastic net, a weighted mix of both penalties, is the practical middle ground when you want some sparsity without that instability.

Rule out a measurement problem before a real one. Check whether a logging pipeline broke, a tracking SDK update shipped, or a timezone bug double-counted or dropped a day of data, this explains a surprising fraction of scary-looking drops. If the data's genuinely clean, segment the drop: is it every platform or just iOS, every geography or just one market, every user cohort or specifically new signups versus long-tenured users. A drop concentrated in one segment points to a specific cause, a broken release, an app store outage, a competitor's launch, faster than staring at the aggregate number ever will. Only after ruling out measurement and localizing the segment would I start looking at what shipped the day before.

A trend that appears in several groups of data reverses or disappears once the groups are combined. The often-cited real case: a hospital's treatment appeared to have a lower overall success rate than another hospital's, but a higher success rate for both mild and severe cases individually, because the first hospital treated a much higher proportion of severe cases. Aggregating hid the confound, case severity, that explained the reversal. In a product context, this shows up when a feature "works" for every individual user segment but the aggregate metric looks flat, because the mix of segments shifted between the before and after periods, not the feature's actual effect.

Standard formulas for a confidence interval (mean plus or minus 1.96 times standard error) assume the sampling distribution is roughly normal, which CLT gives you for large samples but not necessarily for small ones, especially with skew. Bootstrap resampling sidesteps that assumption: resample your data with replacement thousands of times, compute the statistic on each resample, and take the 2.5th and 97.5th percentiles of the resulting distribution as your interval. It's computationally heavier but doesn't require assuming a distribution shape you can't verify with a handful of data points.

import numpy as np

data = np.array(sample_values)
boot_medians = []
for _ in range(10000):
  resample = np.random.choice(data, size=len(data), replace=True)
  boot_medians.append(np.median(resample))

lower, upper = np.percentile(boot_medians, [2.5, 97.5])

At alpha equals 0.05, each individual metric has a 5 percent chance of showing a false positive purely by chance. Test 12 independent metrics and the probability at least one shows "significant" by pure chance climbs to roughly 1 minus 0.95 to the 12th power, about 46 percent. That's closer to a coin flip than a rare edge case. Bonferroni correction (divide alpha by the number of tests, so 0.05 divided by 12 becomes your new threshold) is the blunt fix, conservative and easy to explain. Benjamini-Hochberg controlling the false discovery rate is less conservative and more common in practice when you have many metrics and can tolerate a controlled rate of false positives among the significant findings rather than eliminating them entirely. The fix that matters more than either: designate one primary metric before the test starts, and treat the other 11 as secondary or diagnostic, not equal co-winners.

No, and this is one of the more common ways teams accidentally p-hack themselves without meaning to. Early significance is exactly what you'd expect to see some fraction of the time from pure noise, that's what the peeking risk means. There's also a weekly seasonality issue, 2 days almost never spans a full weekly cycle, and user behavior on a Tuesday looks nothing like a Saturday for most consumer products. Run the pre-registered duration. If there's real business pressure to decide faster, design a shorter test with a pre-calculated sample size and duration from the start, using a group sequential design with adjusted thresholds, not abandon the plan mid-flight because the number happened to look good early.

Standard user-level randomization breaks down because treatment and control users interact with each other, a driver in the treatment group who gets faster matching pulls demand away from a rider in the control group, contaminating both arms. Cluster randomization by geographic market, randomizing entire cities rather than individual users, contains the spillover within each cluster, at the cost of needing far more clusters than you'd need individual users for the same statistical power, since cities are the actual unit of independence now, not people. Switchback experiments, randomizing time windows within the same market instead of splitting markets, are another common fix when there aren't enough independent markets to cluster-randomize cleanly.

Classic gaps-and-islands problem. Use LAG to compare each event's timestamp to the previous event for that user, flag a new session whenever the gap exceeds 30 minutes, then run a cumulative sum over those flags to assign a session ID.

WITH gaps AS (
 SELECT
  user_id,
  event_timestamp,
  event_timestamp - LAG(event_timestamp) OVER (
   PARTITION BY user_id ORDER BY event_timestamp
  ) AS gap
 FROM events
),
flagged AS (
 SELECT
  user_id,
  event_timestamp,
  CASE WHEN gap IS NULL OR gap > INTERVAL '30 minutes' THEN 1 ELSE 0 END AS new_session
 FROM gaps
)
SELECT
 user_id,
 event_timestamp,
 SUM(new_session) OVER (
  PARTITION BY user_id ORDER BY event_timestamp
 ) AS session_id
FROM flagged;

The gap IS NULL check matters, it's what marks each user's very first event as the start of session 1 instead of leaving it unclassified.

An index helps a targeted lookup, but a dashboard query scanning most of the table for an aggregate doesn't benefit much from one, it's not a needle-in-a-haystack problem, it's a scan-the-whole-haystack problem. Pre-aggregation is the real fix: build a materialized view or a nightly batch job that rolls raw events up to the daily or hourly grain the dashboard actually needs, so the dashboard queries a table with thousands of rows instead of hundreds of millions. Partitioning the raw events table by date also helps if queries filter on a date range, since the engine can skip entire partitions outside that range instead of scanning all of them. The real answer to give an interviewer: "it depends whether this needs to be real-time or can tolerate being an hour stale," because that single answer determines whether pre-aggregation is even an option.

First check whether the denominator changed. If the new signup flow brought in a wave of lower-intent users, a simplified form, a promotional push, activation rate can drop mechanically just because the mix of who's in the denominator shifted, with zero change in how well activation actually works for any individual user. Compute activation rate separately for the pre-launch signup cohort and the post-launch one, matched on some proxy for intent if one exists. If activation genuinely dropped even within a matched, similar-intent cohort, that's a real trade-off worth discussing, more volume at the top of the funnel in exchange for a lower-quality average lead, and whether that trade is worth it depends entirely on what happens further down the funnel, not on this one metric pair alone.

A churn model answers "who is likely to leave," a ranking problem. "Why are they leaving" is a causal, explanatory question, and a predictive model's feature importances aren't automatically a reliable answer to it. A feature can predict churn well while being a symptom rather than a cause, low usage in the final week predicts churn extremely well and is nearly useless as an explanation, since it's often just churn already happening, not a driver of it. I'd say this directly rather than silently building the model requested: "I can build a model that ranks who's at risk, and separately I can look at what changed for customers before they left, cancellation surveys, support ticket themes, usage pattern shifts weeks before the final drop-off, which gets closer to the why question you're actually asking." Committing to one deliverable that quietly doesn't answer the real question is worse than surfacing the gap up front.

A fixed-split A/B test sends a constant fraction of traffic to each variant for the whole run and only acts on the result at the end. That's clean for inference, since a stable, pre-committed allocation gives you an unbiased read on which variant is actually better, but it's wasteful if one variant is clearly underperforming early, because you keep sending real users to it for the full duration just to preserve the statistical guarantees.

A multi-armed bandit shifts allocation dynamically as it learns, sending more traffic toward whichever arm currently looks best, which minimizes regret, the cumulative cost of exposing users to the worse option while you're still figuring out which one wins. Thompson sampling maintains a probability distribution over each arm's true conversion rate and samples from it to decide allocation on the fly, naturally exploring more while uncertain and exploiting more once a clear winner emerges. UCB does something similar by picking the arm with the highest upper confidence bound at each step.

The cost is that a bandit makes it harder to hand stakeholders a precise, unbiased effect size at the end, since the allocation itself changed based on the data as it came in, which complicates the statistics behind any "variant B lifted conversion by X percent" claim. I'd reach for a bandit when the number of comparisons is high and the cost of running a losing arm for a full fixed duration is real money, ad creative rotation or dynamic pricing tests. I'd stick with a fixed-split A/B test when I need a defensible, precise causal estimate to justify a single, high-stakes ship decision.

First I'd check whether the offline evaluation had access to information that wouldn't exist at serving time, since that's the single most common cause. A feature computed using a window that includes the event being predicted, a join pulling in a value that gets updated after the prediction moment, or a label built from downstream data (using a field that only gets populated after the outcome already happened) will all inflate offline metrics without ever showing up in production, because production only has whatever existed at the actual prediction timestamp.

Second, I'd check for train-serve skew: does the pipeline generating training features actually match the pipeline computing features at inference time, value for value? These commonly diverge when one team owns the batch pipeline for training data and another owns the real-time feature store, and small differences, like null handling or a slightly stale join, produce features that look similar in aggregate but aren't computed identically. I'd log the actual feature vectors served in production and compare their distributions directly against training, feature by feature, rather than assume the two pipelines match because the code claims they should.

If both of those check out clean, I'd look at the exposure loop specific to recommenders: offline backtests score the new model against historical logs that were generated by the old policy's choices, so the new model looks good partly because it's only ever evaluated on items the old policy already decided to show. The honest test is an online holdout against live traffic, not another pass over historical logs.

I'd split this into feature drift and concept drift, since they need different tooling. For feature drift, I'd track each input feature's production distribution against its training-time distribution, using something like population stability index or a KS test for continuous features and a chi-squared test for categorical ones, computed on a rolling window and alerted when divergence crosses a threshold. On a model with hundreds of features I wouldn't monitor all of them equally, I'd prioritize by feature importance so alerts aren't dominated by noise on a feature that barely moves predictions.

Concept drift, where the relationship between features and the label shifts even while feature distributions look stable, is harder to catch directly because the true label often hasn't arrived yet, which is the whole reason a model is serving predictions instead of waiting. The proxy I'd track is the model's own prediction distribution over time, plus any faster proxy signal that correlates with the eventual label, like click-through as an early read on conversion. Once real labels do arrive, even with a lag, backtesting the live model against freshly labeled data on a rolling basis is the actual ground truth check.

What happens after detecting drift depends on the cause. A pipeline bug, a feature going stale or a schema change upstream, needs a fix, not a retrain. Genuine distribution shift in the world needs a retrain on a more recent window, and if the shift moves faster than your retrain cadence can keep up with, that's the signal to shorten retrain cycles or invest in online learning, with a shadow deployment to compare the new model against the old one on live traffic before fully cutting over.