Machine Learning Interview Questions (2026): Must-Know Q&A

Supervised learning trains on labeled examples, input paired with a known correct output, and learns a mapping from one to the other (regression for continuous outputs, classification for discrete ones). Unsupervised learning gets inputs with no labels and looks for structure on its own: clusters, dimensionality reduction, density estimation. Reinforcement learning is neither. An agent takes actions in an environment and learns from a reward signal that arrives later, sometimes much later, than the action that earned it.

Bias is error from a model that's too simple to capture the real pattern, it makes the same kind of mistake consistently regardless of which training set you give it. Variance is error from a model that's too sensitive to the specific training set it saw, it would produce a noticeably different model if you retrained it on a different sample from the same distribution. Total expected error decomposes roughly into bias squared plus variance plus irreducible noise, and pushing one down usually pushes the other up. A linear model fit to a curved relationship has high bias. A deep, unpruned decision tree fit to 200 rows has high variance.

Overfitting: training error is low, validation error is noticeably higher, and the gap tends to widen as you keep training or add model complexity. The model has memorized noise specific to the training set. Underfitting: both training and validation error are high and close together, the model hasn't captured the underlying pattern at all, adding more epochs or more data won't fix it because the model itself is too constrained.

The validation set is where you tune hyperparameters and pick between candidate models, and every time you look at it and adjust something based on what you saw, you leak a little bit of information about it into your decisions. If you tune against the test set instead, your final "test performance" is optimistic, it's really measuring how well you fit that specific set of numbers, not how the model generalizes. The test set only gets touched once, at the very end, to report a number you actually believe.

A confusion matrix for binary classification has four cells: true positives (correctly predicted positive), false positives (predicted positive, actually negative), false negatives (predicted negative, actually positive), and true negatives (correctly predicted negative). Accuracy is (TP+TN) divided by everything. Precision is TP divided by (TP+FP), of everything you flagged as positive, how much actually was. Recall is TP divided by (TP+FN), of everything that actually was positive, how much did you catch.

Linear regression fits a straight line (or hyperplane) directly to a continuous target and minimizes squared error. Logistic regression fits a linear combination of features too, but passes it through a sigmoid function to squash the output into a 0 to 1 probability, and it's trained with log loss (cross-entropy) instead of squared error, because squared error on a probability output doesn't punish confidently wrong predictions the way you'd want. The "logistic" part is the sigmoid; everything upstream of it is still linear, which is why logistic regression struggles on problems where the true decision boundary isn't roughly linear in the feature space.

kNN is supervised: given a new point, look at its k nearest labeled neighbors and vote (classification) or average (regression). k-means is unsupervised: no labels at all, it iteratively assigns points to the nearest of k cluster centers and recomputes those centers until they stop moving. Same letter k, same "distance to nearest things" intuition, completely different problem type, and mixing them up in an interview is a fast way to look like you crammed definitions without understanding them.

Naive Bayes is "naive" because it assumes every feature is conditionally independent of every other feature given the class label, which is almost never actually true (word order matters in a sentence, for instance) but the model works surprisingly well anyway, especially for text classification, because it only needs the ranking of probabilities to be roughly right, not the exact values.

PCA finds a new set of axes, called principal components, that are linear combinations of your original features, ordered so the first axis captures the most variance in the data, the second captures the most remaining variance while staying perpendicular to the first, and so on. Projecting your data onto the first few of these axes gives you a lower-dimensional representation that keeps as much of the original spread as possible for that number of dimensions. It's not selecting existing features, it's constructing new ones out of combinations of the old ones.

It computes the gradient (the direction of steepest increase) of the loss function with respect to the model's parameters, then updates the parameters a small step in the opposite direction, since that's the direction that decreases loss fastest locally. The learning rate controls step size. Too large and updates overshoot and can diverge; too small and training crawls or gets stuck in a shallow local dip that a bigger step would have escaped.

Anything based on distance (kNN, k-means, SVM) or gradient descent (logistic regression, neural nets) is sensitive to feature scale, because a feature ranging 0 to 100,000 dominates the distance calculation or the gradient magnitude compared to a feature ranging 0 to 1, regardless of which one actually matters more. Standardization (zero mean, unit variance) or min-max scaling fixes that. Tree-based models (decision trees, random forests, gradient boosting) don't care, because splits are based on relative ordering within a feature, not the absolute scale of the numbers, so scaling a tree-based model's inputs is wasted effort, not harmful, just pointless.

Customer segmentation is the classic one. You don't have ground-truth labels for "which segment does this customer belong to," you're trying to discover segments that don't exist yet in your data. K-means or hierarchical clustering finds groups based on feature similarity alone.

Evaluation is genuinely harder because there's no accuracy score to fall back on. Metrics like silhouette score or Davies-Bouldin index measure how tight and separated the clusters are mathematically, but they can't tell you whether the clusters are useful to a human. I've seen a clustering result score well on silhouette and still get rejected by the marketing team because the groups didn't map to anything they could act on.

Plot training error and validation error against training set size. If both curves converge to a high error and stay close together, that's bias, more data won't help much because the model is too weak to fit the pattern even with everything it already has. If training error stays low while validation error stays noticeably higher, a persistent gap that doesn't close, that's variance, and more training data, regularization, or a simpler model architecture will generally help.

The gotcha candidates miss: a gap that's shrinking as training size grows but hasn't closed yet doesn't mean you're done, it means you need to check whether it would close with more data or whether it's plateauing. Extrapolating the curve a bit, mentally or by actually fitting more points, matters more than eyeballing one snapshot.

Both add a penalty term to the loss function based on the size of the model's weights, but the penalty shape is different. L1 (Lasso) adds the sum of absolute values of the weights, and because that penalty has a sharp corner at zero, gradient-based optimization tends to push some weights exactly to zero, which gives you automatic feature selection. L2 (Ridge) adds the sum of squared weights, which shrinks all weights toward zero smoothly but rarely all the way to zero, so every feature keeps some small influence.

Rule of thumb I actually use: if I suspect a lot of the features are irrelevant and I want a sparser, more interpretable model, L1. If I think most features carry some real signal and I mainly want to tame overfitting without discarding anything, L2. Elastic Net splits the difference when you don't want to commit to one.

Dropout randomly zeroes out a fraction of neurons on each forward pass during training, so no single neuron can rely on any other specific neuron always being present, which forces the network to learn more redundant, distributed representations instead of memorizing exact co-activation patterns. Early stopping is simpler: watch validation loss during training and stop once it stops improving for a set number of epochs, rather than training to convergence on the training set alone. Both work because they cap how much the model can adapt to the specific noise in front of it.

The "more is better" instinct breaks down fast with dropout. Push the rate past roughly 0.5 on most architectures and the network starts underfitting, it's throwing away so much signal on each pass that it can't learn the real pattern either. 0.2 to 0.5 covers most cases I've tuned, and I'd rather start low and increase it only if validation loss shows real overfitting, than start high and wonder why training stalled.

Split the training data into k roughly equal folds. Train on k-1 of them, validate on the one held out, repeat k times so every fold gets a turn as the validation set, then average the k scores. Five folds is a common default; it balances a reasonably low-variance estimate against not retraining the model twenty times.

Plain k-fold assumes rows are exchangeable, which breaks for time series (future data would leak into training folds that come "before" it chronologically) and for grouped data (multiple rows from the same patient, user, or session ending up split across train and validation, which leaks identity-specific patterns rather than general ones). Use TimeSeriesSplit for the former and GroupKFold for the latter.

from sklearn.model_selection import cross_val_score, KFold

model = LogisticRegression(max_iter=1000)
cv = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="f1")
print(scores.mean(), scores.std())

Leave-one-out CV (LOOCV) is k-fold taken to its extreme, k equals the number of training examples, so each fold trains on every row except one and validates on that single held-out row, repeated once per row. It uses the maximum possible amount of training data on every iteration, which sounds ideal.

In practice it's rarely worth it for two reasons. Computationally it means training the model n times instead of 5 or 10, which is fine for a small dataset and painful for anything with tens of thousands of rows. More subtly, the n resulting models are trained on almost identical data (they differ by exactly one row), so their errors are highly correlated, and the resulting variance estimate of your CV score is actually less reliable than what you'd get from 10-fold. I reach for LOOCV only when the dataset is small enough that 10-fold would leave each validation fold with an uncomfortably small number of examples.

Optimize for precision when a false positive is expensive and a false negative is tolerable. Spam filtering: flagging a real email as spam (false positive) means someone misses a message they needed, which is worse than a spam email slipping through (false negative) once in a while. Optimize for recall when a false negative is the expensive mistake. Cancer screening: missing an actual case (false negative) can cost a life, so you'd rather flag extra people for a follow-up test (false positive) than let a real case through.

Fraud detection is the interesting middle case candidates sometimes get wrong, it's not automatically a recall problem. Depends on whether a false positive means declining a legitimate $40 purchase (mildly annoying) or freezing a legitimate $40,000 wire transfer (very annoying, possibly a lost customer). The right threshold is a business decision, not a math one.

F1 is the harmonic mean of precision and recall, 2 times (precision times recall) divided by (precision plus recall). The harmonic mean punishes imbalance between the two more than a plain average would, a model with 0.9 precision and 0.1 recall gets an F1 around 0.18, not 0.5.

What it hides: a single F1 number can't tell you which of precision or recall is the weak one, two very different models can land on the same F1 score. It also completely ignores true negatives, which matters if your business actually cares about correctly identifying the negative class too. I default to reporting precision and recall separately alongside F1, not instead of it, because the single number alone has cost me an argument or two about what a model is actually good at.

RMSE squares errors before averaging, which means a few large misses dominate the score, useful when big errors are disproportionately costly (a 50-unit miss on a house price prediction is worse than five separate 10-unit misses). MAE averages the absolute errors directly, treating every unit of error the same regardless of size, which is the more honest number to report when outliers in your target aren't actually more important, just noisier.

R² is often introduced as "percent of variance explained," which is true but easy to misread. An R² of 0.85 doesn't mean 85 percent of predictions are correct, it means the model's errors have 85 percent less variance than a naive model that just predicted the mean every time. A negative R² is possible too, and it means your model is doing worse than that naive mean-predictor, which I've actually seen happen on a badly leaked feature set once the leak was removed.

Macro-averaging computes precision and recall separately for each class, then takes an unweighted mean across classes, so a class with 20 examples counts exactly as much as a class with 20,000. Weighted-averaging does the same per-class computation but weights the mean by how many true examples each class has, so common classes dominate the final number.

Pick macro when every class matters equally regardless of frequency, a rare defect category in manufacturing QA shouldn't get buried by how well you do on the common ones. Pick weighted when overall performance on the realistic class mix is what the business actually cares about. I've watched a macro F1 of 0.55 and a weighted F1 of 0.91 both get reported for the same model in the same meeting, and the room went quiet because nobody had agreed in advance which one was "the" number.

At each node, the tree tries every feature and every reasonable threshold and picks the split that most reduces impurity, measured by Gini impurity or entropy for classification, variance reduction for regression. It repeats this recursively on each resulting branch.

Left unconstrained, a tree keeps splitting until every leaf is pure or has one example, which means it can carve out a rule for literally every training point, including the noisy ones. That's about as high-variance as a model gets. Max depth, minimum samples per leaf, and pruning after the fact are the standard fixes, or you skip the problem entirely by using an ensemble.

Two sources of randomness. Each tree trains on a bootstrap sample (random sample with replacement) of the training data, and at each split, only a random subset of features gets considered instead of all of them. Individually, each tree is still overfit and high-variance. But the trees' errors are decorrelated because they saw different data and different feature subsets, so averaging their predictions (majority vote for classification, mean for regression) cancels out a lot of that individual variance while keeping the low bias intact.

C controls the trade-off between a wide margin and classifying every training point correctly. A small C tolerates more margin violations for a smoother, simpler boundary (more bias, less variance); a large C tries hard to classify every training point right, even if that means a jagged, tightly-fit boundary (less bias, more variance, and more prone to overfitting on noisy points).

Gamma controls how far a single training example's influence reaches. A small gamma means each point's influence stretches far, producing smoother decision boundaries; a large gamma means each point only influences its immediate neighborhood, which can carve out a boundary that hugs the training data tightly enough to overfit badly. In practice I grid search both together rather than tuning one at a time, they interact, and a high C paired with a high gamma is where I've seen SVMs overfit hardest.

Scikit-learn's default feature_importances_ on tree ensembles is mean decrease in impurity, averaged across every split in every tree where that feature was used, weighted by how many samples reached that split. It's fast because it's a byproduct of training, no extra computation needed.

The catch: it's biased toward high-cardinality and continuous features, because they offer more possible split points, so the tree gets more chances to (over)use them even when a categorical feature with fewer options is just as predictive. Permutation importance fixes this, shuffle one feature's values, measure how much validation performance drops, repeat per feature, since it measures the effect on held-out predictions directly rather than counting splits in the training process. It costs more compute but I trust it more once a stakeholder asks "which feature actually matters" for a decision they're going to act on.

Two standard approaches, used together more often than either alone. The elbow method plots within-cluster sum of squared distances (inertia) against k, and you look for the point where adding another cluster stops meaningfully reducing that number, the "elbow" in the curve. Silhouette score measures, for each point, how much closer it is to its own cluster than to the nearest other cluster, averaged across all points, and it's more useful when the elbow is ambiguous, which it often is on real data.

Neither is a substitute for actually looking at what the clusters mean once you have them. I've picked a k that scored well on both metrics and still gotten it rejected because two of the "distinct" clusters turned out to represent the same customer behavior measured at different account ages.

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    print(k, km.inertia_, silhouette_score(X, km.labels_))

Look at the cumulative explained variance ratio and pick the number of components where it crosses a threshold you're comfortable with, 90 or 95 percent is common, though the right number depends entirely on what you're using the components for. Downstream visualization usually wants exactly 2 or 3 regardless of how much variance that captures. Downstream modeling can usually afford to keep more.

A scree plot, explained variance per component plotted in order, gives you the same elbow-hunting intuition as choosing k in k-means, an early component that captures 40 percent of variance followed by ten components each adding a fraction of a percent tells you the data's real dimensionality is much lower than its raw feature count.

from sklearn.decomposition import PCA
import numpy as np

pca = PCA().fit(X_train_scaled)
cumulative = np.cumsum(pca.explained_variance_ratio_)
n_components = int(np.argmax(cumulative >= 0.95) + 1)

Batch computes the gradient using the entire training set before making a single update, exact but slow, and one pass through the data yields exactly one update. Stochastic (SGD) updates after every single example, noisy but fast, and that noise can actually help the optimizer bounce out of shallow local minima or saddle points that a smoother gradient would settle into. Mini-batch splits the difference, updating after a batch of, say, 32 or 256 examples, and it's the practical default almost everyone actually uses because it's vectorizable on GPU hardware in a way single-example SGD isn't.

I don't think "SGD helps escape local minima" gets emphasized enough in prep material relative to how often it's the actual answer an interviewer is fishing for when they ask why you'd ever want noisier gradients on purpose.

Plain SGD updates parameters using only the current gradient, which means it can zigzag badly across a loss surface shaped like a narrow ravine, oscillating back and forth across the steep direction while making frustratingly slow progress along the shallow direction toward the actual minimum. Momentum keeps a running, exponentially-weighted average of past gradients and updates using that average instead of the raw current gradient, so consistent directions accumulate speed while oscillating directions partially cancel out.

It's the same intuition as a ball rolling downhill picking up speed rather than a series of independent, memoryless steps. In practice a momentum term around 0.9 is a common default, and it's usually the first thing I'd add to plain SGD before reaching for anything fancier.

Adam keeps two running averages per parameter: one of past gradients (like momentum, giving it direction and speed) and one of past squared gradients (which it uses to scale the effective learning rate down for parameters that have been getting large, frequent updates, and up for parameters that have been getting small, rare ones). That second piece is the adaptive part, borrowed from earlier optimizers like RMSprop, and it means Adam doesn't need as much manual learning-rate tuning as plain SGD to get reasonable results quickly.

It's the default almost everywhere because it works reasonably well out of the box across a wide range of architectures without much tuning, which matters when you're iterating fast. The honest caveat, one that shows up more in research discussions than interview prep, is that plain SGD with a well-tuned learning rate schedule and momentum still generalizes slightly better than Adam on some architectures, image classification being the most cited case, so "Adam by default, SGD if you have the tuning budget and it's a well-studied architecture" is closer to the real practitioner's answer than "always use Adam."

MSE (mean squared error) is the default for a reason, it's smooth and differentiable everywhere, which gradient-based optimizers like, but it squares errors, so a handful of big outliers can dominate the loss and pull the whole model toward fitting them at the expense of everything else. MAE (mean absolute error) treats every unit of error equally regardless of size, which is less swayed by outliers but has a kink at zero that makes optimization slightly less well-behaved near the minimum.

Huber loss is the compromise, quadratic for small errors, linear for large ones, past a threshold you set. It behaves like MSE where errors are small (smooth, well-behaved gradients) and like MAE where errors are large (doesn't let a handful of outliers dictate the whole model). I reach for it specifically when I know the target has a few genuine outliers I don't want to ignore but also don't want steering the fit.

One-hot encoding creates a binary column per category, safe and leak-free, but it explodes column count for high-cardinality features (a zip code column with 30,000 unique values becomes 30,000 sparse columns). Target encoding replaces each category with the mean of the target variable for that category, which stays compact regardless of cardinality but introduces real leakage risk if you compute the encoding using the full dataset before splitting, the encoding for a category literally contains information about that category's target values, some of which sit in your validation set.

The fix is computing target encodings only from the training fold, inside cross-validation, never from the full dataset up front. Skipping that step is one of the more common ways I've seen a model look great in validation and then quietly disappoint in production.

Filter methods score each feature independently of any model, correlation with the target, chi-squared test, mutual information, and rank or threshold on that score before training even starts. Fast, model-agnostic, but blind to interactions between features. Wrapper methods actually train a model repeatedly with different feature subsets and keep whichever subset scores best, recursive feature elimination is the common example, which captures feature interactions the model cares about but gets expensive fast as feature count grows.

Embedded methods build selection into training itself, L1 regularization zeroing out irrelevant weights, or a tree-based model's split-based importance, so you get selection as a side effect of a single training run rather than a separate step. I default to embedded when I'm already using a tree ensemble or L1-regularized model anyway, and reach for filter methods first on anything with hundreds of candidate features, just to cut the search space before something slower runs on it.

Mean imputation collapses variance in that feature, every filled-in row now sits at exactly the same value, which understates the true spread and can weaken the feature's apparent relationship with the target even when missingness itself is meaningless. Worse, if missingness isn't random, if income is more likely to be missing for the highest earners because they skip that survey field, mean imputation actively introduces bias rather than just noise.

Better defaults depend on why the data is missing. If it's random, median imputation (less sensitive to outliers) or model-based imputation (predicting the missing value from other features) both beat a flat mean. If missingness itself carries signal, add a binary "was this missing" indicator column alongside whatever imputed value you use, so the model can learn from the fact of missingness separately from the guessed value.

Variance inflation factor (VIF) is the standard check, regress each feature against every other feature and see how well the others predict it; a VIF above 5 or 10 (conventions vary) flags a feature that's largely redundant with the rest. A simple correlation matrix catches the obvious pairwise cases faster, though it misses multicollinearity that only shows up across three or more features combined.

It matters even outside feature selection because it destabilizes coefficient estimates in linear and logistic regression specifically, two correlated features can end up with wildly different, even oppositely-signed, coefficients depending on tiny changes in the training data, which makes the model's coefficients meaningless for interpretation even though its predictions might still be fine. Tree-based models mostly shrug this off since they split on one feature at a time regardless of what else correlates with it.

Bagging (bootstrap aggregating) trains multiple independent models in parallel, each on a different bootstrap sample of the training data, then averages their predictions. Because the models are trained independently and see different data, their errors are only weakly correlated, and averaging weakly correlated errors cancels out a chunk of variance without touching bias much. Random forest is bagging plus random feature subsetting at each split, layered on top.

If 99 percent of your data is the negative class, a model that predicts "negative" for everything scores 99 percent accuracy while catching zero positives, the metric looks great and the model is useless. Report precision, recall, F1, or PR-AUC instead. For the imbalance itself: class_weight="balanced" (cheap, no data duplication, works on most scikit-learn classifiers), random undersampling of the majority class (fast but throws away data), or SMOTE, which generates synthetic minority examples by interpolating between real ones instead of just duplicating them.

Backpropagation computes the gradient of the loss function with respect to every weight in the network, layer by layer, working backward from the output. It's an application of the chain rule from calculus, the gradient at an early layer is the product of the local gradients at every layer between it and the loss, computed once and reused rather than recalculated from scratch for each weight. Without that reuse, computing gradients for a deep network one weight at a time would be computationally impossible at any real scale.

What trips candidates up is treating it as a mysterious separate algorithm rather than the chain rule applied systematically, with intermediate results cached. Once that clicks, questions about vanishing gradients stop being abstract, they're just what happens when you multiply a long chain of small numbers, derivatives less than 1, together and the product shrinks toward zero by the time it reaches the earliest layers.

Stack any number of purely linear layers together and the composition is still just one linear function, no amount of depth buys you anything beyond what a single layer could do. Non-linear activations between layers are what let a deep network approximate genuinely non-linear relationships, each layer can bend the decision boundary in a way the previous layer couldn't on its own.

Sigmoid and tanh squash their input into a fixed range (0 to 1, or -1 to 1) and both saturate at the extremes, meaning their gradient approaches zero for large positive or negative inputs, which is exactly the vanishing gradient problem showing up layer after layer in a deep network. ReLU (the max of 0 and the input) doesn't saturate on the positive side, its gradient is a flat 1 for any positive input, which is most of why it displaced sigmoid and tanh as the default hidden-layer activation. Its own failure mode is the "dying ReLU" problem, a neuron whose input is consistently negative outputs zero and has zero gradient, so it stops updating entirely and effectively drops out of the network permanently. Leaky ReLU, which allows a small negative slope instead of a hard zero, is the common fix.

An RNN processes a sequence one token at a time and has to compress everything it's seen so far into a single fixed-size hidden state, which becomes a bottleneck on long sequences, information from ten tokens ago has to survive being repeatedly overwritten before it can influence token fifty. LSTMs extended how long that information could survive with gating mechanisms, but the sequential, one-token-at-a-time processing itself was still the constraint, and it made training slow since you can't parallelize across the sequence, each step genuinely depends on the previous one finishing.

Attention lets every position in a sequence look directly at every other position and weigh how relevant each one is, without routing that information through a chain of hidden states first. That solves the long-range dependency problem directly, token fifty can attend straight back to token ten if it's relevant, and it's fully parallelizable across the sequence during training, which is a large part of why transformer training scales so much better on GPU hardware than RNN training does.

Data drift is a change in the input distribution, the features themselves start looking different from what the model trained on, average transaction size creeps up over a year, a new user segment starts showing up that barely existed in training data. The relationship between features and target hasn't necessarily changed, the inputs just moved. Concept drift is the relationship itself changing, the same input now maps to a different correct output, fraud patterns shifting after a new payment method launches, customer behavior shifting after a pricing change.

Detecting data drift is more tractable since you don't need fresh labels, compare the distribution of live features against the training distribution using something like population stability index or a KS test, and flag features that have moved past a threshold. Concept drift is harder to catch quickly because it usually only becomes visible once you have enough fresh labels to measure actual model performance dropping, which can lag the real shift by weeks depending on how fast labels come back. I'd rather have both a lightweight drift monitor on the inputs and a slower, label-based performance monitor running side by side, since either one alone misses a different failure mode.

Labeling more data is often the right answer eventually, but it's slow and expensive, and 800 examples is enough to get started with semi-supervised techniques while labeling continues in parallel. A common approach: train a model on the 800 labeled examples, use it to generate pseudo-labels for the confident predictions on the unlabeled set, fold those back into training, and repeat. Self-training like this works best when the labeled set is reasonably representative of the full distribution, if it's skewed, the pseudo-labels just amplify that skew.

An alternative worth mentioning: use the unlabeled 50,000 for unsupervised pretraining, a simple autoencoder or contrastive objective, then fine-tune on the 800 labeled examples. Which approach wins depends on the data and I wouldn't commit to one without trying both on a held-out slice first.

This throws people because it seems backwards, but it's common and usually not a bug. Three likely causes. First, regularization techniques like dropout are active during training but turned off during validation, so training loss gets computed on a handicapped model while validation loss gets computed on the full one. Second, if training loss is averaged across an epoch while the model is still improving early in that epoch, and validation is measured at the epoch's end with the fully-updated weights, you're comparing an average against a snapshot. Third, and worth checking first because it's the embarrassing one: the validation set might be smaller, less noisy, or accidentally easier than the training distribution, sometimes because of how a split was done.

I'd check case three before assuming it's the healthy explanation. A suspiciously good validation number has burned me before.

If you use the same k-fold splits to both tune hyperparameters (picking the max_depth that scores best) and report a final performance number, you're optimistically biased, you picked the hyperparameter that happened to fit those particular folds best, so the reported score partly reflects lucky alignment with that specific split rather than true generalization.

Nested cross-validation fixes this with two loops. An outer loop splits the data into k folds for the final performance estimate. Inside each outer training fold, an inner loop runs its own k-fold (or grid search) purely to select hyperparameters, without ever touching the outer validation fold. The outer score you report was never used to pick anything. It costs roughly k times more compute than plain k-fold with grid search, which is exactly why teams skip it under deadline pressure and then wonder why the "94 percent F1" model came in around 89 percent in production.

from sklearn.model_selection import GridSearchCV, cross_val_score, KFold

inner_cv = KFold(n_splits=5, shuffle=True, random_state=1)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=2)

clf = GridSearchCV(estimator=model, param_grid=params, cv=inner_cv, scoring="f1")
nested_scores = cross_val_score(clf, X, y, cv=outer_cv, scoring="f1")
print(nested_scores.mean(), nested_scores.std())

ROC-AUC is the probability that a randomly chosen positive example gets ranked higher (a higher predicted score) than a randomly chosen negative example, across every possible classification threshold at once. 0.5 is random guessing, 1.0 is perfect ranking. It's threshold-independent, which is useful when you haven't picked an operating point yet.

It's misleading on heavily imbalanced data because the false positive rate on the x-axis is measured against a large pool of negatives, so a model can rack up a lot of false positives in absolute terms while the rate still looks small and the curve still looks good (scikit-learn's model evaluation guide covers this trade-off directly). Precision-recall AUC is the better metric to lead with once positives are rare, since it's sensitive to exactly the failure mode ROC-AUC hides.

Log loss (cross-entropy) penalizes a wrong prediction more the more confident the model was in it. A model that's 51 percent confident and wrong gets a small penalty. A model that's 99 percent confident and wrong gets hammered, log loss approaches infinity as predicted probability for the true class approaches zero.

This is why a model can look fine on accuracy and still have an ugly log loss, accuracy only cares whether the argmax prediction crossed the 0.5 threshold correctly, it doesn't care whether the model said 0.51 or 0.99 to get there. A poorly calibrated model, one that's systematically overconfident, racks up accuracy just fine while its log loss quietly tells a worse story. This matters a lot more than most candidates expect once a model's output probability gets used downstream for anything, ranking leads by risk score, setting a threshold that varies by business unit, rather than just as a binary yes or no.

Worth knowing the concept, less worth reaching for in production on most tabular problems now. The kernel trick, projecting data into a higher-dimensional space without explicitly computing the transformation, is genuinely elegant and still shows up as a "explain how this works" question. But gradient-boosted trees and, for large datasets, neural nets have mostly replaced SVMs for real tabular work, partly because SVM training scales poorly past roughly 100,000 rows and partly because tree ensembles need less feature preprocessing to hit similar accuracy.

I still see SVM interview questions land more often than I'd expect given how rarely I see SVM in production code, which tells me it survives as a conceptual gate (do you understand margins and kernels) more than a practical one.

Classic gradient boosting fits each new tree to the residuals of the ensemble so far and stops there. XGBoost adds an explicit regularization term to the loss function that penalizes tree complexity directly, the number of leaves and the magnitude of the leaf weights, so the optimizer is choosing splits that improve the loss and stay simple, not just splits that improve the loss.

It also handles missing values natively by learning a default split direction for them during training instead of requiring imputation upfront, uses a second-order (Newton) approximation of the loss instead of just the gradient, which converges faster, and parallelizes the search for the best split point across features rather than growing trees fully sequentially. None of that changes the core boosting idea, but it's the difference between a from-scratch gradient boosting implementation and one that survives contact with a messy, real dataset.

import xgboost as xgb

model = xgb.XGBClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=4,
    reg_lambda=1.0,   # L2 penalty on leaf weights
    reg_alpha=0.0,    # L1 penalty on leaf weights
)
model.fit(X_train, y_train)

K-means assumes clusters are roughly spherical and similarly sized, because it's minimizing distance to a single centroid per cluster, and it's sensitive to initialization, a bad random start can converge to a worse local optimum (which is why n_init defaults to running it multiple times and keeping the best). It also needs you to specify k upfront and it can't find clusters shaped like crescents or nested rings.

DBSCAN doesn't need k specified, it groups points that are densely packed together and labels sparse points as noise, which makes it good for irregularly shaped clusters and for datasets where some points genuinely don't belong to any cluster. The trade-off is DBSCAN struggles when clusters have very different densities, a single epsilon radius that works for one dense cluster can swallow a sparser one whole. Hierarchical clustering builds a full tree of nested clusters and lets you cut it at any level after the fact, which is useful when you're not sure what k should be and want to explore a few options without rerunning the algorithm each time, but it scales poorly, O(n squared) or worse, past a few tens of thousands of points.

Fitting the PCA transform on the full dataset before splitting into train and test. PCA is learning the directions of maximum variance from whatever data you give it, and if that includes your test set, the components themselves already encode information about the test distribution before the model ever sees a single row, which is leakage even though no label ever crossed the line.

Second mistake, smaller but common: running PCA on unscaled features. Because PCA is chasing variance, a feature measured in the thousands will dominate the first component over a feature measured in single digits, regardless of which one actually matters, so standardizing first is close to mandatory unless every feature already happens to share the same scale.

You technically can, MSE against a 0/1 label is a valid loss function, but it punishes confidently wrong predictions much less than it should. If the true label is 1 and the model predicts 0.01, MSE's penalty is (1-0.01) squared, about 0.98, roughly the same penalty it would apply if the model predicted 0.1 instead, about 0.81. Cross-entropy penalizes that same 0.01 prediction with a loss that shoots toward infinity as the predicted probability for the true class approaches zero, which matches the intuition that a confidently wrong prediction should hurt a lot more than a mildly wrong one.

There's also a gradient-flow reason. Combined with a sigmoid or softmax output, cross-entropy's gradient with respect to the model's raw output (before the activation) simplifies to just the predicted probability minus the true label, a clean, well-scaled signal. MSE through a sigmoid produces a gradient that gets multiplied by the sigmoid's derivative, which is close to zero when the sigmoid is saturated near 0 or 1, exactly where a confidently wrong prediction most needs a strong correction. That's the vanishing gradient problem showing up in miniature, and it's the real reason cross-entropy won out, not just convention.

Boosting trains models sequentially, not in parallel, and each new model focuses specifically on the examples the previous models got wrong (in gradient boosting, each new tree fits the residual errors of the ensemble so far, scaled by a learning rate). Bagging reduces variance from already-decent base models; boosting reduces bias by chaining together deliberately weak learners that each fix a specific gap.

The risk: because each round is explicitly correcting the previous rounds' mistakes, including whatever noise happened to be in those mistakes, too many rounds with too high a learning rate memorizes the training set's idiosyncrasies. Early stopping on a validation set, a smaller learning rate paired with more rounds, and limiting individual tree depth are the standard guardrails (scikit-learn's cross-validation docs cover using held-out folds for exactly this kind of early-stopping decision).

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

bagged = RandomForestClassifier(n_estimators=300, max_features="sqrt", random_state=42)
boosted = GradientBoostingClassifier(n_estimators=300, learning_rate=0.05, max_depth=3)
# bagged: parallel, independent trees, averaged
# boosted: sequential trees, each correcting the last

The most common mistake: applying SMOTE before splitting into train and validation, which means synthetic points generated from validation-set neighbors leak into training, and the model gets evaluated on data that's statistically close to what it trained on. Fit SMOTE only on the training fold, inside cross-validation if you're doing CV, never on the full dataset up front.

Second issue, less about a bug and more about interpretation: your validation set is now balanced, but the real world isn't. A model that looks great against an artificially 50/50 validation set can still perform differently against production traffic that's still 99/1. I'd always keep one untouched, naturally-imbalanced holdout set around specifically to catch this, separate from whatever you used to tune the model.

Batch norm normalizes the inputs to a layer, subtracting the batch mean and dividing by the batch standard deviation, then applies a learned scale and shift so the network can undo the normalization if that's actually what's optimal. It's computed per mini-batch during training and uses a running average of mean and variance at inference, when there's no batch to compute statistics from.

The higher-learning-rate benefit comes from keeping each layer's input distribution stable across training steps. Without batch norm, a change in an early layer's weights shifts the distribution of inputs every later layer sees, which the original paper called internal covariate shift, forcing later layers to constantly readjust to a moving target and making large learning rates unstable. Normalizing those inputs at every layer means later layers see a roughly consistent distribution regardless of what earlier layers are doing mid-training, so you can push the learning rate higher without the whole thing diverging. It also has a mild regularizing side effect, since the batch statistics add a bit of noise per mini-batch, which is part of why dropout is often used more sparingly in architectures that already have batch norm.

Every token gets projected into three vectors: a query (what this token is looking for), a key (what this token has to offer, as a kind of label other tokens can match against), and a value (the actual content this token contributes if it gets attended to). A token's query is compared against every other token's key, usually via a dot product, to produce a relevance score for each pair, those scores get scaled and passed through softmax to turn them into weights that sum to 1, and the token's new representation becomes a weighted sum of every token's value vector, weighted by those scores.

The query-key-value split is what lets the mechanism separate what a token is looking for from what it has to offer from what it actually contributes, a single combined vector per token couldn't do all three jobs independently. Multi-head attention runs several of these query-key-value projections in parallel with different learned weights, so different heads can specialize, one head might end up tracking subject-verb agreement, another might track coreference, without anyone explicitly telling it to.

Training-serving skew is when the features a model sees in production are computed differently, even subtly, from how they were computed during training, so the model is effectively being evaluated on a slightly different problem than the one it learned. A classic version: a feature like "average order value over the last 30 days" gets computed with a batch job during training using a clean historical snapshot, but at serving time the same feature is computed from a live, possibly incomplete or differently-timed data source, missing today's orders, a different timezone cutoff, a join against a table that hasn't finished updating yet.

It passes offline tests because those tests validate the model against the training pipeline's version of the features, never against what the serving pipeline actually produces at request time. The fix that actually works at scale is a shared feature store, one canonical definition and computation path for every feature, used identically by both the training job and the serving path, rather than two teams or two codebases independently reimplementing "average order value" and hoping they match. Short of that, I'd at minimum log the actual feature values the model receives in production and periodically diff them against what the training pipeline would have computed for the same raw inputs.