Deep Learning Interview Questions (2026): With Answers

A neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function. On its own that's just logistic regression with extra steps. Stacking neurons into layers, and stacking layers into a deep network, lets the model compose simple functions into much more complex ones: early layers can learn edges or basic patterns, later layers combine those into higher-level concepts. Without the nonlinearity, stacking layers would be pointless, since a composition of linear functions is still just a linear function no matter how many layers you add.

Because without one, a deep network collapses mathematically into a single linear transformation, regardless of how many layers you stack. Matrix multiplication is associative, so W3(W2(W1x)) is just some matrix W times x. The nonlinearity, ReLU, sigmoid, tanh, GELU, is what actually gives the network the ability to represent curved decision boundaries and non-linear relationships in the data.

Two reasons dominate. Sigmoid saturates at both ends, its gradient approaches zero for large positive or large negative inputs, which makes gradients vanish as they flow backward through many layers. ReLU's gradient is exactly 1 for any positive input, so it doesn't squash the gradient the same way. ReLU is also just cheaper to compute, it's a max with zero, no exponential involved.

import torch.nn as nn

sigmoid = nn.Sigmoid()  # gradient near 0 outside roughly [-4, 4]
relu = nn.ReLU()     # gradient is 0 or 1, never squashed for x > 0

The tradeoff is "dying ReLU": a neuron whose input is always negative gets a gradient of exactly zero forever and never updates again. Leaky ReLU and GELU exist partly to patch that failure mode.

A loss function has to be differentiable, because gradient descent needs a gradient to follow. A metric just has to be meaningful to a human reading a report. Accuracy is a perfectly good metric but a terrible loss function, it's a step function with a gradient of zero almost everywhere, so gradient descent has nothing to climb down. Cross-entropy is differentiable and correlates with accuracy closely enough that optimizing it usually improves accuracy too, which is why we train on one and report the other.

A parameter is a value the training process learns by gradient descent, weights and biases. A hyperparameter is a value you set before training starts and that gradient descent never touches, learning rate, batch size, number of layers, dropout probability. The distinction matters because you can't learn a hyperparameter the same way you learn a weight; you have to search for it, by hand, grid search, or something like Bayesian optimization, and evaluate each choice by actually training a model with it.

The learning rate scales how big a step gradient descent takes in the direction of the negative gradient. Set it too high and the loss oscillates or diverges outright, because each update overshoots the minimum by more than the previous step got you closer. Set it too low and training technically still works, but takes an impractically long time to converge, and it's easy to mistake "the learning rate is too low" for "the model has stopped improving" if you're not watching the loss curve closely.

Overfitting is when a model has learned patterns specific to the training data, including its noise, rather than patterns that generalize to new data. You detect it by tracking training loss and validation loss on separate data throughout training: if training loss keeps dropping while validation loss flattens out or starts rising, the model is memorizing rather than generalizing from that point forward. A single snapshot of "training accuracy is higher than validation accuracy" isn't itself proof of overfitting, some gap is normal, what matters is whether the validation curve is still improving or has turned around.

Early stopping monitors validation loss during training and halts once it stops improving for a set number of epochs, keeping the checkpoint from the best epoch rather than the final one. The risk of relying on it alone is that it doesn't actually change what the model learns, it just picks a stopping point before overfitting gets bad, which means you're still training a model with whatever capacity and architecture you chose, potentially leaving accuracy on the table that better regularization would have unlocked by letting you train longer without the same overfitting cost.

A convolution layer slides a small learned filter across the input and reuses the exact same weights at every spatial position, instead of learning a separate weight for every input pixel the way a fully connected layer would. Two properties fall out of that: far fewer parameters for the same input size, and translation invariance, a feature detector that learns to spot an edge in the top-left corner of an image will recognize the same edge if it shows up in the bottom-right, because it's the same filter applied everywhere.

A feedforward network expects a fixed-size input and treats every input independently, with no notion of order or memory of what came before. An RNN maintains a hidden state that gets updated at every timestep and carried forward to the next one, which lets it process a variable-length sequence and let earlier elements influence how later elements are interpreted, the meaning of a word in a sentence depending on the words before it, for instance.

Feature extraction freezes every pretrained weight and only trains a new head, usually a single linear layer, on top of the frozen features. Fine-tuning unfreezes some or all of the pretrained weights and updates them too, usually with a much smaller learning rate than you'd use training from scratch, since you don't want to destroy the useful pretrained representations with a large early update. Feature extraction is faster and needs less data, but caps out lower in accuracy; fine-tuning can reach a better result but needs more data and more careful learning-rate choices to avoid catastrophic forgetting of the pretrained knowledge.

Unnormalized inputs with very different scales across features, one feature ranging 0 to 1, another ranging 0 to 10,000, distort the loss landscape into a badly stretched, elongated shape, and gradient descent on that shape tends to oscillate rather than move directly toward the minimum. Normalizing every feature to a similar scale makes the loss surface closer to symmetric, which lets a single learning rate work reasonably well across all the weights at once, instead of needing a different effective step size per feature.

MSE assumes the output is a continuous value and that errors are Gaussian-distributed around the true value, so it's the default for regression. Cross-entropy assumes the output is a probability distribution over discrete classes and directly measures how far your predicted distribution is from the true one-hot distribution, so it's the default for classification.

import torch.nn as nn

regression_loss = nn.MSELoss()
classification_loss = nn.CrossEntropyLoss() # expects raw logits, applies softmax internally

A detail candidates trip over: PyTorch's CrossEntropyLoss expects raw logits, not softmax probabilities, because it applies log_softmax internally for numerical stability. Applying softmax yourself first and then feeding that into CrossEntropyLoss is a bug that silently trains a worse model instead of throwing an error.

Softmax turns a vector of raw scores into a probability distribution by exponentiating each value and dividing by the sum of all the exponentials. Mathematically, subtracting a constant from every input before exponentiating doesn't change the output at all, because that constant factors out of both the numerator and denominator and cancels. In practice, if any logit is large, say 1000, exp(1000) overflows a float32 and you get NaN. Subtracting the max logit from every value first guarantees the largest exponent computed is exp(0) = 1, which keeps every intermediate value in a safe numeric range.

Backpropagation computes how much each weight in the network contributed to the final loss, so gradient descent knows which direction to nudge each one. It does this efficiently by working backward from the output: the gradient of the loss with respect to the last layer's weights is easy to compute directly, and the chain rule lets you reuse that result to compute the gradient one layer further back, without recomputing the whole forward pass from scratch for every single weight.

The efficiency is the actual insight. A naive approach that perturbed each weight slightly and re-ran the forward pass to estimate its gradient numerically would need one forward pass per weight, millions of them in a modern network. Backpropagation gets every weight's gradient in one backward pass, roughly the same cost as one extra forward pass.

Backpropagation multiplies gradients together layer by layer as it moves backward. If each layer's local gradient is consistently less than 1, which happens with sigmoid or tanh activations that saturate, the product of many small numbers shrinks toward zero as you go further back. In a 50-layer network, the gradient reaching the earliest layers can be so close to zero that those layers effectively stop learning, even though the loss is still nonzero.

ReLU activations, careful weight initialization, batch normalization, and residual connections all exist specifically to fight this. Residual connections in particular give the gradient a direct path backward that skips the multiplicative chain entirely, which is a big part of why ResNet-style architectures can go far deeper than plain feedforward stacks without gradients dying out.

The mirror image of vanishing gradients: if the local gradients multiplied together during backprop are consistently greater than 1, the product grows exponentially instead of shrinking, and weight updates become so large that training diverges, loss spikes to NaN or infinity. It's especially common in recurrent networks, where the same weight matrix gets applied repeatedly across many timesteps.

import torch

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

Gradient clipping is the standard fix: rescale the gradient vector if its norm exceeds a threshold, so no single update step can be arbitrarily large. It doesn't fix the underlying instability, it just puts a ceiling on how much damage one bad batch can do.

Adam keeps a running estimate of both the first moment (the mean of past gradients, like momentum) and the second moment (the variance of past gradients) for every parameter, and uses both to adapt the effective learning rate per parameter. Parameters with a history of small, noisy gradients get a relatively larger step; parameters with large, consistent gradients get a relatively smaller one. That adaptivity is why Adam often converges faster with less learning-rate tuning, especially early in training.

Plain SGD with momentum, on the other hand, often generalizes better on the final result for image classification tasks trained for a long time, which is part of why a lot of computer vision training recipes, including the original ResNet paper, still use SGD with a hand-tuned learning rate schedule instead of Adam. There's a real, documented tradeoff here, not just personal preference: Adam gets you to a decent result faster, SGD sometimes gets you to a slightly better final result if you have the compute budget to tune it properly.

The loss landscape is the surface you get by plotting the loss as a function of every weight in the network, an extremely high-dimensional surface with millions or billions of dimensions for a modern model. We can't verify we've reached the global minimum because checking that would require evaluating the loss at every possible weight configuration, which is computationally impossible. In practice it doesn't matter much either: research on the loss landscapes of over-parameterized networks suggests most local minima found by SGD or Adam are close in quality to each other, and the real risk in deep learning is a poorly conditioned saddle-point-heavy region slowing training down, not getting permanently stuck in a bad local minimum the way you might in a small convex problem.

During training, dropout randomly zeroes out a fraction of a layer's activations on each forward pass, a different random subset each time. That forces the network to not rely too heavily on any single neuron or narrow combination of neurons, since that neuron might be absent on the next batch. The effect is roughly equivalent to training a large ensemble of thinner sub-networks that share weights, and averaging their predictions.

import torch.nn as nn

class Net(nn.Module):
  def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(784, 256)
    self.dropout = nn.Dropout(p=0.5)
    self.fc2 = nn.Linear(256, 10)

  def forward(self, x):
    x = torch.relu(self.fc1(x))
    x = self.dropout(x)  # only active in model.train() mode
    return self.fc2(x)

At inference time, dropout is turned off entirely, and PyTorch automatically scales the remaining activations to keep the expected output magnitude consistent with training. Forgetting to call model.eval() before evaluation is the single most common bug related to this, and it produces exactly the symptom from the intro of this page, misleadingly poor validation numbers that have nothing to do with the model's actual quality.

L2 regularization adds a penalty term proportional to the sum of squared weights to the loss function, so the optimizer is rewarded for keeping weights small in addition to minimizing prediction error. Mechanically, this shrinks every weight toward zero a little on every update, proportional to its current magnitude, which discourages the network from relying too heavily on any single feature or input dimension.

optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)

The intuition worth having ready: smaller weights generally mean a smoother, less erratic decision boundary, which tends to generalize better than a boundary shaped tightly around every quirk of the training set.

Data augmentation increases the effective diversity of the training data itself, random crops, flips, color jitter, so the model sees more variation and has a harder time memorizing exact training examples. Dropout and weight decay instead constrain the model's capacity or add noise inside the network, independent of what the input data looks like. They attack the same problem, overfitting, from different angles, and in practice most production computer vision pipelines use both together rather than picking one, because they're not redundant with each other.

The receptive field is the region of the original input image that can influence a given neuron's output. A neuron in the first convolutional layer only sees whatever its filter's kernel size covers, say a 3x3 patch. A neuron in the second layer sees a 3x3 patch of the first layer's output, but each of those first-layer positions already summarized its own 3x3 patch of the original image, so the effective receptive field compounds. Stacking convolutional layers is how a network builds up the ability to recognize large, complex patterns, a whole face, an object, from small local filters, without ever needing a single filter as large as the object itself.

Pooling, typically max pooling, downsamples a feature map by taking the max (or average) value in each small local region, which reduces spatial resolution and gives a small amount of translation invariance for free, a feature shifting by one pixel usually still gets picked up by the same pooling window. It also cuts compute for every subsequent layer since the feature map is smaller.

import torch.nn as nn

pool = nn.MaxPool2d(kernel_size=2, stride=2) # halves height and width

Modern architectures increasingly use a strided convolution instead, a convolution with stride greater than 1 that downsamples while also learning the downsampling operation rather than using a fixed rule like max. That gives the network one more thing it can optimize instead of hardcoding, which is part of why architectures like ResNet use strided convolutions for downsampling and reserve pooling mostly for the very end of the network.

Dataset size is the deciding factor almost every time. With a few hundred or a few thousand labeled images, training from scratch usually underperforms badly, there simply isn't enough data for the network to learn good low-level features like edges and textures on its own. Fine-tuning a network pretrained on a large dataset gives you those low-level features for free, since edges and textures generalize across almost any visual domain, and lets your limited data focus on learning the task-specific higher layers. Training from scratch only makes sense with a genuinely large dataset, or when the target domain, medical imaging, satellite imagery, is different enough from natural images that the pretrained low-level features don't transfer well.

A plain RNN reapplies the same weight matrix to the hidden state at every single timestep, so the vanishing (or exploding) gradient problem is especially severe over long sequences, information from timestep 1 has to survive being multiplied through the same matrix dozens or hundreds of times to still influence timestep 200. An LSTM introduces a separate cell state alongside the hidden state, plus three learned gates, forget, input, and output, that control what gets added to, removed from, and read out of that cell state at each step. The cell state is updated mostly through addition rather than repeated multiplication by the same matrix, which gives gradients a much more direct path backward across many timesteps.

A GRU merges the LSTM's separate cell state and hidden state into one, and reduces the gate count from three to two, an update gate and a reset gate. Fewer parameters means faster training and less memory per layer, with roughly comparable performance on most tasks in practice. There's no consistent winner across the board, empirically they land close to each other on most benchmarks, so the real deciding factor in a production setting is usually compute budget and how much tuning time you have, not a strong prior that one architecture is inherently better.

Sequence length and parallelism matter most. An LSTM processes a sequence strictly one step at a time, so training throughput scales linearly with sequence length and can't be parallelized across timesteps on a GPU. A transformer's self-attention looks at the whole sequence at once, which parallelizes far better during training but costs quadratic compute and memory in sequence length, which becomes the bottleneck for very long sequences instead. For short sequences with limited data, an LSTM can still be a completely reasonable, cheaper choice; for anything where you can lean on large-scale pretraining or need long-range dependencies, a transformer wins in practice almost every time now.

Self-attention lets every position in a sequence look at every other position and decide, dynamically, how much to weight each one when building its own updated representation. Instead of a fixed rule like "only look at the previous 3 words," the model learns, per input, which other tokens are relevant, a pronoun attending strongly to the noun it refers to several words earlier, for instance, regardless of the distance between them.

import torch
import torch.nn.functional as F

def self_attention(x, Wq, Wk, Wv):
  Q, K, V = x @ Wq, x @ Wk, x @ Wv
  scores = Q @ K.transpose(-2, -1) / (Q.shape[-1] ** 0.5)
  weights = F.softmax(scores, dim=-1)
  return weights @ V

Mechanically, query, key, and value are three learned linear projections of the same input. The query for a position is compared against every position's key to produce a relevance score, softmax turns those scores into weights that sum to 1, and the output is a weighted sum of every position's value vector using those weights.

As the dimensionality of the query and key vectors grows, the dot product between them tends to grow in magnitude too, since it's a sum over more terms. Large dot products push the softmax into a region where its gradient is nearly zero, one score dominates completely and the rest collapse to near-zero weight, which makes learning unstable. Dividing by the square root of the key dimension keeps the scores in a range where softmax still has a usable gradient, independent of how large the embedding dimension happens to be.

Each head learns its own query, key, and value projections, into a smaller subspace, so different heads can specialize in different kinds of relationships in the same layer, one head tracking syntactic dependency, another tracking coreference, another something with no clean linguistic name at all. A single attention mechanism with the full dimensionality would have to represent all of that in one shared set of weights, whereas splitting into heads gives the model multiple independent "views" of the sequence at the same computational cost, since the total dimensionality across all heads still matches the original embedding size.

Self-attention has no inherent sense of order, the operation is permutation-equivariant, shuffle the input tokens and, without positional information, attention would produce the same set of outputs just reordered. An RNN gets order for free because it processes tokens strictly one at a time in sequence, order is baked into how the computation happens. A transformer processes the whole sequence in parallel with no such structural bias, so position has to be injected explicitly, added to (or otherwise combined with) each token's embedding before the first attention layer, so the model has some signal about which token came first.

Layer normalization normalizes across the feature dimension for a single example, independent of every other example in the batch, unlike batch normalization, which normalizes across the batch dimension for a single feature. That independence matters for two reasons in a transformer: sequence lengths vary between examples, which makes batch statistics awkward to define consistently, and batch norm's reliance on batch statistics performs poorly with the very small batch sizes that large models are often forced into by memory limits. Layer normalization has no dependency on batch size or composition at all, which makes it a cleaner fit for variable-length sequence inputs and for training regimes where batch size might be as small as 1 per device.

First confirm it's really overfitting and not a data leak or a bug in how validation loss gets logged, checking the validation set is genuinely held out and the eval code path uses model.eval(). If it's genuinely overfitting, the fastest levers are usually, in rough order of effort: add or strengthen data augmentation, increase dropout or weight decay, reduce model capacity if the dataset is small relative to the model, or simply add more training data if that's available. Restoring the checkpoint from the epoch where validation loss was lowest, rather than the final epoch, is the immediate fix regardless of which longer-term change you make.

Batch size is mostly constrained by GPU memory in practice, larger batches need more memory to hold activations for the backward pass. Within whatever memory allows, larger batches give a less noisy gradient estimate per step, since it's averaged over more examples, which usually means you can use a higher learning rate and get more stable, if not necessarily faster, convergence in terms of wall-clock time. Very large batches can hurt generalization slightly in some setups, since the noise in smaller-batch gradients acts as a mild regularizer, so bigger isn't automatically better even when memory allows it.

Class imbalance means one class vastly outnumbers another in the training data, fraud detection or medical diagnosis are common real examples. A model that always predicts the majority class gets 99% accuracy on a 99-to-1 dataset while being completely useless, it never catches a single positive case. The fix is measuring the right thing, precision, recall, F1, or a confusion matrix, instead of raw accuracy, and often addressing the imbalance directly during training, weighted loss functions that penalize errors on the minority class more heavily, oversampling the minority class, or undersampling the majority class.

Mixed-precision training runs most of the forward and backward pass in float16 or bfloat16 instead of float32, which roughly halves memory usage and can significantly speed up training on GPUs with dedicated lower-precision tensor cores. The risk is that float16 has a much smaller representable range than float32, and gradients that are already small can underflow to exactly zero in float16 before they'd have underflowed in float32, silently stopping certain weights from updating.

scaler = torch.cuda.amp.GradScaler()

with torch.autocast(device_type="cuda", dtype=torch.float16):
  output = model(x)
  loss = criterion(output, y)

scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

Loss scaling is the standard mitigation: multiply the loss by a large constant before the backward pass, which scales up every gradient enough to stay representable in float16, then divide back out before the optimizer step. Master weights are typically still kept in float32 for the actual parameter update, precisely because accumulating small updates in float16 loses precision over many steps.

Catastrophic forgetting is when a network, while learning a new task, overwrites weights that encoded useful general knowledge from its original pretraining, degrading performance on things it used to do well even though nobody explicitly retrained it on those tasks. It shows up most sharply with a learning rate that's too high for fine-tuning, or too many epochs on a narrow dataset, both of which push the weights far from their pretrained values in the direction of the new, narrower objective. The common mitigations are a much smaller learning rate during fine-tuning than pretraining used, freezing earlier layers entirely and only fine-tuning later ones, and early stopping based on a held-out set that includes some of the original broader distribution, not just the new narrow task's validation set.

A bidirectional RNN processes the sequence in both directions, forward and backward, and combines both hidden states at each position, which means every output depends on the entire sequence including tokens that haven't happened yet at that point in time. That's fine for a task where the whole input is available up front, translating a finished sentence, but it's structurally incompatible with a streaming setting where you need to produce output for timestep 10 before timestep 50 has even occurred. Real-time systems are stuck with a forward-only, causal architecture, or have to buffer a fixed window of future context and accept the resulting latency.

Warmup starts training with a small learning rate and increases it linearly over the first several hundred or thousand steps before switching to the main schedule, usually cosine decay. Early in training, weights are randomly initialized and the loss landscape near that starting point is poorly conditioned; a large learning rate applied immediately can push some layers, especially the deeper ones in a transformer with layer normalization stacked many times, into a bad region they never recover from.

Transformers rely on this more than CNNs partly because of how layer normalization interacts with the residual stream: early in training, before the normalization statistics have stabilized, the effective gradient scale is much less predictable than in a batch-normalized CNN. Skipping warmup on a transformer from scratch is one of the more reliable ways to get a training run that diverges in the first few hundred steps for no reason that shows up in the code.

Batch normalization normalizes a layer's activations to zero mean and unit variance across the current mini-batch, then applies a learned scale and shift so the network can undo the normalization if that's actually better for a given layer (Ioffe and Szegedy, arXiv, 2015). During training, the mean and variance used are computed from the current batch, which introduces a small amount of noise batch to batch and acts as a mild regularizer on its own.

During evaluation, using the current batch's statistics would be a problem for two reasons: batch size might be 1, making variance meaningless, and you want deterministic output for the same input regardless of what else happens to be in the batch. So at inference time, batch norm switches to a running average of mean and variance accumulated across training, fixed constants rather than something computed on the fly. That's why calling model.eval() changes batch norm's behavior in addition to dropout's, and why a model evaluated in .train() mode by mistake can give inconsistent, batch-dependent predictions that look like a bug in the model itself.

A residual block computes output = F(x) + x instead of just output = F(x), adding the block's input directly to its output. That skip path gives the gradient a direct, unimpeded route backward to earlier layers during backpropagation, bypassing the multiplicative chain through F entirely. Without it, a plain 100-layer network suffers badly from vanishing gradients and, surprisingly, often trains to worse accuracy than a shallower version of the same network, not because of overfitting but because it's genuinely harder to optimize.

The other framing worth knowing: if the optimal function for a block really is close to the identity, a residual block only has to learn a small correction, F(x) close to zero, which is an easier optimization target than learning the identity function outright through a stack of nonlinear layers.

A 1x1 convolution applies the same learned linear combination of channels at every spatial position independently, which functions like a fully connected layer applied per-pixel across the channel dimension only, without mixing information across spatial positions the way a larger kernel would. Its main practical use is changing the number of channels cheaply, shrinking 256 channels down to 64 before an expensive 3x3 convolution, then expanding back, which is exactly the bottleneck design used in ResNet-50 and later to keep parameter count and compute down while still going deep.

Teacher forcing feeds the ground-truth previous token as input to the decoder at each training step, instead of feeding the model's own previous prediction. It makes training faster to converge and more stable, since the model isn't compounding its own early mistakes across the sequence during training. The problem shows up at inference: the model has never actually practiced recovering from its own error, since at inference time there's no ground truth to feed back in, only its own possibly-wrong previous output. That mismatch between training and inference conditions is sometimes called exposure bias, and it's part of the motivation behind scheduled sampling, gradually mixing in the model's own predictions during later training, as a mitigation.

An encoder-only model, BERT is the canonical example, uses bidirectional attention, every token can attend to every other token including ones that come after it, which makes it well suited for understanding tasks: classification, extracting a representation, question answering over a given passage. A decoder-only model, the GPT family, uses causal attention, a token can only attend to itself and earlier tokens, which is what makes autoregressive generation, predicting the next token one at a time, coherent and trainable at all. An encoder-decoder model, the original Transformer and T5, runs a bidirectional encoder over the input and a causal decoder that also attends to the encoder's output, which fits sequence-to-sequence tasks with a clear distinct input and output, translation, summarization, best.

Full self-attention computes a score between every pair of positions in the sequence, so both compute and memory for the attention matrix scale with the square of sequence length. Doubling the context window doesn't double the cost, it roughly quadruples it, which is why extending context length from a few thousand to a few hundred thousand tokens is a genuinely hard engineering problem, not just a config change. FlashAttention and similar approaches don't reduce the asymptotic quadratic cost, but they restructure the computation to avoid ever materializing the full attention matrix in slow GPU memory, which cuts real wall-clock time and memory usage substantially even though the underlying math is unchanged.

This almost always points to something structural rather than a hyperparameter that's slightly off. Check, in rough order: is the learning rate literally zero or is the optimizer actually attached to the model's parameters; are gradients actually flowing, print param.grad after a backward pass and confirm it's not None or all zeros; is a layer accidentally frozen, requires_grad=False left over from an earlier experiment; and is the loss function receiving the shapes it expects, a silent shape mismatch that broadcasts instead of erroring can produce a loss that looks plausible but never actually corresponds to a real comparison between prediction and target. A loss that's stuck rather than just converging slowly is a sign the gradient isn't reaching the weights at all, not a sign you need a different learning rate.

Data parallelism copies the entire model onto every GPU and splits a batch of data across them, each GPU computes gradients on its slice, then gradients get averaged (all-reduced) across GPUs before every device applies the same update, keeping all copies identical. It works as long as the full model fits in memory on a single GPU. Model parallelism instead splits the model itself across GPUs, different layers or different pieces of the same layer live on different devices, which is the only option once a single model no longer fits on one GPU's memory at all, which is routinely the case for large language models today.

In practice, large-scale training usually combines both: model parallelism to fit the model across a set of GPUs, and data parallelism replicated across multiple such sets, sometimes alongside pipeline parallelism to keep every GPU busy instead of idle while waiting on a sequential chain of layers.

Normally, the forward pass stores every intermediate activation in memory because backpropagation needs them to compute gradients. Gradient checkpointing stores only a subset of activations, at a few checkpoint layers, and recomputes the rest by re-running the forward pass for that segment during the backward pass instead of keeping everything in memory the whole time. That trades memory for compute directly: you use meaningfully less GPU memory, which can be the difference between a model fitting on your hardware or not, at the cost of roughly 20 to 30 percent more compute time, since parts of the forward pass effectively run twice.

First, whether the learning rate was scaled to account for the larger effective batch size, since eight GPUs at the same per-GPU batch size means an 8x larger effective batch, and the same learning rate that worked for the smaller batch is often now too small to make comparable progress, or occasionally, with certain schedules, too aggressive relative to the smoothed, lower-variance gradient the larger batch produces. Second, whether batch normalization statistics are being synchronized across GPUs, plain batch norm computes statistics per-GPU independently in a naive multi-GPU setup, so with a small per-GPU batch size the normalization becomes much noisier than it would be with the full batch's statistics, and can behave differently across runs. Synchronized batch norm, or switching to layer norm, fixes that specific inconsistency.