Computer Vision Interview Questions (2026): 50 Q&A

A color image is a 3D array: height by width by channels, usually three channels for RGB, each pixel value an integer from 0 to 255 (or a float after normalization). A grayscale image drops to a single channel. Before anything reaches the network, that raw array almost always gets resized to a fixed shape and normalized, subtracting a per-channel mean and dividing by a standard deviation, so the pixel distribution roughly matches whatever the backbone was pretrained on.

Batching stacks images along a new leading dimension, giving the familiar (batch, channels, height, width) tensor shape that PyTorch and most frameworks expect. Get the channel order or the normalization stats wrong and the model still runs, it just performs quietly worse, which is one of the more frustrating classes of bug in this field.

A convolution slides a small learned filter, say 3x3, across the image and computes a weighted sum at each position, producing one value per position in an output feature map. The same filter weights get reused at every spatial location, which is the key idea: a filter that learns to detect a vertical edge in the top-left corner uses those exact same weights to detect a vertical edge anywhere else in the image.

import torch.nn as nn

conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)
# 16 filters, each 3x3x3, shared across every spatial position

A fully connected layer would need a separate weight for every pixel-to-neuron connection, which for a 224x224 image is millions of parameters before you've learned anything useful, and it throws away the fact that nearby pixels are related. Weight sharing plus locality is why convolutions generalize from far fewer examples.

Without padding, every convolution shrinks the spatial size a little, since a filter can't center itself on a border pixel without running off the edge. Valid padding means no padding at all, the output is smaller than the input by (kernel_size - 1). Same padding adds zeros around the border so the output keeps the same height and width as the input.

Same padding is the default in most modern backbones because it lets you stack many layers without the feature map shrinking to nothing before you've extracted useful features, and it keeps skip connections in architectures like ResNet dimensionally compatible without extra cropping logic.

The first conv layer's filters have a depth matching the number of input channels, three for RGB, one for grayscale. Dropping to grayscale means each first-layer filter is now 1 channel deep instead of 3, so it loses any ability to distinguish objects purely by color, a red apple and a green apple that share the same shape and texture become indistinguishable to the model at that first layer and every layer after it.

For tasks where color genuinely carries the discriminative signal, ripeness detection, traffic light state, this is a real accuracy cost, not a minor implementation detail. For tasks where shape and texture dominate, document text detection, most industrial defect detection, grayscale often performs almost identically while cutting input data volume by two-thirds, which is worth it purely for the storage and bandwidth savings.

Feature extraction freezes the pretrained backbone entirely and only trains a new head, a linear layer or small classifier, on top of the frozen features. Fine-tuning unfreezes some or all of the backbone and continues training it, usually with a much smaller learning rate than you'd use from scratch.

Feature extraction wins when your dataset is small, a few hundred to a couple thousand images, since fine-tuning that many parameters on that little data tends to overfit or actively destroy the pretrained features. Fine-tuning wins once you have enough data, typically several thousand images and up, and especially when your target domain looks meaningfully different from what the backbone was pretrained on, medical imaging or satellite photos versus the natural images ImageNet is built from.

Classification assigns one label to the whole image. Detection finds every instance of an object, drawing a bounding box around each one and labeling it, so an image can have five boxes labeled "car" and two labeled "person." Semantic segmentation labels every pixel with a class, but doesn't distinguish between two instances of the same class, two overlapping cars both get the "car" pixel label with no boundary between them. Instance segmentation is the combination: pixel-level masks, but each instance gets its own separate mask.

IoU divides the area of overlap between a predicted box and the ground truth box by the area of their union. A perfect match gives IoU of 1.0, no overlap gives 0.0.

def iou(box_a, box_b):
  xa = max(box_a[0], box_b[0])
  ya = max(box_a[1], box_b[1])
  xb = min(box_a[2], box_b[2])
  yb = min(box_a[3], box_b[3])
  inter = max(0, xb - xa) * max(0, yb - ya)
  area_a = (box_a[2] - box_a[0]) * (box_a[3] - box_a[1])
  area_b = (box_b[2] - box_b[0]) * (box_b[3] - box_b[1])
  return inter / (area_a + area_b - inter)

It's the standard because it captures both position and scale error in one number, penalizing a box that's shifted, a box that's the right size in the wrong place, and a box that's the wrong size, the right size in the right place with wrong dimensions, without needing separate metrics for each failure mode.

Semantic segmentation outputs one class label per pixel, full stop. Two overlapping cars in the image both get labeled "car" with no distinction between which pixel belongs to which specific car. Instance segmentation has to additionally separate individual object instances, so it outputs both a class label and an instance identity per pixel, letting you say "these particular pixels are car number one" and "these are car number two" even though both regions share the same class.

That extra requirement is why instance segmentation models, Mask R-CNN being the reference architecture, typically build on top of a detection pipeline first, detect individual object instances as boxes, then predict a mask within each box, rather than trying to solve pixel labeling and instance separation in a single unified step.

Random horizontal flip, random crop, color jitter (brightness, contrast, saturation), and small random rotations are close to defaults for most natural-image detection tasks, they're cheap, they simulate realistic variation the camera would actually encounter, and they don't distort the object's identity. Mosaic augmentation, stitching four training images into one, has become a common addition in modern detectors since it forces the model to learn from smaller, partially occluded, and unusually scaled objects within a single training step.

One I'd avoid without a specific reason: vertical flip on most natural imagery. Flipping a car or a person upside down produces a training example that doesn't resemble anything the deployed model will actually see, unless your domain genuinely includes upside-down objects, satellite or aerial imagery being the obvious exception where "up" isn't a meaningful concept in the first place.

Data augmentation happens during training, generating varied versions of training images so the model sees more diversity than the raw dataset alone provides. Test-time augmentation applies similar transforms, flips, small crops, multiple scales, at inference time, running the model on each augmented version of the same input image and averaging the predictions.

Whether it's worth it depends entirely on the deployment constraint. TTA typically gives a modest accuracy bump, often a fraction of a percentage point to a couple points depending on the task, at the cost of running inference multiple times per image. For a batch offline pipeline processing images overnight, that's usually a fine trade. For a real-time system with a hard latency budget, running the model 4-8x per frame for a small accuracy gain is rarely worth it, and I'd look for that accuracy elsewhere first.

The IoU threshold decides how strictly a predicted box has to overlap with the ground truth to count as a correct detection rather than a miss. At a 0.5 threshold, a box that's noticeably loose, off-center or slightly the wrong size, still counts as correct as long as it clears that 50 percent overlap bar. At 0.75, the predicted box has to be much tighter and more precisely placed to count.

Papers report both, and often an average across several thresholds (mAP at IoU 0.5 to 0.95, in steps of 0.05, is the COCO standard), because a model can look good at the loose 0.5 threshold while its boxes are actually sloppy, and reporting only the loose threshold would hide that. The tighter threshold surfaces localization quality specifically, separate from whether the model found the object at all.

A confusion matrix lays out predicted class against true class in a grid, so each cell shows how many examples of a given true class got predicted as each possible class. The diagonal is correct predictions, everything off the diagonal is a specific kind of mistake, not just "wrong" in the abstract.

Overall accuracy collapses all of that into one number and hides exactly which classes get confused with which. A classifier can have 90 percent overall accuracy while consistently mixing up two visually similar classes, wolf and husky, say, a pattern that's invisible in the accuracy number but obvious the moment you look at the matrix and see one large off-diagonal cell between those two rows.

Pooling downsamples a feature map, max pooling takes the largest value in each small window, average pooling takes the mean, reducing spatial resolution while keeping the strongest activations. That gives some translation invariance (a feature that shifts by a pixel or two still gets picked up) and cuts the compute and memory needed for every layer after it.

Newer architectures increasingly use strided convolutions instead of a separate pooling layer to downsample, since a strided conv learns the downsampling instead of applying a fixed rule, and it's one less hyperparameter (pooling window size) to tune. ResNet still uses one max-pool near the input stem, but the aggressive pooling stacks common in older VGG-style networks are mostly gone from architectures designed in the last several years.

The receptive field is the region of the original input image that influences a given neuron's activation. A single 3x3 conv layer has a receptive field of 3x3 pixels, but stack several of them and the effective receptive field grows, because each layer's neurons already summarize a patch of the previous layer's output.

This matters directly for detection: a network needs a receptive field at least as large as the object it's trying to recognize, or it's making a decision using only part of the object. That's the practical reason detection architectures use feature pyramids, combining shallow layers (small receptive field, good for small objects) with deep layers (large receptive field, good for large objects) rather than picking one depth and hoping it covers every object size in the dataset.

Before ResNet, stacking more layers past a certain depth made accuracy worse, not just slower to train, a result that surprised the field because a deeper network should in theory be able to represent everything a shallower one can plus more. The issue was optimization, not capacity: gradients shrink as they backpropagate through many layers, and a very deep plain network struggles to even learn the identity function that would let it match a shallower one's performance.

class ResidualBlock(nn.Module):
  def forward(self, x):
    out = self.conv2(self.relu(self.conv1(x)))
    return self.relu(out + x) # skip connection adds input back

A skip connection adds the block's input directly to its output, so the block only has to learn the residual, the difference from identity, rather than the full transformation from scratch. If a block turns out to be unnecessary, it can push its learned weights toward zero and let the identity path dominate, instead of having to learn identity the hard way through several nonlinear layers. This let ResNet (He et al., 2015) train networks over 150 layers deep that actually outperformed shallower ones.

A standard convolution mixes spatial information and channel information in one operation. A depthwise separable convolution splits that into two steps: a depthwise convolution that applies one filter per input channel independently (spatial mixing only), followed by a 1x1 pointwise convolution that mixes across channels. The combination approximates a regular convolution using far fewer parameters and multiply-accumulate operations.

depthwise = nn.Conv2d(32, 32, kernel_size=3, groups=32, padding=1) # one filter per channel
pointwise = nn.Conv2d(32, 64, kernel_size=1) # mixes channels

MobileNet popularized this specifically for phone-class hardware, where memory bandwidth and battery cost dominate over raw FLOPs, and depthwise separable convs cut both by roughly 8-9x compared to an equivalent standard convolution at the same channel counts. That's the difference between a model that runs at 30fps on a phone's NPU and one that doesn't run at all without a server round-trip.

Batch norm normalizes each layer's activations across the batch dimension to zero mean and unit variance, then applies a learned scale and shift. This stabilizes training by keeping the input distribution to each layer roughly consistent as the weights in earlier layers change during training, which lets you use higher learning rates and converge faster.

The mean and variance it computes are batch statistics, estimated from whatever examples happen to be in that mini-batch. With a batch size of 2 or 4, common when training on high-resolution medical images that barely fit in GPU memory, those estimates are noisy and unstable, which can actively hurt training rather than help it. That's why detection and segmentation models trained with small batches often switch to group normalization or layer normalization instead, which compute statistics per-example rather than across the batch.

My honest answer depends entirely on the dataset size and the deployment target. With a large labeled dataset, hundreds of thousands of images, or a pretrained CLIP or DINOv2 backbone to fine-tune, a ViT-based model tends to edge out a CNN on accuracy. With a small dataset trained from scratch, CNNs' built-in inductive bias toward locality and translation invariance still tends to win, since a ViT has to learn those properties from data rather than getting them built into the architecture.

For anything deploying to constrained hardware, a phone, an embedded camera, a CNN is still the safer default. The mobile-optimized transformer variants exist, MobileViT and similar, but the tooling, quantization support, and hardware-specific optimizations for standard CNN ops are more mature across the board right now.

An anchor box is a predefined box shape, a fixed width, height, and position relative to a grid cell, that the network predicts an offset from rather than predicting raw box coordinates directly. Instead of asking the network "where exactly is this box," you ask "given this reasonable starting guess, how should it shift and resize."

Predicting raw coordinates from scratch is a much harder regression problem, especially early in training when the network has no prior at all about plausible box shapes. Anchors give the network a sensible starting point, chosen ahead of time based on the aspect ratios and scales common in the training data, usually via k-means clustering over the ground truth boxes, so learning becomes "predict a small correction" instead of "predict an absolute position from nothing."

A detector typically produces many overlapping candidate boxes around the same object, since nearby anchors or grid cells all fire on the same thing to varying degrees. Non-max suppression cleans this up: sort all candidate boxes by confidence score, take the highest-scoring one, then discard every remaining box whose IoU with it exceeds a threshold (commonly 0.5), and repeat with the next highest-scoring box that survived.

def nms(boxes, scores, iou_threshold=0.5):
  order = scores.argsort(descending=True)
  keep = []
  while len(order) > 0:
    i = order[0]
    keep.append(i)
    remaining = order[1:]
    ious = torch.tensor([iou(boxes[i], boxes[j]) for j in remaining])
    order = remaining[ious <= iou_threshold]
  return keep

Without it, the same object shows up as five or ten overlapping boxes in the final output, all technically "correct" detections of the same thing, which is useless for anything downstream that expects one box per object.

A two-stage detector splits the job into two networks run in sequence: a region proposal network first generates a set of candidate regions likely to contain an object, then a second network classifies and refines each of those proposals. A single-stage detector skips the separate proposal step and predicts boxes and class scores directly from a dense grid over the whole image in one forward pass.

Two-stage detectors tend to win on accuracy, particularly for small or crowded objects, because the proposal stage filters out most of the background before the expensive classification step runs on it. Single-stage detectors trade some of that accuracy for speed, since one forward pass is cheaper than two networks run in sequence, which is why YOLO variants dominate anywhere real-time inference matters, video, robotics, live camera feeds, more than raw top-line accuracy does.

The last convolutional layer in a backbone has the strongest semantic features (it's had the most layers to build up abstraction) but the coarsest spatial resolution, since every pooling and stride-2 conv along the way has shrunk the feature map. Running detection on that layer alone means small objects, which might occupy just a few pixels at that resolution, effectively disappear before the detection head ever sees them.

A feature pyramid network combines feature maps from multiple depths, upsampling the coarse, semantically strong deep features and merging them with the finer, spatially precise shallow features, so the detector gets both properties at multiple scales instead of having to pick one depth and accept its tradeoff. This is why FPN-based detectors handle a wide range of object sizes in the same image noticeably better than detectors that predict from a single feature map.

A standard classification CNN downsamples repeatedly and ends with a single low-resolution prediction, fine for "what's in this image" but useless for "which exact pixels belong to this object." U-Net's encoder does the same downsampling to build up semantic features, but a mirrored decoder then upsamples back to the original resolution, producing a full-resolution segmentation mask.

The skip connections between matching encoder and decoder levels are the part that actually makes this work well rather than producing a blurry mask. Upsampling alone can't recover fine spatial detail that was lost during downsampling, a sharp edge that got compressed into one low-resolution value can't be perfectly reconstructed from that value alone. The skip connection feeds the decoder the original, higher-resolution encoder features from before that information was lost, letting the decoder combine coarse semantic context with fine spatial detail at every level, which is why U-Net masks have sharp boundaries instead of soft, blurry ones.

Pixel-wise cross-entropy treats every pixel's prediction as an independent classification problem and averages the loss across all of them. If a tumor in a medical scan occupies 2 percent of the image, a model that just predicts "background" for every single pixel already gets a low average loss, since 98 percent of its predictions were correct by doing nothing useful at all. The loss function doesn't inherently care that the 2 percent it got wrong is the entire point of the task.

Dice loss (or a combination of Dice and cross-entropy) is the standard fix, since Dice directly optimizes for the overlap between predicted and ground truth foreground regions rather than per-pixel accuracy averaged uniformly across an overwhelmingly background image. Weighted cross-entropy, giving the rare foreground class a higher loss weight, is a simpler alternative that also helps, though Dice or a Dice-CE combination tends to be the more reliable default in practice for this exact imbalance problem.

A few hundred labeled masks is genuinely small for segmentation, since per-pixel labeling has far less redundancy in supervision signal per image than a single classification label does. My first move is transfer learning from a backbone pretrained on a large segmentation dataset if one exists in a related domain, or ImageNet pretraining for the encoder at minimum, rather than training an encoder from scratch on a few hundred images.

Second, aggressive but label-preserving augmentation, random crops, flips, rotations, elastic deformation for anything organic like medical or satellite imagery, color jitter, since every geometric transform applied consistently to both the image and its mask effectively multiplies the training set without new labeling work. Third, I'd seriously consider whether the task can be reframed with weaker supervision, bounding boxes instead of full masks, which are far cheaper to label and can bootstrap a decent initial model through something like GrabCut-style refinement, before committing to fully labeling more pixel masks by hand.

ViT splits the image into a grid of fixed-size patches, 16x16 pixels is the original paper's default, flattens each patch into a vector, and passes each flattened patch through a learned linear projection to get a patch embedding. That sequence of patch embeddings, plus a learnable position embedding added to each one so the model knows spatial order, gets fed straight into a standard transformer encoder exactly as if it were a sequence of word tokens.

# conceptual patch embedding, no attention shown
patches = image.unfold(2, 16, 16).unfold(3, 16, 16) # split into 16x16 patches
flattened = patches.reshape(batch, num_patches, -1)  # flatten each patch
embeddings = linear_projection(flattened)       # project to model dimension
embeddings += position_embeddings           # add learned position info

The Vision Transformer paper (Dosovitskiy et al., 2020) showed this works, but it needed pretraining on hundreds of millions of images to match a CNN's performance at smaller data scales, since a plain ViT has none of a CNN's built-in locality or translation-invariance bias and has to learn those properties from data instead.

Self-attention lets every patch attend to every other patch in the image directly, regardless of distance, in a single layer. A convolution's receptive field only grows gradually through depth, a shallow layer only sees a small local neighborhood, and information from opposite corners of the image has to pass through many layers to interact at all.

This matters for images with important long-range relationships, a person's hand relevant to interpreting an object across the frame, or repeated structural patterns in a scene, since a ViT can model that relationship at layer one instead of waiting for the receptive field to grow large enough deep into the network. The cost is quadratic compute in the number of patches, since every patch attends to every other patch, which is why ViTs get expensive fast at high resolution and why a lot of production work still fixes a modest patch grid rather than running attention over every pixel individually.

CLIP trains an image encoder and a text encoder jointly using a contrastive objective: given a batch of image-caption pairs, it pulls the embedding of each image close to its matching caption's embedding and pushes it away from every other caption's embedding in that batch, without ever training on a fixed set of class labels at all.

Because both encoders end up mapping into the same shared embedding space, zero-shot classification becomes a matter of embedding the image once, embedding a set of candidate text prompts like "a photo of a dog" for every class you care about, and picking whichever text embedding is closest to the image embedding. There's no retraining needed to add a new class, you just write a new text prompt for it, which is the property that makes CLIP useful well beyond the specific categories in its original training data.

Yes, and hybrids are common in practice, not just a theoretical middle ground. A typical pattern uses a small CNN stem to do early downsampling and extract local, low-level features cheaply, convolutions are far more compute-efficient at high resolution than attention is, then hands off to transformer blocks once the spatial resolution has already shrunk enough that quadratic attention cost stops being prohibitive.

This tends to win specifically when you need attention's long-range modeling but can't afford full-resolution attention across the whole image, which describes a lot of real production constraints. Pure ViT still wins in settings with enormous pretraining data and no tight latency budget, where the added inductive bias from convolution stops being worth its architectural rigidity.

Before assuming overfitting, I'd check whether the validation set was actually drawn from the same distribution as production traffic, since it's a common setup mistake, splitting a dataset randomly into train and validation gives you a validation set that shares the exact same collection biases, camera, lighting, geography, as training, and neither reflects the true production distribution at all.

A validation set that's a random split of the same curated dataset can look great while a model that's genuinely learned spurious correlations specific to that collection, background clutter that happened to correlate with a class, a particular camera's color profile, sails through it undetected. I'd want a held-out set collected the same way production data actually arrives, ideally collected later in time and from a different source than training, before trusting a validation number as a proxy for real-world performance at all.

Focal loss down-weights the loss contribution from examples the model already classifies confidently and correctly, so training gradient concentrates on the hard, misclassified examples instead of being dominated by a flood of easy ones. It does this with a modulating factor, (1-p)^gamma multiplied into the standard cross-entropy term, where p is the model's predicted probability for the correct class, so a confident correct prediction (p close to 1) contributes almost nothing to the loss.

def focal_loss(p, target, gamma=2.0, alpha=0.25):
  ce = -torch.log(p) if target == 1 else -torch.log(1 - p)
  modulating = (1 - p) ** gamma if target == 1 else p ** gamma
  return alpha * modulating * ce

Standard cross-entropy weights every example's loss equally regardless of how easy it already is, and in single-stage detectors where background examples vastly outnumber foreground objects, that flood of easy, already-correct background predictions dominates the total loss even though they contribute almost nothing useful to learning. Focal loss was introduced specifically to fix this in RetinaNet-style detectors and has become standard well beyond that original use case.

Fifty images is few-shot territory, not enough to train a detector's backbone from scratch reliably. My first move is transfer learning from a strong pretrained detector, freezing most of the backbone and fine-tuning only the detection head and maybe the last backbone stage, since the early and middle layers already encode general visual features that transfer regardless of the specific new class.

Second, I'd lean hard on augmentation to stretch those 50 images further, and consider synthetic data if the object has a 3D model or can be photographed against varied backgrounds cheaply, compositing the object onto diverse backgrounds is a legitimate way to multiply effective training diversity without new real-world collection. Third, I'd set expectations honestly with whoever's asking for this model: 50 images gets you a usable prototype for validating the approach, not a production-grade model, and I'd say so upfront rather than quietly shipping something under-tested.

First, I'd verify the loss function and the metric are actually measuring related things at the value ranges that matter, cross-entropy loss can keep dropping from getting the easy majority-class pixels slightly more confident while barely touching the harder minority-class boundary pixels that IoU is most sensitive to. That's the class imbalance problem showing up as a training-metric mismatch rather than a bug.

Second, I'd check the learning rate schedule and whether the model has actually converged versus is stuck in a low-loss plateau, sometimes a plateau in loss doesn't mean convergence, it means the optimizer has found a flat region that isn't the right minimum for the metric you actually care about. Third, I'd sanity check the IoU computation itself directly against a handful of predictions I inspect visually, since a metric implementation bug, wrong axis for the argmax, a class-index mismatch between prediction and ground truth, produces exactly this symptom and is more common than it should be.

Average precision for one class is the area under that class's precision-recall curve, computed by varying the confidence threshold used to decide which predictions count and measuring precision at each corresponding recall level. Mean average precision then averages that AP value across every class in the dataset.

A single precision or recall number depends entirely on which confidence threshold you happened to pick when you computed it, and that threshold is often an arbitrary operational choice, not an inherent property of the model. Two models could have identical precision at threshold 0.5 and very different precision at threshold 0.9, mAP integrates across the whole range of thresholds instead of committing to one, giving a threshold-independent measure of how well-ranked the model's confidence scores are overall.

Inference latency at the percentile that matters for your use case, p95 or p99 rather than average, since a real-time system's worst-case frame time determines whether it's usable at all, an average that looks fine can hide a tail of slow frames that stall a video pipeline or miss a control loop deadline.

I'd also track calibration, whether a predicted confidence of 0.9 actually corresponds to roughly 90 percent of those predictions being correct, since mAP rewards good ranking of confidence scores but says nothing about whether the raw confidence values are trustworthy on their own. That distinction matters directly for any downstream logic that uses the confidence score as a threshold, an alerting system that only flags detections above 0.8 confidence needs that 0.8 to mean something consistent, not just be well-ranked relative to other predictions.

Domain shift is any mismatch between the distribution the model was trained on and the distribution it encounters in deployment, beyond ordinary sample-to-sample variation within the same distribution. A concrete case I've run into: a defect-detection model trained on images from one camera model, then deployed on a second, cheaper camera with a different sensor and slightly different color response, small enough that a human inspector wouldn't notice a difference glancing at the two feeds side by side, large enough that the model's accuracy measurably dropped.

Lighting is the other one that shows up constantly in physical deployments, a model trained entirely on daytime footage seeing a meaningful accuracy drop at dusk or under artificial lighting, even though nothing about the objects themselves changed. Domain shift is rarely as dramatic as "trained on cats, tested on trucks," it's usually a subtle sensor or environment difference that's easy to miss until the accuracy numbers in production don't match validation.

Adversarial robustness deals with inputs deliberately, often imperceptibly, perturbed by an adversary specifically to fool the model, small pixel-level changes invisible to a human eye that flip a model's prediction with high confidence. Domain-shift robustness deals with naturally occurring distribution differences, no adversary involved, just the world looking a bit different than the training data captured it.

Whether a team needs to worry about adversarial robustness specifically depends on the threat model. A system with a real adversary incentivized to fool it, fraud detection, content moderation bypass attempts, security camera evasion, genuinely needs adversarial-robustness work. A defect-detection system on a factory line has essentially no adversary trying to fool it on purpose, so that team's engineering effort is almost always better spent on domain-shift robustness, more representative training data, better monitoring, than on adversarial defenses that address a threat they don't actually face.

Two 3x3 convs give the same 5x5 effective receptive field but with fewer parameters, 2 x (3x3xC²) versus 5x5xC², and critically, two nonlinear activations instead of one. That extra nonlinearity between the two convs lets the network learn a more expressive function over that receptive field than a single linear-ish 5x5 filter can, even though both see the same input region.

This is the exact reasoning the VGG paper (Simonyan & Zisserman, 2014) used to justify replacing large filters with stacks of small ones, and it's stayed standard practice since. The tradeoff is more sequential operations, which costs a little more memory for intermediate activations during training, and slightly more latency on hardware that isn't well pipelined for narrow, deep chains of small ops.

Before EfficientNet, scaling up a CNN usually meant picking one dimension, more layers (depth), more channels per layer (width), or higher input resolution, and cranking it up in isolation. EfficientNet's authors (Tan & Le, 2019) showed that scaling all three together, using a single compound coefficient that distributes the scaling budget across depth, width, and resolution according to fixed ratios found via a small grid search, produces a better accuracy-per-FLOP tradeoff than scaling any one dimension alone.

The intuition: a higher-resolution input has more pixels to cover, so it benefits from more layers (a larger receptive field to see the whole object) and more channels (capacity to represent finer detail), scaling resolution alone without the other two leaves accuracy on the table. This is the reasoning behind the EfficientNet-B0 through B7 family, each step up scales all three dimensions together rather than independently.

Before touching architecture, I'd check the actual class distribution the model saw during training versus what's true in production traffic, since "balanced dataset" usually means balanced at the image level, not the object level. A dataset with equal numbers of images per class can still have wildly unequal object counts if the rare class tends to appear as one small object per image while common classes appear five or six times per image.

Assuming the distribution really is balanced, the next suspect is the foreground-background imbalance that's inherent to detection, the vast majority of anchor boxes or grid cells in any image are background, not object, and a rare foreground class gets an even weaker gradient signal relative to that flood of easy negative examples. Focal loss exists specifically for this, down-weighting the loss contribution from easy, well-classified examples so the harder, rarer positive examples actually move the gradient. I'd try that before touching anything structural in the model.

Anchor-free detectors, CenterNet and FCOS are common examples, predict objects directly as keypoints, typically a box's center point plus its width and height regressed from that point, instead of assigning ground truth boxes to a fixed set of predefined anchor shapes. This removes an entire category of hyperparameters, anchor scales, aspect ratios, how many anchors per location, that otherwise need tuning per dataset and can silently hurt performance if the anchor shapes don't match the objects actually present.

The tradeoff shows up with overlapping objects of the same class whose centers land close together, since anchor-free methods can struggle to disambiguate two nearby centers the way an anchor-based method's IoU-based assignment naturally handles. In practice the accuracy gap between the two approaches has mostly closed over the last several years, and the anchor-free tuning simplicity is enough of a win that a lot of newer detection research defaults to it.

Mask R-CNN adds a third branch to Faster R-CNN's existing box classification and box regression branches, a small fully convolutional network that predicts a binary mask for each detected region of interest, run in parallel with the existing branches rather than replacing anything.

RoIAlign fixes a quantization problem in the original RoIPooling operation. RoIPooling rounds a region proposal's floating-point coordinates to the nearest integer grid cell when extracting features for that region, a small misalignment for classification (where you just need "is there an object here") but a meaningful one for masks, where pixel-level boundary precision actually matters. RoIAlign uses bilinear interpolation to sample feature values at the exact, non-rounded coordinates instead of snapping to the nearest grid cell, and that precision is a large part of why Mask R-CNN's masks are noticeably sharper than what the original RoIPooling would produce on the same backbone.

A transposed convolution learns to upsample a feature map, it takes a smaller input and produces a larger output, using a learned kernel the same way a regular convolution does, just applied in a way that increases spatial size instead of decreasing it. It's called "deconvolution" informally, but that name is misleading, it isn't mathematically inverting a specific forward convolution the way deconvolution means in signal processing. It's a separate learned operation that happens to move information in the opposite spatial direction.

A known practical issue: transposed convolutions with certain stride and kernel size combinations produce a checkerboard artifact, a regular grid pattern visible in the output, because the kernel overlaps unevenly across output positions. A common workaround is a simple upsampling operation (nearest-neighbor or bilinear) followed by a regular convolution, which avoids the artifact entirely since the upsampling and the learned filtering happen as separate steps instead of intertwined in one operation.

Zero-shot inference uses CLIP's pretrained encoders exactly as-is, no gradient updates at all, just embedding your images and text prompts and comparing similarity. It costs nothing to set up and needs zero labeled data, but accuracy on a specific narrow domain, fine-grained defect types in a manufacturing line, say, tends to trail a model actually trained on that domain's data.

Fine-tuning, or the lighter-weight alternative of linear probing (freezing CLIP's encoder and only training a small classifier head on top of its embeddings), closes that gap once you have even a modest amount of labeled data for your specific task, a few hundred to a few thousand examples is often enough for linear probing to meaningfully beat zero-shot. Full fine-tuning of the whole image encoder needs more data and more compute, and risks catastrophic forgetting of the general visual understanding that made CLIP useful in the first place, so I'd try linear probing before committing to full fine-tuning.

Masked Autoencoders mask out a large fraction of an image's patches, commonly around 75 percent, and train the model to reconstruct the missing pixels from only the small visible fraction that remains. This is a self-supervised objective, it needs no class labels at all, unlike training on ImageNet classification, which requires a human-annotated label for every training image.

The representations that emerge differ in a meaningful way: a classification-pretrained backbone learns features specifically useful for discriminating between the label categories it was trained on, and can overfit subtly to whatever quirks that label set has. A reconstruction-pretrained backbone has to learn enough about an image's overall structure, texture, and spatial layout to fill in three-quarters of it convincingly, which tends to produce features that transfer more broadly across downstream tasks, since nothing about the objective was tied to a specific label taxonomy in the first place.

Mixed precision training runs most operations in float16 (or bfloat16) instead of float32, which roughly halves memory usage and speeds up computation substantially on hardware with dedicated low-precision tensor cores, while keeping certain numerically sensitive operations, the master copy of weights during the optimizer update, in float32 to avoid precision loss there.

scaler = torch.cuda.amp.GradScaler()
with torch.cuda.amp.autocast():
  output = model(input)
  loss = criterion(output, target)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

Loss scaling exists because float16's exponent range is much narrower than float32's, small gradient values that would be fine in float32 can underflow to exactly zero in float16, silently killing the learning signal for those parameters. Scaling the loss up by a large factor before backprop, then unscaling the gradients before the optimizer step, shifts those small values into float16's representable range without changing the underlying math. Skip it and training often appears to run without errors while quietly learning nothing in whichever layers happen to produce the smallest gradients, which is a genuinely hard bug to notice without specifically checking gradient magnitudes.

The most common gap is that the reported precision was computed on a test set with a very different class balance or difficulty distribution than what users actually feed the model. A test set curated with mostly clean, unambiguous examples produces a precision number that doesn't reflect performance on the messier long tail of real inputs, cluttered backgrounds, occluded objects, unusual lighting, that users are actually feeding it.

Second suspect: the confidence threshold used in the deployed system might not match the threshold the reported precision was computed at, someone tuning for high recall in a demo, then shipping that same low threshold to production where users experience it as noisy false positives. Third, if precision is genuinely correct at the operating threshold in use, it's worth checking whether "false positive" from the user's perspective actually means something narrower than the ground truth labels captured, users flagging correct detections of an object they simply didn't want flagged, which isn't a model accuracy problem at all, it's a product scoping problem wearing an accuracy complaint's clothes.

Post-training quantization takes an already-trained float32 model and converts its weights (and sometimes activations) to int8 after the fact, using a small calibration dataset to estimate the value ranges needed for the conversion. It's fast to apply, no retraining needed, but the model wasn't trained with any awareness that it would eventually be quantized, so accuracy can drop meaningfully for architectures sensitive to precision loss.

Quantization-aware training simulates the effect of int8 rounding during the actual training forward pass, so the model's weights adjust to compensate for that quantization noise as part of learning, rather than absorbing it as a surprise after the fact. The gap between the two matters most for smaller, already-efficient architectures, MobileNet-class models that have less redundancy to spare, and for tasks with tight accuracy requirements, medical or safety-critical applications, where even a percentage point of accuracy loss from naive post-training quantization isn't acceptable and the extra training cost of QAT is worth it.

First, export to an inference-optimized runtime, ONNX Runtime or TensorRT depending on the target hardware, which fuses operations and picks hardware-specific optimized kernels that a general-purpose training framework doesn't apply by default. This alone commonly gets a meaningful speedup with zero accuracy cost, since it's purely a deployment format change, not a model change.

If that's not enough, I'd try int8 quantization next, checking accuracy carefully afterward since this is the first step that can actually change the model's output. Only after those two would I consider architectural changes, swapping to a smaller or more efficient backbone, since that requires retraining and risks a real accuracy tradeoff that the previous two steps don't. Reducing input resolution is another lever, cheap to test, and sometimes the accuracy cost is smaller than expected if the objects in frame are large relative to the image.

I'd monitor the distribution of the model's own confidence scores over time as a first, cheap signal, a sustained shift toward lower average confidence, or a growing fraction of predictions near the decision threshold, often shows up before any labeled ground truth is available to confirm accuracy has actually dropped.

Second, I'd track basic image statistics, average brightness, contrast, blur estimates, compared against the training distribution's same statistics, since a lot of real domain shift is exactly this kind of low-level sensor or environment drift and it's cheap to compute on every incoming image without needing any labels at all. Third, wherever feasible, I'd set up a small human-review sampling pipeline, periodically routing a random slice of production predictions to a human for spot-checking, since confidence and image statistics are useful early-warning signals but neither one directly confirms accuracy the way even a small labeled sample does.