Computer Vision · Intermediate interview

As networks got deeper, accuracy degraded due to vanishing gradients and optimization difficulty. ResNet's skip connections add a layer's input to its output (learning a residual), giving gradients a shortcut path during backprop and making it easy for layers to learn identity if needed. This enabled training networks hundreds of layers deep, a breakthrough that's now standard in most architectures.

Depth lets a network build more abstract, hierarchical features, improving representational power. But naively deeper networks suffer vanishing/exploding gradients, harder optimization, and more overfitting and compute. Innovations that make depth trainable include better initialization, batch normalization, and residual connections. So depth helps when paired with these techniques; otherwise it hurts.

The receptive field is the region of the input that influences a given output unit — it grows with depth, pooling, and stride, so deeper neurons 'see' more context, which matters for tasks needing global understanding. A 1×1 convolution operates across channels at each pixel, used to change channel depth (dimensionality reduction, as in bottleneck blocks) and mix channel information cheaply. Both are key tools in designing efficient architectures.

Two-stage detectors (e.g. Faster R-CNN) first propose candidate regions, then classify and refine them — typically more accurate but slower. One-stage detectors (e.g. YOLO, SSD) predict boxes and classes directly over a grid in a single pass — faster and suited to real-time, historically with a small accuracy trade-off that has narrowed. You choose based on the accuracy-vs-speed needs of the application.

Anchor boxes are predefined box shapes/sizes the detector predicts offsets from, helping handle objects of varying scale and aspect ratio. IoU (Intersection over Union) measures overlap between two boxes, used to match predictions to ground truth and to define correctness. Non-max suppression removes duplicate overlapping detections of the same object by keeping the highest-confidence box and discarding others with high IoU to it. Together they turn raw predictions into clean detections.

mAP (mean Average Precision) summarizes detection quality. For each class, predictions are ranked by confidence and matched to ground truth via an IoU threshold, producing a precision-recall curve whose area is the Average Precision; mAP averages AP across classes (and often across IoU thresholds, e.g. mAP@[.5:.95]). It rewards both finding objects (recall) and being precise (few false positives) at the right locations.

Semantic segmentation labels every pixel with a class but doesn't distinguish individual objects — all cars are 'car'. Instance segmentation labels pixels and separates each object instance — car 1 vs car 2 — combining detection and per-object masks (e.g. Mask R-CNN). Panoptic segmentation unifies both (stuff + things). You choose based on whether counting/separating individual objects matters.

An encoder progressively downsamples the image to capture context (what's present), and a decoder upsamples back to full resolution to produce a per-pixel mask (where). U-Net adds skip connections from encoder to decoder at matching resolutions, restoring fine spatial detail lost during downsampling. This encoder-decoder-with-skips design is the workhorse for segmentation, especially in medical imaging.

IoU (Jaccard) is the intersection over union of predicted and ground-truth masks; mean IoU averages it across classes — the standard segmentation metric. The Dice coefficient (F1 for masks) is 2×intersection / (sum of sizes), more forgiving and popular in medical imaging, often used directly as a loss for imbalanced foreground. Both measure overlap quality per pixel, complementing pixel accuracy which misleads when classes are imbalanced.

Start from a model pretrained on a large dataset (ImageNet, or self-supervised backbones), replace the task-specific head, and either freeze the backbone and train the head (small data) or fine-tune the whole network at a low learning rate (more data). Match preprocessing/normalization to the pretrained model. For detection/segmentation you reuse a pretrained backbone within the larger architecture. It saves data and compute and usually boosts accuracy.

Geometric (flip, rotate, crop, scale, translate), photometric (brightness, contrast, hue, noise), and advanced ones (Cutout, Mixup, CutMix) that mix images/labels. They improve robustness and reduce overfitting. Caution: augmentations must be label-preserving and domain-appropriate — vertical flips break text or some medical orientations, and unrealistic transforms can hurt. You choose augmentations reflecting real-world variation the model will face.

Use transfer learning (pretrained backbones need far less data), heavy but realistic augmentation, and possibly synthetic data or self-supervised pretraining on unlabeled images. For class imbalance, use class weights or focal loss, oversample rare classes, and evaluate with per-class metrics and mIoU rather than accuracy. Collecting more labeled examples of rare cases, when feasible, often helps most.