Computer Vision · Advanced interview

A Vision Transformer splits an image into patches, embeds them as tokens, and applies self-attention so every patch can attend to every other — capturing global relationships from the first layer, unlike CNNs that build up locality through stacked convolutions. ViTs lack CNNs' built-in inductive biases (locality, translation equivariance), so they need more data or strong augmentation/pretraining, but they scale exceptionally well and often surpass CNNs at scale. Hybrids and CNN-like ViTs blend both strengths.

Self-supervised learning pretrains on unlabeled images using pretext tasks, so you don't need expensive labels. Contrastive methods (SimCLR, MoCo) pull augmented views of the same image together and push different images apart; masked image modeling (MAE) reconstructs masked patches; methods like DINO use self-distillation. The learned representations transfer well to downstream tasks with little labeled data, which is valuable since labeling images is costly.

Attention models long-range dependencies that convolutions capture only with depth. It appears in detection/segmentation transformers (DETR frames detection as set prediction with attention, removing anchors and NMS), in hierarchical ViTs (Swin uses windowed attention for efficiency and multi-scale features), and in attention modules added to CNNs (SE blocks, CBAM) to reweight features. The theme is letting the model focus on relevant regions/channels globally.

A GAN pits a generator that synthesizes images against a discriminator that judges real vs fake, training adversarially until generated images are realistic. Failure modes include mode collapse (the generator produces limited variety), training instability and non-convergence, and sensitivity to the generator-discriminator balance. Techniques like Wasserstein loss, spectral normalization, progressive growing, and StyleGAN's architecture improved stability and quality.

Diffusion models define a forward process that gradually adds Gaussian noise to an image over many steps, then train a network to reverse it — predicting and removing noise step by step to turn random noise into a coherent image. They produce state-of-the-art, diverse, high-fidelity images and are easier to train stably than GANs, at the cost of slow multi-step sampling (mitigated by faster samplers and latent-space diffusion). Conditioning (e.g. on text) enables controllable generation.

Image-to-image translation maps an image from one domain to another (sketch→photo, day→night). Paired approaches (pix2pix) train on aligned pairs; unpaired ones (CycleGAN) use a cycle-consistency loss so translating back recovers the original, removing the need for pairs. Neural style transfer separates and recombines content and style by matching feature statistics (Gram matrices) of a content and a style image. Diffusion models now also do this conditionally.

Shrink and speed up the model: quantization (FP32→INT8) cuts size and latency with hardware support; pruning removes redundant weights/channels; knowledge distillation trains a small student to mimic a large teacher; and using efficient architectures (MobileNet, EfficientNet) designed for mobile. You also use optimized runtimes (TensorRT, TFLite, ONNX Runtime, Core ML) and hardware accelerators. The goal is meeting latency/memory/power budgets with acceptable accuracy loss, validated on-device.

Choose a fast architecture (one-stage detectors, efficient backbones), reduce input resolution to the minimum that preserves accuracy, batch where possible, and apply quantization and an optimized runtime/accelerator (GPU/TPU/NPU with TensorRT). Pipeline preprocessing and inference, use frame skipping or tracking between detections in video, and profile to find bottlenecks. Real-time is a budget problem balancing accuracy, resolution, model size, and hardware.

Edge devices have limited compute, memory, and power, no reliable connectivity, and diverse hardware. Challenges include fitting the model (compression), meeting latency/power budgets, handling varied real-world conditions (lighting, occlusion) the training data may not cover, and updating/monitoring models in the field. You address these with model optimization, on-device runtimes, robust data collection, and over-the-air update and monitoring pipelines.

3D vision reconstructs or reasons about the 3D structure of scenes, used in robotics, AR/VR, and autonomous driving. Depth can come from stereo (disparity between two cameras), structure-from-motion (multiple views over time), active sensors (LiDAR, structured light), or monocular depth estimation (a network predicting depth from a single image). Representations include point clouds, voxels, meshes, and neural fields (NeRF) that render novel views. The core challenge is recovering 3D from inherently 2D projections.

Video adds the temporal dimension, so models must capture motion and changes over time, not just spatial appearance. Approaches include 3D convolutions (space+time), two-stream networks (appearance + optical flow), and video transformers attending across frames, plus efficiency tricks since video is heavy. Tasks include action recognition, temporal localization, and tracking. The key challenges are temporal modeling and the large compute/memory cost of many frames.

CLIP trains an image encoder and a text encoder jointly with a contrastive objective on large image-caption pairs, pulling matching image-text pairs close in a shared embedding space and pushing mismatches apart. This shared space enables zero-shot classification (compare an image's embedding to text-label embeddings) and powers text-to-image retrieval and conditioning for generators. It's a foundation for vision-language tasks, captioning, and open-vocabulary recognition.