Deep Learning · Advanced interview

Self-attention lets each token attend to all others to build context-aware representations. Each token is projected into query, key, and value vectors; the attention weight between two tokens is the scaled dot product of one's query with another's key, softmax-normalized, and the output is the weighted sum of values. This lets the model dynamically focus on relevant tokens regardless of distance, and multi-head attention runs several such mechanisms in parallel to capture different relationships.

RNNs process sequences step by step, so they're inherently sequential (hard to parallelize) and struggle with long-range dependencies. Transformers use attention to relate all positions simultaneously, enabling massive parallelism on GPUs and direct modeling of long-range relationships, which scales far better with data and compute. That scalability is what enabled large language models. The cost is attention's quadratic complexity in sequence length.

Self-attention is permutation-invariant — it has no inherent notion of token order — so without position information 'dog bites man' and 'man bites dog' look the same. Positional encodings inject order, either as fixed sinusoidal functions added to embeddings or as learned position embeddings, and modern variants like rotary (RoPE) or relative encodings improve length generalization. They let the model use word order.

Data parallelism replicates the full model on each device, splits the batch across them, and averages gradients (all-reduce) — simple and common when the model fits in one device's memory. Model parallelism splits the model itself across devices (tensor parallelism within layers, pipeline parallelism across layers) when the model is too large to fit. Large-model training combines them (plus sharding optimizer state, e.g. ZeRO/FSDP) to scale across many GPUs.

Mixed precision performs most operations in lower precision (FP16/BF16) while keeping a master copy of weights and sensitive computations in FP32, often with loss scaling to avoid underflow. It roughly halves memory use and speeds up training on hardware with tensor cores, letting you fit bigger models or batches with minimal accuracy loss. BF16 is popular now because its wider exponent range avoids much of the loss-scaling fuss.

Gradient accumulation runs several forward/backward passes on small micro-batches, summing their gradients, and only updates weights after N of them — simulating a large effective batch size without the memory of one big batch. You use it when a desired batch size won't fit in GPU memory, which matters because large models often train better with large effective batches. It trades extra time for reduced memory.

A GAN trains two networks adversarially: a generator creates fake samples and a discriminator tries to distinguish real from fake, each improving against the other until samples look real. Challenges are training instability, mode collapse (the generator produces limited variety), and sensitivity to hyperparameters and balance between the two networks. Techniques like Wasserstein loss, spectral normalization, and careful architectures stabilize them.

A VAE encodes inputs into a probabilistic latent space and decodes samples from it, trained to reconstruct data plus a regularizer (KL term) so the latent space is smooth — generation samples the latent and decodes. Diffusion models gradually add noise to data over many steps, then train a network to reverse that process, denoising from pure noise to a sample; they produce high-quality, diverse images and underpin modern image generators, at the cost of slower multi-step sampling.

An autoregressive model generates a sequence one token at a time, each conditioned on all previously generated tokens — it models the probability of the next token given the prefix. LLMs are decoder transformers trained with this next-token prediction objective on huge corpora. Generation is sequential (so latency grows with length), and sampling strategies (temperature, top-k/top-p) control the diversity of outputs.

Warmup ramps the learning rate from near zero over the first steps to avoid destabilizing the model early (especially with adaptive optimizers and large batches), then cosine (or linear) decay gradually lowers it so training refines toward a minimum. This schedule is standard for transformers — it stabilizes early training and improves final quality. Combined with the right optimizer (e.g. AdamW) and weight decay, it's key to stable large-scale training.

Large models can memorize, so you use weight decay, dropout, large and diverse datasets, data augmentation, and early stopping, plus label smoothing for classification. At scale, the dominant levers are dataset size/quality and the compute-optimal balance of model and data (scaling laws), since an undertrained huge model wastes capacity. Monitoring the train/validation gap guides whether you need more data or more regularization.

Work systematically: verify the data pipeline and labels (overfit a tiny batch to confirm the model can learn at all), check the learning rate (too high diverges, too low stalls), confirm loss/target alignment and correct shapes, inspect for vanishing/exploding gradients and NaNs, ensure inputs are normalized and weights initialized well, and check for bugs like a frozen layer or zeroed gradients. Changing one thing at a time and watching curves isolates the cause.