Deep Learning · Intermediate interview

Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network. After the forward pass computes the loss, it propagates the error from the output layer back to the input, multiplying local derivatives at each layer, so each weight gets a gradient indicating how to change it to reduce loss. The optimizer then updates weights using those gradients.

Plain SGD updates weights by the gradient times the learning rate — simple but can be slow and oscillate. Momentum accumulates a velocity from past gradients to accelerate in consistent directions and dampen oscillations. Adam combines momentum with per-parameter adaptive learning rates (using running estimates of gradient mean and variance), making it robust and a common default. SGD with momentum can generalize better with tuning; Adam converges fast with less tuning.

In deep networks, gradients are products of many terms during backprop; if those terms are consistently small they shrink toward zero (vanishing — early layers stop learning), and if large they blow up (exploding — unstable training). Fixes include ReLU activations, careful weight initialization (He/Xavier), batch/layer normalization, residual/skip connections that give gradients a shortcut, gradient clipping (for exploding), and gated units (LSTM/GRU) for sequences.

A convolution slides a small learnable filter over the image computing dot products, detecting local patterns while sharing weights across positions — giving translation invariance and far fewer parameters than a dense layer. Pooling (e.g. max pooling) downsamples feature maps, adding robustness to small shifts and reducing computation. Stacking these builds a hierarchy from edges to textures to object parts, which is why CNNs are efficient and effective for images.

Vanilla RNNs struggle to learn long-range dependencies because gradients vanish over many time steps. LSTMs add a cell state and gates (input, forget, output) that control what information to keep, discard, or expose, letting them carry information across long sequences and mitigating vanishing gradients. GRUs are a simpler variant with fewer gates. They were the standard for sequences before transformers largely replaced them.

An embedding is a learned, dense, low-dimensional vector representation of a discrete item (a word, user, product) where similar items end up close together in the vector space. It replaces sparse one-hot encodings with compact representations that capture relationships, and the embedding matrix is learned during training. Embeddings are fundamental in NLP and recommendation systems.

Dropout randomly deactivates a fraction of neurons each training step, preventing co-adaptation and acting like training an ensemble of subnetworks; at inference all neurons are used (scaled). L2 (weight decay) adds a penalty on squared weights to the loss, keeping weights small and the function smoother. Both discourage the network from fitting noise, and they're often used together along with early stopping.

Batch normalization normalizes a layer's inputs across the mini-batch (zero mean, unit variance) and then scales/shifts with learnable parameters. It stabilizes and speeds training by reducing internal covariate shift, allows higher learning rates, and has a mild regularizing effect. It's standard in deep CNNs; for sequences/transformers, layer normalization is preferred since it doesn't depend on batch statistics.

Data augmentation creates new training examples by applying label-preserving transformations — for images, flips, rotations, crops, color jitter; for text, synonym swaps or back-translation. It effectively enlarges the dataset, exposes the model to variation, and reduces overfitting, improving generalization. It's especially valuable when labeled data is limited.

Transfer learning reuses a model pretrained on a large dataset (e.g. ImageNet) and adapts it to your task, since early layers learn general features. You typically replace the final layer(s) for your classes and either freeze the pretrained layers and train only the head (when you have little data, to avoid overfitting) or fine-tune some/all layers at a low learning rate (when you have more data). It saves compute and data and usually boosts accuracy.

Poor initialization can cause activations and gradients to vanish or explode, making deep networks fail to train. Schemes like Xavier/Glorot (for tanh/sigmoid) and He initialization (for ReLU) set initial weight scales based on layer sizes so signal variance is preserved across layers. Good initialization keeps gradients healthy at the start, which is critical for deep networks to converge.

A learning-rate schedule changes the learning rate during training rather than keeping it fixed — for example, warmup (start small, ramp up) to stabilize early training, then decay (step, exponential, or cosine annealing) to fine-tune as you approach a minimum. It often improves convergence and final accuracy: a higher rate early explores, a lower rate later refines. Adaptive optimizers help but schedules still add value.