Deep Learning · Beginner interview

Deep learning is a subfield of machine learning using neural networks with many layers to learn hierarchical representations directly from raw data. Unlike classical ML, which often needs hand-engineered features, deep networks learn features automatically — which is why they excel at images, audio, and text. The trade-off is they need lots of data and compute and are less interpretable.

An artificial neuron computes a weighted sum of its inputs plus a bias, then applies an activation function to produce an output. A perceptron is the simplest such unit. Stacking many neurons into layers, and layers into a network, lets the model represent complex functions — the weights are what the network learns during training.

An activation function applies a nonlinearity to a neuron's output. Without it, stacking layers would just compose linear functions into one linear function, so the network couldn't model complex, nonlinear patterns. Common ones are ReLU (fast, default for hidden layers), sigmoid and tanh (older, can saturate), and softmax (for output probabilities). Nonlinearity is what gives deep networks their expressive power.

A network has an input layer (the features), one or more hidden layers that transform the data, and an output layer producing the prediction. 'Deep' refers to having many hidden layers, letting the network build increasingly abstract representations — early layers learn simple patterns, later layers combine them. Each layer's outputs feed the next.

A Convolutional Neural Network is designed for grid-like data such as images. Instead of connecting every input to every neuron, it slides small learnable filters (convolutions) over the input to detect local patterns like edges and textures, sharing weights across positions. This makes it parameter-efficient and translation-aware, and stacking convolutions builds up from edges to shapes to objects.

A Recurrent Neural Network processes sequences (text, time series, audio) by maintaining a hidden state that carries information from previous steps as it reads the sequence element by element. This lets it model order and context. Vanilla RNNs struggle with long-range dependencies (vanishing gradients), which led to LSTMs and GRUs, and later to transformers.

The forward pass runs inputs through the network to produce a prediction and computes the loss (how wrong it is). Backpropagation then works backward, using the chain rule to compute how much each weight contributed to the loss (the gradients), so an optimizer can adjust the weights to reduce it. Together, forward + backward + update is one training step.

The loss function measures the gap between predictions and targets. Gradient descent uses the gradients from backpropagation to nudge each weight in the direction that reduces the loss, repeating over many examples and iterations until the loss stops improving. The learning rate controls the step size. Training is essentially minimizing the loss by adjusting weights.

An epoch is one full pass through the training data. Batch size is how many examples are processed before updating weights (mini-batches balance speed and stability). Learning rate is the step size for weight updates — too high and training diverges, too low and it's slow or stuck. These are key hyperparameters you tune for stable, efficient training.

Overfitting appears as training loss continuing to drop while validation loss rises — the network memorizes training data instead of generalizing. Dropout combats it by randomly 'turning off' a fraction of neurons during each training step, forcing the network not to rely on any single unit and effectively training an ensemble. Other tools are more data, augmentation, weight decay, and early stopping.

Neural network training is dominated by large matrix multiplications, which are highly parallel. GPUs have thousands of cores optimized for exactly this kind of parallel math, so they train networks orders of magnitude faster than CPUs. This is why deep learning took off alongside GPU computing, with specialized accelerators (TPUs) extending the idea.

PyTorch and TensorFlow are the two dominant frameworks. PyTorch is favored in research and increasingly industry for its intuitive, Pythonic, dynamic-graph style; TensorFlow (with Keras) is widely used in production with strong deployment tooling. Both provide automatic differentiation, GPU support, and prebuilt layers, so you define the architecture and they handle gradients and hardware.