Machine Learning · Intermediate interview

Bias is error from overly simplistic assumptions (underfitting); variance is error from sensitivity to the training data's noise (overfitting). Increasing model complexity lowers bias but raises variance, and vice versa, so total error is minimized at a balance. The goal is a model complex enough to capture the signal but not so complex it fits noise — you diagnose where you are via the train/test error gap.

Both add a penalty on weight magnitudes to the loss to discourage overfitting. L2 (Ridge) penalizes squared weights, shrinking them smoothly toward (but not to) zero. L1 (Lasso) penalizes absolute weights and drives some exactly to zero, performing feature selection. You use L2 for general shrinkage and L1 when you want a sparse model that ignores irrelevant features; elastic net combines both.

k-fold splits data into k parts, training on k−1 and validating on the rest, rotating and averaging — giving a robust performance estimate and using all data for both training and validation. Stratified k-fold preserves class proportions in each fold (important for imbalanced data), and time-series CV respects temporal order (no training on the future). It's the backbone of reliable model selection.

Precision is, of the items predicted positive, how many actually are (TP / (TP+FP)) — it penalizes false alarms. Recall is, of all actual positives, how many you found (TP / (TP+FN)) — it penalizes misses. F1 is their harmonic mean, balancing the two. You optimize precision when false positives are costly (spam filtering) and recall when misses are costly (disease screening).

The ROC curve plots true-positive rate against false-positive rate across all classification thresholds, showing the trade-off as you move the decision boundary. AUC (area under it) summarizes this as a single number — the probability the model ranks a random positive above a random negative — where 0.5 is random and 1.0 is perfect. AUC is threshold-independent and useful for comparing models, though PR curves are better for heavy imbalance.

First, use the right metrics (precision/recall/F1, PR-AUC) instead of accuracy. Then techniques: resampling (oversample the minority, e.g. SMOTE, or undersample the majority), class weights to penalize minority errors more, adjusting the decision threshold, and choosing algorithms robust to imbalance. The best choice depends on data size and the cost of each error type — and you evaluate on the original distribution.

Feature engineering creates informative inputs from raw data — transformations, aggregations, interactions, date parts, domain-derived signals — often the highest-leverage work in ML. Feature selection removes irrelevant or redundant features to reduce overfitting, training time, and complexity, via filter (correlation), wrapper (recursive elimination), or embedded (L1, tree importance) methods. Good features can make a simple model beat a complex one on raw data.

One-hot encoding creates a binary column per category — simple and safe for low-cardinality features but explosive for high-cardinality ones. Label/ordinal encoding maps categories to integers (only valid when there's a real order). Target/mean encoding replaces a category with a statistic of the target, powerful for high cardinality but prone to leakage if not done within CV folds. You pick based on cardinality and whether order is meaningful.

Data leakage is when information that wouldn't be available at prediction time sneaks into training, inflating offline performance but failing in production — e.g. scaling or imputing using the whole dataset before splitting, or using a feature derived from the target. You prevent it by fitting all preprocessing on the training fold only (inside a pipeline/CV), and scrutinizing features for anything that encodes the future or the label.

Both are ensembles that combine many weak learners. Bagging (e.g. Random Forest) trains models in parallel on bootstrapped samples and averages them, reducing variance. Boosting (e.g. Gradient Boosting, XGBoost) trains models sequentially, each correcting the previous one's errors, reducing bias and often achieving higher accuracy but more prone to overfitting and slower. Bagging for variance reduction and robustness; boosting for accuracy with careful tuning.

At each node it picks the feature and threshold that best separates the data according to an impurity measure — Gini impurity or entropy/information gain for classification, variance reduction for regression — choosing the split that most reduces impurity. It recurses until a stopping criterion (max depth, min samples). Left unconstrained, trees overfit, which is why we limit depth or use ensembles.

Hyperparameters (set before training, like tree depth or learning rate) are tuned by searching combinations and evaluating with cross-validation. Grid search tries all combinations (exhaustive but expensive); random search samples them (often more efficient); Bayesian optimization uses past results to choose promising settings. You define a sensible search space, use CV to avoid overfitting the validation set, and keep the test set untouched for final evaluation.