Machine Learning · Advanced interview

Gradient descent minimizes a loss by iteratively stepping the parameters in the direction of the negative gradient, scaled by a learning rate. Batch uses all data per step (stable, slow), stochastic (SGD) uses one example (noisy, fast), mini-batch balances both. Problems: a learning rate too high diverges and too low crawls; it can get stuck in poor local minima/saddle points or plateaus; and unscaled features distort the landscape — addressed with adaptive optimizers, scaling, and learning-rate schedules.

Match it to the task and what errors you care about. Regression: MSE penalizes large errors heavily (sensitive to outliers), MAE is robust, Huber blends them. Classification: cross-entropy (log loss) for probabilistic outputs, hinge for margins. You may customize it to reflect business costs (e.g. weighting false negatives). The loss defines what 'good' means, so it must encode the real objective, not just a default.

As dimensions grow, data becomes sparse — points are far apart, distance metrics lose meaning, and you need exponentially more data to cover the space, hurting distance-based and density methods. You address it with dimensionality reduction (PCA, which projects onto directions of maximum variance; t-SNE/UMAP for visualization), feature selection, and regularization. Fewer, more informative dimensions improve both performance and generalization.

MLOps covers the full lifecycle: data ingestion and validation, feature engineering (often via a feature store), experiment tracking, training and evaluation, model registry and versioning, deployment, and monitoring with feedback loops for retraining. It applies CI/CD and reproducibility to ML, treating data and models as versioned artifacts. The goal is reliable, repeatable, observable models in production rather than one-off notebooks.

Version everything that affects the result: code (git), data (DVC or dataset snapshots/hashes), features, model artifacts (a model registry), and the environment (containers, pinned dependencies). Fix random seeds, log hyperparameters and metrics with experiment tracking (MLflow/W&B), and capture lineage from data to model. Reproducibility lets you audit, debug, and roll back models — essential when a model misbehaves in production.

A feature store is a centralized system for defining, computing, storing, and serving features consistently for training and inference. It solves training/serving skew (the same feature logic in both places), enables feature reuse across teams/models, and serves features with low latency online plus historical values for training. You use it when features are shared, complex, or need consistent online/offline computation at scale.

Data drift is when the input distribution shifts from training (e.g. new user behavior); concept drift is when the relationship between inputs and target changes, degrading the model even if inputs look similar. You detect drift by monitoring input feature distributions (statistical tests like KS/PSI), prediction distributions, and — when labels arrive — live performance metrics. On drift you alert, investigate, and retrain or recalibrate.

Batch (offline) scoring runs predictions on a schedule over large datasets and stores results — simple, cheap, fine when freshness isn't critical (e.g. nightly churn scores). Online (real-time) serving exposes the model behind an API for low-latency, per-request predictions — needed for interactive features like fraud checks or recommendations. Online adds requirements: latency budgets, autoscaling, feature freshness, and monitoring. You choose based on latency needs and how fresh predictions must be.

Offline metrics don't guarantee business impact, so you roll the new model to a fraction of traffic and compare against the current one on the real KPI (conversion, revenue, engagement), with proper randomization and statistical significance. Techniques include shadow deployment (run it without serving its output) to check behavior safely, then canary/gradual ramp. You watch for guardrail metrics and only fully roll out if the lift is real and stable.

Use inherently interpretable models (linear, trees) when stakes/regulation demand it, or apply post-hoc explanations to black boxes: global methods (feature importance, partial dependence) for overall behavior and local methods (SHAP, LIME) to explain individual predictions. SHAP attributes a prediction to each feature with game-theoretic guarantees. Interpretability supports debugging, trust, fairness audits, and compliance — but explanations themselves must be validated, not blindly trusted.

Bias can enter through skewed data, proxy features for protected attributes, or labels reflecting historical discrimination. You audit with fairness metrics (demographic parity, equalized odds — which can conflict, so you choose based on context), examine data representation, and mitigate via reweighting, constraints, or post-processing. There's an inherent trade-off between some fairness definitions and with accuracy, so it's a stakeholder decision, documented and monitored, not a single technical fix.

Diagnose first. If the model underfits (high train error), a better/larger model or features help. If it overfits (large train-test gap) or errors come from underrepresented cases, more or better-quality data usually helps more — learning curves tell you whether performance is still improving with data. Often data quality, labeling, and coverage beat algorithmic tweaks, so you let the error analysis, not preference, decide.