System Design · Advanced interview

Consensus — getting multiple nodes to agree on a value despite failures and network delays — is hard because messages can be lost, delayed, or reordered, and you can't distinguish a slow node from a dead one. Algorithms like Raft and Paxos provide agreement and a consistent replicated log as long as a majority (quorum) is alive, by electing a leader and committing entries once a majority acknowledges. They underpin things like leader election, configuration stores, and strongly consistent databases.

You generally avoid distributed ACID transactions because two-phase commit is slow and blocks on coordinator failure. The common pattern is a saga: a sequence of local transactions, each with a compensating action to undo it if a later step fails, coordinated via orchestration or events. You accept eventual consistency and design idempotent, reversible steps. The key trade-off is 2PC's strong consistency and locking versus sagas' availability and complexity.

At-most-once may drop messages but never duplicates; at-least-once never drops but may duplicate (the usual default for reliability); exactly-once is the ideal but is very hard across a network. In practice you get exactly-once *effects* by combining at-least-once delivery with idempotent consumers or deduplication, rather than truly delivering once. Designing consumers to tolerate duplicates is the realistic path.

A hotspot is a key that gets disproportionate traffic — a celebrity's profile, a trending item — overloading whatever shard or cache holds it. Mitigations: replicate/fan-out that hot data to many caches or read replicas, add a dedicated cache layer with high TTL, shard the hot key further (e.g. key-splitting for counters), and use request coalescing. The principle is to spread a single hot key's load instead of letting one node own it.

You estimate scale early — requests/sec, data volume per day, storage growth, bandwidth, memory for a cache — to ground the design in reality and reveal what's hard. For example, QPS = daily active users × actions × peak factor / 86400 tells you how many servers and whether one database suffices. It's not about precision but about catching order-of-magnitude problems before you design the wrong thing.

Use expand-and-contract: add the new column/table (expand) while writing to both old and new (dual-write), backfill existing data in batches, switch reads to the new path once verified, then remove the old (contract). Each step is backward-compatible and reversible, and you deploy code and schema changes separately so old and new versions coexist. The goal is no single step that requires a stop-the-world cutover.

Apply isolation and fail-fast patterns: timeouts on every call so a slow dependency doesn't pile up threads, circuit breakers that stop calling a failing dependency and let it recover, bulkheads that isolate resource pools per dependency, retries with exponential backoff and jitter, and graceful degradation/fallbacks. Backpressure and load shedding cap intake. Together these contain a failure to its blast radius instead of letting it spread.

Three pillars: metrics (rates, errors, latency percentiles — the RED/USE methods) for dashboards and alerts; structured logs for detail; and distributed tracing (propagating a trace id across services) to follow a request end-to-end and find where latency or errors originate. You tie alerts to SLOs/error budgets rather than raw thresholds so you alert on user-impacting symptoms. Observability is what makes incidents debuggable when many services interact.

When a popular cached key expires, many concurrent requests miss simultaneously and all hammer the database to recompute it, potentially overwhelming it. Prevent it with request coalescing (a single in-flight recompute, others wait), early/probabilistic recomputation before expiry, locks or single-flight on the recompute, and staggered TTLs so keys don't expire together. The aim is one recompute per expiry, not thousands.

Clarify scope and scale (reads ≫ writes, billions of URLs, low latency). API: POST a long URL → short code, GET code → redirect. Generate codes via a counter base-62 encoded or a hash with collision handling. Store the mapping in a key-value/SQL store, sharded by code; cache hot codes in Redis and front redirects with a CDN. Add analytics asynchronously via a queue. Then discuss trade-offs: code length vs space, custom aliases, consistency of click counts, and handling hotspots.

Ask what a stale or rejected read costs the user. For money, inventory, or anything where a wrong value causes real harm, choose consistency even at the cost of availability. For feeds, counts, recommendations, or social data where brief staleness is invisible, choose availability and reconcile eventually. The decision is per-feature, not system-wide, and you should be able to justify it in business terms, not just technical ones.

Measure under realistic load and find the constrained resource — CPU, memory, I/O, locks, a single database, network — using metrics, profiling, and load tests rather than guessing. Then relieve it: cache, add read replicas, shard, move work async, optimize the hot query, or scale the tier. Bottlenecks move once you fix one, so it's iterative — you re-measure and address the new limiting factor, guided by the actual data.