Kubernetes · Advanced interview

Kubernetes is built on controllers that continuously compare desired state (in etcd, via the API server) to actual state and take actions to converge them — a level-triggered control loop, not edge-triggered events. For example, the Deployment controller notices replicas don't match and creates/deletes Pods. This declarative, self-correcting model is why the cluster heals drift and why you describe what you want, not the steps.

When a Pod is unscheduled, the scheduler filters nodes that can run it (enough requested resources, matching node selectors/affinity, tolerating taints, satisfying topology constraints) then scores the survivors (spreading, resource balance, affinity preferences) and binds the Pod to the best node. The kubelet on that node then starts it. Requests, affinities, and taints are your levers over this decision.

A CustomResourceDefinition extends the Kubernetes API with your own resource type, and an Operator is a custom controller that reconciles that resource — encoding operational knowledge to manage complex apps (e.g. provisioning and failing over a database) the same declarative way Kubernetes manages built-ins. You write an operator when an app needs domain-specific automation beyond what Deployments/StatefulSets provide.

Kubernetes requires a flat network where every Pod gets its own IP and can reach any other Pod without NAT. A CNI plugin (Calico, Cilium, etc.) implements this — assigning IPs and wiring up routes/overlays between nodes. This model means containers in a Pod share an IP, and Services sit on top to provide stable addressing and load balancing over the Pod IPs.

kube-proxy programs each node's networking (via iptables or IPVS, or is replaced by eBPF in Cilium) so that traffic to a Service's virtual ClusterIP is load-balanced to the current healthy backend Pod IPs. It watches the API for Service and Endpoint changes and updates the rules. This is what makes a Service IP route to whatever Pods exist, transparently to clients.

By default all Pods can talk to each other; a NetworkPolicy restricts ingress/egress by labels and namespaces, letting you implement default-deny and least-privilege traffic (enforced by the CNI). A service mesh (Istio, Linkerd) adds a sidecar layer for mTLS encryption, fine-grained traffic management (retries, canaries), and observability without app changes. You add a mesh when you need those cross-cutting concerns uniformly across services.

Grant the minimum verbs on the minimum resources in the minimum scope: prefer namespaced Roles over ClusterRoles, bind to specific service accounts rather than users, give each workload its own service account with only what it needs, and avoid wildcards and cluster-admin. Audit bindings regularly. The goal is that a compromised workload or token can do as little as possible.

Set a securityContext to run as non-root, drop Linux capabilities, use a read-only root filesystem, and disallow privilege escalation. Enforce these cluster-wide with Pod Security Admission (the baseline/restricted standards) or a policy engine like OPA/Gatekeeper or Kyverno, which reject non-compliant Pods at admission. This stops insecure workloads from being deployed in the first place.

Native Secrets are only base64-encoded, so enable encryption at rest for etcd, restrict access with RBAC, and avoid exposing them in logs or env where practical. For stronger security, integrate an external secrets manager (Vault, cloud secret stores) via the External Secrets Operator or CSI driver, which injects secrets at runtime and supports rotation. Never commit secrets to Git; use sealed/encrypted secrets for GitOps.

HPA scales the number of Pods based on load; VPA adjusts a Pod's CPU/memory requests to right-size it; the Cluster Autoscaler adds or removes nodes when Pods can't be scheduled or nodes are underused. They operate at different levels — replicas, per-Pod resources, and node count — and you combine HPA with the Cluster Autoscaler so scaling out Pods also provisions the nodes to run them (HPA and VPA on the same metric can conflict).

Use rolling updates gated by readiness probes, set PodDisruptionBudgets so voluntary disruptions (node drains, upgrades) never take more replicas down than allowed, and add preStop hooks plus graceful termination so in-flight requests finish before a Pod dies. For node maintenance, kubectl drain respects PDBs and reschedules Pods elsewhere first. Together these keep the service available during deploys and cluster operations.

Start with kubectl describe pod to see events and the last state/exit code, and kubectl logs --previous to see why the crashed container exited. Common causes: a failing liveness probe killing it, a bad config/missing secret or env, the app crashing on startup, OOMKill (check limits), or an image/permission error. You fix the root cause — the back-off is just Kubernetes restarting it with increasing delay — and may temporarily adjust the probe or command to get a shell for inspection.