Health Checks and Kubernetes Probes Visualization

Health checks in Kubernetes prevent traffic from routing to broken containers and automatically restart dead application instances.

Mental Model

Kubernetes manages container health using a three-probe model representing distinct lifecycle phases. A Startup Probe gates all other checks, protecting slow-booting applications from premature restarts. Once the container is initialized, the Readiness Probe determines if it is ready to receive network traffic; if this probe fails (due to startup lag or missing dependencies), the pod is drained and removed from service endpoints. In parallel, the Liveness Probe checks if the container is deadlocked or permanently wedged; if it fails repeatedly, Kubernetes restarts the container to recover.

Components and Why They Exist

Client

Represents the external users sending requests to the application. Generates network traffic and targets the Kubernetes Service endpoint to consume the application. In the demo, watch it send requests to the service.

Service

Exposes a logical set of pods to clients, abstracting away individual pod instances. Acts as a load balancer, routing incoming requests only to healthy pods listed in the active endpoints. In the demo, watch it balance incoming requests.

Endpoints

Maintains the dynamic list of healthy, ready Pod IP addresses. Serves as the routing registry; pods are added when ready probes pass and removed when ready probes fail. In the demo, watch Pod A be added or removed from Endpoints based on probe outcomes.

Kubelet

The local node agent responsible for managing container lifecycles on that node. Periodically executes configured probes against the container, processes results, and restarts containers when needed. In the demo, watch it execute probe checks.

Startup Probe

Gating mechanism for applications that take a long time to boot or perform initial cache warming. Delays liveness and readiness probes until the container is initialized, preventing premature restarts. In the demo, watch it transition from a gating state to success.

Readiness Probe

Ensures clients do not hit pods that are starting up, shutting down, or missing crucial dependencies. Queries the container health endpoint; if it fails, Kubernetes removes the pod’s IP from the active endpoints list. In the demo, watch it fail and trigger traffic draining when dependencies are down.

Liveness Probe

Automatically recovers containers that are deadlocked, in infinite loops, or permanently unresponsive. Checks container responsiveness; if it fails repeatedly, it commands the container runtime to restart the process. In the demo, watch failure counters accumulate and trigger a restart.

Pod A / Pod B

The actual containerized instances running the application code. Processes requests routed by the Service, hosts the runtime, and responds to Kubelet probes. In the demo, watch Pod A go from initializing to running, crashing, restarting, and recovering.

Dependency

External systems (like databases) required for the application to function. Responds to application queries; outages here will cause readiness probes to fail to prevent broken client requests. In the demo, watch readiness probes fail when this dependency goes down.

Restart Manager

Coordinates container recovery actions when failures occur. Tears down failed container processes and schedules fresh starts while tracking restart history. In the demo, watch it trigger container restarts.

CrashLoopBackOff

Protects the cluster and node from resource starvation during repeated crash cycles. Enforces an exponential backoff delay before each successive restart, limiting CPU/Memory waste. In the demo, watch it enforce backoff delays.

Edge Cases and Misconceptions

• Misconception: Liveness probe failures are the best way to handle dependency outages: If a database is down, restarting the container will not fix it. Liveness probes should only check internal application state. Dependency outages should fail readiness probes to drain traffic, not liveness probes.
• Misconception: Readiness probes protect applications from boot-up crashes: Readiness probes only manage traffic routing. Startup probes are designed to shield container initialization.
• Edge Case: Cascading Failures due to Sudden Drains: If Pod A’s readiness probe fails, it is removed from the endpoints. The remaining pods must handle all the traffic. If the traffic is too heavy, the other pods might fail, leading to a cascading outage.