Debugging CrashLoopBackOff in Kubernetes: A Step-by-Step Guide

Few sights raise a developer’s heart rate faster than CrashLoopBackOff. The pod name is right there, the image pulled fine — and then kubectl get pods shows the dreaded status while the restart counter ticks upward. Here’s the good news: CrashLoopBackOff is a symptom, and it always has a real cause hiding one or two commands away.

What CrashLoopBackOff actually means

The status CrashLoopBackOff says: the container exited (or failed probes) repeatedly, and Kubernetes is now backing off between restart attempts — 10s, 20s, 40s, capped at 5 minutes. The kubelet is doing its job; the container itself is failing.

The status field tells you where in the lifecycle it’s failing:

Status	Meaning
CrashLoopBackOff	Container starts, then exits/crashes repeatedly
Error	Container failed, backoff timer in effect
ImagePullBackOff	Different problem — image can’t be pulled
CreateContainerConfigError	Bad ConfigMap/Secret reference
Pending	Can’t even be scheduled — node/resources issue

CrashLoopBackOff specifically means the container started — so your image is pullable and your command is at least valid. Something at runtime is killing it.

Step 1: Read what Kubernetes already told you

kubectl describe pod my-app-7d9f8b4c5-x2k4l

Scroll to Events. This often reveals the cause outright:

Warning  Unhealthy  Liveness probe failed: HTTP probe failed with statuscode: 500
Normal   Killing    Container failed liveness probe, will be restarted

→ The container isn’t crashing on its own — your liveness probe is killing it. Fix the probe (or the endpoint it hits), not the container.

OOMKilled

→ The process exceeded its memory limit. Raise resources.limits.memory or fix the leak. Note: OOMKilled with exit code 137 is the signature.

Also check Last State under Containers in the same output:

Last State: Terminated
  Reason: Error
  Exit Code: 1        ← the application's own exit code

The exit code narrows the search:

• 1 — application error (config, missing file, unhandled exception).
• 126/127 — command or binary not executable/found (wrong ENTRYPOINT/command).
• 134/137 — SIGKILL (OOM or probe kill).
• 139 — segfault.

Step 2: Read the logs

First the current attempt:

kubectl logs my-app-7d9f8b4c5-x2k4l --previous

--previous is the crucial flag for crash loops — it returns logs from the last terminated container, which is where the fatal error lives. Without it you often get an empty log or only the new attempt’s startup lines.

If logs are empty, the process is dying before it can log — suspect entrypoint issues, a bad shebang, or immediate OOM.

Step 3: Compare with a working baseline

kubectl get pods -l app=my-app        # are ALL replicas crashing?
kubectl get events --sort-by=.lastTimestamp | head -30

• All replicas crashing → image, manifest, or config problem (bad ConfigMap value, missing Secret key, wrong env var).
• Some replicas crash → node-specific issue (node pressure, failed volume mount, tainted node) or a probe race.

Step 4: Reproduce interactively

When logs and events aren’t enough, run the same image with a command that won’t exit:

kubectl run debug --rm -it --image=my-app:2.1.0 -- /bin/sh

Inside, run the application command manually with full output. Or exec into a crash-looping pod’s previous container is impossible — instead override the command on a new pod to sleep, then kubectl exec in and try starting the app by hand.

Common discoveries at this stage:

# Config that looks fine in YAML but wrong at runtime
echo $DATABASE_URL
ls /etc/secrets/
cat /app/config.yaml

Step 5: The usual suspects checklist

Work through these in rough order of frequency:

1. Application config errors — bad value in a ConfigMap/Secret, wrong env var name, missing required setting. Exit code 1 with a stack trace in --previous logs.
2. Liveness probe too aggressive — probe endpoint depends on an external service, or fires before the app is ready (you need a startupProbe). The events in Step 1 show it.
3. OOMKilled — memory limit too low. Check kubectl describe pod → Last State: Reason: OOMKilled.
4. Wrong command/args — command in the manifest overriding ENTRYPOINT incorrectly; exit 127 = binary not found.
5. Missing files at runtime — ConfigMap volume not mounted where the app reads it, wrong subPath.
6. Readiness ≠ liveness confusion — the app is unready (fine) but liveness restarts it (not fine).
7. Initialization that must be manual — DB migration that failed; the app exits because a table doesn’t exist.

Step 6: Verify the fix

kubectl rollout restart deployment my-app
kubectl rollout status deployment my-app
kubectl get pods -w    # watch RESTARTS stop incrementing and READY stay 1/1

A healthy pod shows READY 1/1, STATUS Running, and — importantly — RESTARTS that doesn’t climb.

Prevention

• Add a startup probe to any app with slow initialization; it stops liveness from killing warm-up.
• Set memory requests and limits deliberately so OOM kills are predictable, not mysterious.
• Fail fast and log loudly on startup errors — a clear FATAL: missing env DATABASE_URL turns a 30-minute investigation into a 30-second one.
• Don’t use liveness for dependency checks — readiness is where external dependencies belong.

Quick reference workflow

kubectl describe pod <pod>                 # events + exit code + last state
kubectl logs <pod> --previous              # the crash's actual output
kubectl get pods -l app=<app>              # scope: all replicas or one node?
kubectl run debug --rm -it --image=<img> -- /bin/sh   # reproduce by hand

CrashLoopBackOff always tells you that something is wrong; describe and logs --previous tell you what. For more patterns beyond crashing, see the Kubernetes troubleshooting docs — but nine times out of ten, these two commands are the whole answer.