Containers
Kubernetes Troubleshooting — Pending, CrashLoopBackOff, and friends
Diagnose stuck and crashing pods and broken Services: confirm context and namespace, read events, pull logs, inspect workload state, and restore Service routing.
Quick reference — reach for these first
kubectl describe pod POD -n NSContainer state + Events at the bottom — start herekubectl logs POD -n NS --previousStdout/stderr from the container's last (crashed) runkubectl get events -n NS --sort-by=.lastTimestampScheduler and kubelet reasons, newest lastkubectl get endpoints SVC -n NSEmpty means the selector matches no ready podskubectl run dnstest --rm -it --image=busybox:1.36 --restart=Never -n NS -- nslookup SVCResolve a Service name from inside the clusterTroubleshooting path: context first, recovery last
Text equivalent: A left-to-right flow of five stages. Stage 1, Context + namespace: kubectl config current-context and set the namespace. Arrow to stage 2, Events: kubectl get events and describe pod. Arrow to stage 3, Logs: kubectl logs with --previous. Arrow to stage 4, Workload state: inspect Pending, CrashLoopBackOff, ImagePullBackOff, and probe failures. Arrow to stage 5, Service recovery: check endpoints, fix the selector or label mismatch, then roll the deployment. A note under the first box reads: confirm the target before any change.
Common tasks
Task · Command · Confirm
| Task | Command | Confirm |
|---|---|---|
| Confirm which cluster you are pointed at before touching anything | kubectl config current-context | Prints the cluster you expect (e.g. minikube, not a prod context) |
| Confirm the active namespace | kubectl config view --minify -o jsonpath='{..namespace}'; echo | Prints NS; empty output means the 'default' namespace |
| Pin the namespace so later commands target it | kubectl config set-context --current --namespace=NS | kubectl config view --minify | grep namespace shows NS |
| Snapshot pod health in the namespace | kubectl get pods -n NS -o wide | STATUS and RESTARTS columns reveal which pods are unhealthy and on which node |
| Pending pod: read the scheduler's reason | kubectl describe pod POD -n NS | sed -n '/Events/,$p' | FailedScheduling line names the blocker: Insufficient cpu/memory, a taint, or an unbound PVC |
| Pending pod: check node capacity and taints | kubectl describe nodes | grep -E 'Taints|Allocated resources' -A6 | Confirms whether any node has room / tolerable taints for the pod |
| Pending pod: check for an unbound PVC | kubectl get pvc -n NS | A claim stuck in Pending blocks the pod until a PV or StorageClass satisfies it |
| CrashLoopBackOff: read the crash output from the last run | kubectl logs POD -n NS --previous --tail=50 | The exit message or stack trace from before the restart is visible |
| CrashLoopBackOff: read the exit code and reason | kubectl get pod POD -n NS -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{" "}{.status.containerStatuses[0].lastState.terminated.reason}'; echo | Exit code is non-zero (137=OOMKilled, 1/2=app error, 127=bad command) |
| ImagePullBackOff/ErrImagePull: see the pull error | kubectl describe pod POD -n NS | grep -A3 -iE 'Failed|ErrImagePull' | Message shows the cause: 'manifest unknown' (bad tag) or '401 Unauthorized' (auth) |
| ImagePullBackOff: verify the imagePullSecret exists and is referenced | kubectl get pod POD -n NS -o jsonpath='{.spec.imagePullSecrets}'; echo; kubectl get secrets -n NS | A referenced secret of type kubernetes.io/dockerconfigjson is present for private registries |
| Probe failures: find why a Running pod is not Ready | kubectl describe pod POD -n NS | grep -iE 'liveness|readiness|unhealthy' | Unhealthy events show the probe failure: 'HTTP probe failed with statuscode: 500' or 'connection refused' |
| Missing endpoints: compare the Service selector against pod labels | kubectl get endpoints SVC -n NS; kubectl get svc SVC -n NS -o jsonpath='{.spec.selector}'; echo; kubectl get pods -n NS --show-labels | Selector key/values match at least one Ready pod's labels; mismatch = empty ENDPOINTS |
| [CHANGES WORKLOAD] After the fix, roll the deployment and watch it settle | kubectl rollout restart deploy/DEPLOY -n NS && kubectl rollout status deploy/DEPLOY -n NS | rollout status reports 'successfully rolled out'; new pods reach Running/Ready |
A Service returns no response — empty endpoints traced to a label mismatch
endpoints
kubectl config current-context
# minikubekubectl get endpoints web -n shop
# NAME ENDPOINTS AGE
# web <none> 10mkubectl get svc web -n shop -o jsonpath='{.spec.selector}'; echo
# {"app":"web"}kubectl get pods -n shop --show-labels
# NAME READY STATUS LABELS
# web-7d9f... 1/1 Running app=frontend,pod-template-hash=7d9fkubectl patch svc web -n shop -p '{"spec":{"selector":{"app":"frontend"}}}'
# service/web patchedkubectl get endpoints web -n shop
# NAME ENDPOINTS AGE
# web 10.1.2.3:8080 12mTroubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| Pod stuck in Pending, never schedules | kubectl describe pod POD -n NS (read Events) | Whether the scheduler rejected it for resources, a taint, node selector, or an unbound PVC | kubectl describe nodes (Allocated resources + Taints) and kubectl get pvc -n NS |
| CrashLoopBackOff — container restarts in a loop | kubectl logs POD -n NS --previous | The app exits on startup; the previous-run log holds the actual error | Check the terminated exitCode (137=OOM, 127=bad cmd) and the container's env/config/mounts |
| ImagePullBackOff / ErrImagePull | kubectl describe pod POD -n NS (grep Failed) | Image name/tag is wrong, registry is unreachable, or auth failed | Fix the tag, or verify the imagePullSecret and registry credentials |
| Pod Running but never Ready; traffic not routed | kubectl describe pod POD -n NS (liveness/readiness events) | The readiness probe is failing, so the pod is excluded from endpoints | Exec in and curl the probe path/port; fix the path or loosen initialDelay/timeout thresholds |
| Service reachable but returns nothing; curl hangs or refuses | kubectl get endpoints SVC -n NS | <none> endpoints = selector matches no Ready pod (label mismatch) or all pods unready | Compare spec.selector to pod labels (kubectl get pods --show-labels); align them |
| In-cluster name resolution fails (service DNS not found) | kubectl run dnstest --rm -it --image=busybox:1.36 --restart=Never -n NS -- nslookup SVC | Whether DNS itself is broken vs the app's own networking | kubectl get pods -n kube-system -l k8s-app=kube-dns and kubectl logs for CoreDNS errors |
References
- Debug Podskubernetes.io
- Debug Running Podskubernetes.io
- Determine the Reason for Pod Failurekubernetes.io
- Debug Serviceskubernetes.io
- Debugging DNS Resolutionkubernetes.io
kubectl v1.31, any conformant cluster (minikube/EKS/GKE), bash shell; replace CAPS placeholders.