Debug Pods and Common Failures

Troubleshoot Kubernetes pods on IronSled — read pod status and common states (Pending, ImagePullBackOff, CrashLoopBackOff), fix each failure mode, view and stream logs with kubectl logs (--previous, --tail, --since), exec into containers, use ephemeral kubectl debug containers, inspect events and describe output, check resource usage with kubectl top, verify ConfigMaps/Secrets/volumes, and follow a debug checklist.

When a workload is unhealthy, most answers come from a handful of kubectl commands: check the pod's status, read its events, and read its logs. This page covers the common pod states, how to fix each one, and the tools for digging deeper — logs, exec, ephemeral debug containers, events, and resource usage. Replace <namespace> and <pod-name> with your own values throughout.

Check Pod Status

# List pods with status
kubectl get pods -n <namespace>

# Wide output with node info
kubectl get pods -n <namespace> -o wide

# Watch pods in real time
kubectl get pods -n <namespace> -w

Common pod states and what they usually mean:

State	Description	Typical causes
Pending	Accepted but not scheduled	Resource constraints, node selector issues
ContainerCreating	Scheduled, pulling image	Slow image pull, large images
Running	All containers started	Normal operation
CrashLoopBackOff	Container crashes repeatedly	Application errors, missing config
ImagePullBackOff	Cannot pull the image	Wrong image name, auth issues
Error	Container exited with an error	Application crash
Completed	Container finished successfully	Job / init container completed
Terminating	Pod being deleted	Normal deletion, or stuck finalizers

Debugging by Pod State

Pending

kubectl describe pod <pod-name> -n <namespace>

Look for Insufficient cpu / Insufficient memory, No nodes available, or FailedScheduling. Common fixes: lower resources.requests, correct or remove a nodeSelector, or add a toleration for a node taint (kubectl describe node <node-name> | grep Taint).

ImagePullBackOff

kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Image:"

Missing or wrong pull secret — create a docker-registry secret and reference it via imagePullSecrets (see Kubernetes Secrets).
Wrong image name/tag — verify the full reference registry/repository/image:tag exists and the tag is correct.

CrashLoopBackOff

# Current logs
kubectl logs <pod-name> -n <namespace>

# Logs from the previous (crashed) container
kubectl logs <pod-name> -n <namespace> --previous

Common causes: an application error (read the logs), a failing liveness probe (increase initialDelaySeconds / failureThreshold), or missing dependencies (a ConfigMap, Secret, or backing service that isn't reachable).

Viewing Logs

# Current logs
kubectl logs <pod-name> -n <namespace>

# Follow / stream
kubectl logs -f <pod-name> -n <namespace>

# Last N lines
kubectl logs --tail=100 <pod-name> -n <namespace>

# Since a time window
kubectl logs --since=1h <pod-name> -n <namespace>

# Previous container (after a restart)
kubectl logs --previous <pod-name> -n <namespace>

For multi-container pods, target a container with -c <container-name>, or read them all with --all-containers. For multiple pods, select by label: kubectl logs -l app=my-app -n <namespace>.

Executing Into Pods

# Interactive shell (fall back to /bin/sh if bash is absent)
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

# Run a single command
kubectl exec <pod-name> -n <namespace> -- env
kubectl exec <pod-name> -n <namespace> -- ps aux
kubectl exec <pod-name> -n <namespace> -- nslookup kubernetes.default

Debug Containers

For minimal or distroless images that have no shell, attach an ephemeral debug container that shares the target's process namespace (Kubernetes 1.23+):

# Attach a busybox debug container
kubectl debug -it <pod-name> -n <namespace> \
  --image=busybox --target=<container-name>

# Full network toolkit
kubectl debug -it <pod-name> -n <namespace> \
  --image=nicolaka/netshoot --target=<container-name>

Network Debugging

# Test service connectivity from inside a pod
kubectl exec -it <pod-name> -n <namespace> -- curl http://service-name:port

# Test service DNS
kubectl exec <pod-name> -n <namespace> -- \
  nslookup my-service.my-namespace.svc.cluster.local

# One-off network debug pod
kubectl run netshoot --rm -it --image=nicolaka/netshoot -- /bin/bash

Resource Usage

# Pod usage (requires metrics-server)
kubectl top pod <pod-name> -n <namespace>
kubectl top pods -n <namespace> --sort-by=memory

# Node usage
kubectl top nodes

# Configured requests/limits
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Limits\|Requests"

Events and Describe

# Namespace events, newest last
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Warnings only
kubectl get events -n <namespace> --field-selector type=Warning

# Full pod detail — check Status, Conditions, container state, restart count, Events
kubectl describe pod <pod-name> -n <namespace>

Verifying Configuration

Confirm the ConfigMaps, Secrets, and volumes a pod depends on actually exist and are mounted:

# Does the ConfigMap / Secret exist?
kubectl get configmap <name> -n <namespace>
kubectl get secret <name> -n <namespace>

# Environment variables seen by the container
kubectl exec <pod-name> -n <namespace> -- env

# Mounted volumes and paths
kubectl exec <pod-name> -n <namespace> -- df -h
kubectl exec <pod-name> -n <namespace> -- ls -la /path/to/volume

For a persistent-volume issue, check the claim: kubectl describe pvc <pvc-name> -n <namespace> (a Pending PVC usually means no storage class or provisioner).

Debug Checklist

When a pod isn't working, in order:

Check status — kubectl get pods -n <namespace>
Describe the pod — kubectl describe pod <name> -n <namespace>
Read events — look for warnings and errors
View logs — kubectl logs <name> -n <namespace>
Check previous logs — kubectl logs <name> -n <namespace> --previous
Verify resources — ConfigMaps, Secrets, and PVCs exist
Test connectivity — network, DNS, and services
Check limits — CPU/memory constraints and throttling
Exec into the pod — investigate from the inside
Use a debug container — for minimal images with no shell

For deployment-level context and jumping into Rancher, see Deploy & Troubleshoot.

Debug Pods and Common Failures

On this page