Kubernetes Troubleshooting Explained from Scratch
- sathyahraj
- 5 days ago
- 4 min read
Introduction
Kubernetes is designed to be self-healing, but that doesn't mean problems never occur.
Applications may fail to start, Pods may keep restarting, nodes can become unavailable, or users may not be able to access services.
The good news? Kubernetes provides powerful tools to diagnose almost every issue.
Think of Kubernetes troubleshooting like a doctor diagnosing a patient:
Check the symptoms.
Identify the root cause.
Apply the correct treatment.
Verify recovery.
The Kubernetes Troubleshooting Workflow
Always follow this order:
Is the cluster healthy?
Is the node healthy?
Is the Pod running?
Is the container healthy?
Is the application reachable?
Are storage and networking working?
Check logs and events.
Fix the root cause and verify.
This structured approach prevents random guessing.
Step 1: Check Cluster Health
The cluster is the foundation. If it's unhealthy, everything else can be affected.
Common issues:
API Server unavailable
Control Plane down
etcd issues
Scheduler not responding
Useful commands:
kubectl cluster-info
kubectl get componentstatuses
kubectl get nodes
Step 2: Check Node Status
Pods run on worker nodes.
If a node is unhealthy, workloads may stop working.
Check:
kubectl get nodes
Example:
NAME STATUS
worker-01 Ready
worker-02 NotReady
worker-03 Ready
A NotReady node indicates Kubernetes cannot schedule or maintain workloads there.
Possible causes:
High CPU or memory usage
Disk full
Network failure
Kubelet stopped
Hardware failure
Step 3: Check Pod Status
Pods are where applications run.
Command:
kubectl get pods
Common Pod states:
Status
Meaning
Running
Healthy
Pending
Waiting for resources
ContainerCreating
Starting
CrashLoopBackOff
Container repeatedly crashes
Error
Application failure
Completed
Job finished
ImagePullBackOff
Image download failed
Terminating
Pod shutting down
Understanding CrashLoopBackOff
This is one of the most common Kubernetes issues.
It means:
Container starts.
Application crashes.
Kubernetes restarts it.
It crashes again.
Common causes:
Application bugs
Wrong configuration
Missing environment variables
Database unavailable
Port conflicts
Memory exhaustion
Check:
kubectl describe pod POD_NAME
kubectl logs POD_NAME
Understanding Pending Pods
A Pod stays in Pending when Kubernetes cannot schedule it.
Reasons include:
No available CPU
Insufficient memory
Storage unavailable
Node selector mismatch
Taints and tolerations
Resource quotas exceeded
Use:
kubectl describe pod POD_NAME
The Events section usually explains why.
Understanding ImagePullBackOff
Kubernetes cannot download the container image.
Possible reasons:
Wrong image name
Incorrect tag
Private registry authentication failure
Registry unavailable
Internet or DNS issue
Example:
Failed to pull image
Verify:
Image exists
Registry credentials
Internet connectivity
Step 4: Check Pod Details
The describe command is one of the most useful troubleshooting tools.
kubectl describe pod POD_NAME
It shows:
Events
Scheduling
Volumes
Environment variables
Resource requests
Mounts
Containers
Health checks
Always check the Events section first.
Step 5: Check Application Logs
Logs reveal what the application is doing.
Command:
kubectl logs POD_NAME
Multiple containers:
kubectl logs POD_NAME -c CONTAINER_NAME
Common errors:
Database connection failed
Authentication failure
Configuration missing
File not found
Out of Memory
Step 6: Check Events
Events explain what Kubernetes is trying to do.
kubectl get events
Examples:
Failed scheduling
Volume mount failure
Image pull failure
Node unavailable
Events often provide the fastest clue to the root cause.
Step 7: Troubleshoot Services
Sometimes Pods are healthy, but users still cannot access the application.
Check:
kubectl get svc
Verify:
Correct selector
Correct ports
Endpoints populated
Then inspect:
kubectl describe service SERVICE_NAME
kubectl get endpoints
If there are no endpoints, the Service is not matching any Pods.
Step 8: Troubleshoot Networking
Networking issues may involve:
DNS failures
CNI plugin problems
Network Policies
Firewall rules
Load Balancer configuration
Useful commands:
kubectl get networkpolicies
kubectl get pods -o wide
kubectl exec -it POD_NAME -- ping SERVICE
kubectl exec -it POD_NAME -- nslookup SERVICE
Step 9: Troubleshoot Storage
Persistent applications rely on storage.
Check:
kubectl get pvc
kubectl get pv
Common problems:
PVC Pending
StorageClass missing
Volume attachment failure
Insufficient storage
CSI driver issues
Describe the PVC for detailed events:
kubectl describe pvc PVC_NAME
Step 10: Troubleshoot Deployments
Deployment problems include:
Wrong image version
Failed rollout
Replica mismatch
Configuration errors
Commands:
kubectl get deployment
kubectl rollout status deployment APP
kubectl rollout history deployment APP
kubectl rollout undo deployment APP
Resource Issues
Applications can fail due to insufficient resources.
Check resource usage:
kubectl top nodes
kubectl top pods
Symptoms:
High CPU
Memory pressure
OOMKilled
Node pressure
Increase resource limits or scale the application if needed.
Kubernetes Troubleshooting Cheat Sheet
Problem | Check / Command |
Cluster issue | kubectl cluster-info |
Node issue | kubectl get nodes |
Pod issue | kubectl get pods |
Pod details | kubectl describe pod |
Application logs | kubectl logs |
Events | kubectl get events |
Service issue | kubectl get svc |
Storage | kubectl get pvc |
Deployment | kubectl rollout status |
Resource usage | kubectl top nodes |
Best Practices
Start with the simplest checks before diving deeper.
Read Events before making changes.
Review application logs alongside Kubernetes events.
Monitor CPU, memory, and disk usage.
Use readiness and liveness probes correctly.
Define appropriate resource requests and limits.
Keep manifests under version control.
Monitor your cluster with tools like Prometheus and Grafana.
Real-World Scenario
An e-commerce application suddenly becomes unavailable.
Investigation:
kubectl get pods
shows CrashLoopBackOff.
kubectl describe pod
shows repeated restarts.
kubectl logs
reveals a database connection timeout.
The database Service has no endpoints because the database Pods are Pending.
kubectl describe pvc
shows the Persistent Volume Claim cannot bind due to insufficient storage.
After expanding storage and allowing the PVC to bind:
Database Pods start.
Service endpoints appear.
Application Pods reconnect successfully.
The website becomes available again.
Final Thoughts
Effective Kubernetes troubleshooting is about following a logical process rather than guessing. By moving from cluster → node → pod → container → application → networking → storage, you can isolate issues quickly and resolve them with confidence. As your experience grows, this systematic approach becomes second nature, reducing downtime and improving the reliability of your Kubernetes environments.
Comments