Bits of .NET

Pod stuck in “Terminating” state for hours? Kubernetes is waiting for graceful shutdown that will never complete. Force delete it properly.

The Problem:

# Pod stuck forever
kubectl get pods
NAME                 STATUS        AGE
stuck-pod-abc123     Terminating   3h

# Normal delete doesn't work
kubectl delete pod stuck-pod-abc123
# Still shows Terminating...

Why Pods Get Stuck:

When deleting a pod:
1. Kubernetes sends SIGTERM to container
2. Waits 30 seconds (terminationGracePeriodSeconds)
3. If still alive, sends SIGKILL

Pod gets stuck when:
- Node is unreachable (network issue)
- Finalizer preventing deletion
- Volume unmount fails
- Container ignoring SIGTERM

The Nuclear Option – Force Delete:

kubectl delete pod stuck-pod-abc123 --grace-period=0 --force

# Explanation:
# --grace-period=0  : Skip graceful shutdown
# --force           : Delete immediately from etcd

Pod disappears instantly!

Better: Check Why It’s Stuck First

# See what's preventing deletion
kubectl describe pod stuck-pod-abc123

# Look for:
# - Finalizers: [...list...]
# - Events: Warning messages
# - Node: NotReady status

# Common culprits:
# 1. Finalizer not removed
# 2. PersistentVolumeClaim stuck detaching
# 3. Node disappeared

Remove Finalizers Manually:

# Check finalizers
kubectl get pod stuck-pod-abc123 -o yaml | grep -A5 finalizers

# Output:
# finalizers:
# - kubernetes.io/pvc-protection

# Remove finalizers
kubectl patch pod stuck-pod-abc123 -p '{"metadata":{"finalizers":null}}'

# Pod deletes immediately

Force Delete All Terminating Pods:

# Find all stuck pods
kubectl get pods --field-selector=status.phase=Terminating

# Delete them all
kubectl delete pods --field-selector=status.phase=Terminating \
  --grace-period=0 --force

# Or with namespace
kubectl delete pods -n my-namespace \
  --field-selector=status.phase=Terminating \
  --grace-period=0 --force

Prevent Future Stuck Pods:

# In your deployment YAML
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 10  # Reduce from 30 to 10
      containers:
      - name: app
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 5"]
        # App has 5 seconds to cleanup before SIGTERM

Why This Matters:

Scenario: Node fails, 20 pods stuck Terminating

Without force delete:
- Pods stuck for hours (until node timeout)
- New pods can't start (resource quota reached)
- Application downtime

With force delete:
- Pods removed in 30 seconds
- New pods start immediately  
- Application recovers

PersistentVolume Stuck? Delete That Too:

# PVC won't delete because pod using it
kubectl delete pvc my-pvc
# Stuck in Terminating

# Force delete PVC
kubectl patch pvc my-pvc -p '{"metadata":{"finalizers":null}}'

# Or force delete PV
kubectl delete pv my-pv --force --grace-period=0

Node Stuck? Delete It:

# Node shows NotReady, pods stuck
kubectl get nodes
NAME       STATUS     AGE
node-1     NotReady   10h

# Force delete node
kubectl delete node node-1 --force --grace-period=0

# This doesn't delete the actual VM!
# It only removes it from Kubernetes
# Pods on that node will be rescheduled elsewhere