☸️ Kubernetes: Fix a PDB Blocking Node Drains During Maintenance
A PodDisruptionBudget with minAvailable set equal to (or very close to) a Deployment’s total replica count leaves zero disruption budget for the Eviction API to actually work with. kubectl drain calls that API for every pod on the node, and the API server rejects any eviction that would push available replicas below minAvailable — so the drain hangs retrying indefinitely instead of failing outright, and nothing points at the PDB unless you check it directly.
🔎 The Problem
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-pdb
spec:
minAvailable: 3 # Deployment also has exactly 3 replicas
selector:
matchLabels:
app: checkout
# kubectl get pdb checkout-pdb
# NAME MIN AVAILABLE ALLOWED DISRUPTIONS
# checkout-pdb 3 0 <- drain can never evict a single pod
✅ Fix: Leave Real Budget for a Planned Disruption
- Check
kubectl get pdb -o widefor ALLOWED DISRUPTIONS before scheduling a node drain, and only proceed if it’s greater than zero for every PDB covering pods on that node. - Size
minAvailable(ormaxUnavailable) to tolerate at least one pod disruption during routine maintenance — for example, N-1 instead of N for an N-replica Deployment. - If a window genuinely can’t allow any disruption, scale the Deployment up temporarily before draining so the extra replica absorbs the eviction without violating the budget.
🚨 Why This Is Dangerous
- A drain that silently hangs on ALLOWED DISRUPTIONS: 0 can leave an operator believing a maintenance step completed when the node is still running production pods, and force-deleting the stuck pods to unblock it bypasses the exact safety guarantee the PDB existed to provide.
A PodDisruptionBudget doesn’t block disruption in general — it blocks the specific eviction that would take you below the floor you set, however long that makes the drain wait.
