🚫 Dedicated Nodes for Special Workloads
Some nodes have GPUs. Some are spot instances. Taints repel pods. Tolerations allow them. Control exactly where workloads run.
📝 Add Taint to Node
# Taint effects: # NoSchedule: Pods without toleration won't be scheduled # PreferNoSchedule: Avoid if possible # NoExecute: Evict existing pods without toleration # Add taint kubectl taint nodes node1 gpu=true:NoSchedule # Remove taint kubectl taint nodes node1 gpu=true:NoSchedule- # View taints kubectl describe node node1 | grep Taints
🎯 Pod Toleration
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
tolerations:
- key: "gpu"
operator: "Equal"
value: "true"
effect: "NoSchedule"
containers:
- name: cuda-app
image: nvidia/cuda:latest
# Toleration for any taint (accept all)
tolerations:
- operator: "Exists"
# Toleration with specific time (NoExecute only)
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 600 # Wait 10 min before evicting
✅ Real-World Patterns
- GPU nodes: Taint: nvidia.com/gpu=true:NoSchedule
- Spot instances: Taint: spot=true:NoExecute + tolerationSeconds
- Master nodes: Default taint prevents general workloads
- Dedicated database nodes: Taint: db=true:NoSchedule
💡 Best Practices
- Use taints for specialized hardware (SSD, GPU)
- Use NoExecute with tolerationSeconds for spot instances
- Don’t overuse — leave most nodes untainted
- Combine with nodeAffinity for positive selection
“General workloads were consuming GPU nodes. Added taint to GPU nodes, toleration only for ML pods. GPU utilization went from 20% to 80%. Saved $2000/month.”

