Skip to content
  • Home
  • About
  • No Access
  • Privacy Policy
Bits of .NET
Bits of .NET
  • Asp.Net Core
  • C#
  • SQL
  • JavaScript
  • CSS
  • About
  • ErcanOPAK.com
  • No Access
  • Privacy Policy

Bits of .NET

Daily micro-tips for C#, SQL, performance, and scalable backend engineering.

  • Asp.Net Core
  • C#
  • SQL
  • JavaScript
  • CSS
  • About
  • ErcanOPAK.com
  • No Access
  • Privacy Policy
Kubernetes

Kubernetes: Fix a Pod That Fails to Schedule Because a Taint on Every Node Was Never Matched by a Toleration

- 16.10.26 - ErcanOPAK

☸️ The Pod That Was Never Invited to Any Node

Taints mark a node as off-limits to any pod that doesn’t explicitly declare a matching toleration – a common setup for dedicated node pools (GPU nodes, a pool reserved for one specific workload). The trouble starts when a cluster-wide change – a default node pool configuration, a maintenance taint applied broadly, or a new node pool template that taints every node including the general-purpose ones – ends up tainting nodes that ordinary pods were always meant to run on. If those pods’ specs were never updated with the matching toleration, they simply never get scheduled anywhere: no crash, no error in the pod’s own logs because it never started, just a scheduling event buried in kubectl describe pod that most people don’t think to check first.

🔎 The Problem

# Node has a taint applied (e.g. during a new node pool rollout):
#   kubectl taint nodes node-1 workload=restricted:NoSchedule

# Pod spec was never updated with a matching toleration - it
# simply stays Pending forever, with "kubectl describe pod" showing:
#   Warning  FailedScheduling  ... 0/3 nodes are available:
#   3 node(s) had untolerated taint {workload: restricted}.

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: web
      image: myapp:latest
  # No tolerations block - this pod can never land on any
  # tainted node, including ones it was always meant to use.

# Fixed: add the matching toleration.
spec:
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "restricted"
      effect: "NoSchedule"
  containers:
    - name: web
      image: myapp:latest

✅ Fix: Read the Scheduling Event, Then Match the Toleration

  • Run kubectl describe pod, naming the specific pod, and read the Events section for ‘had untolerated taint’ before assuming the application itself is broken – this is the one place the real cause is actually reported.
  • Add the exact matching toleration (key, value, and effect) to any pod spec that genuinely needs to run on tainted nodes, rather than guessing at a partial match that won’t actually satisfy the scheduler.
  • Audit cluster-wide taints whenever a new node pool, autoscaler configuration, or maintenance window is introduced – it’s easy to taint more broadly than intended and silently affect workloads that were never meant to be restricted.

🚨 Why This Is Dangerous

  • This looks exactly like a resource-shortage or image-pull problem at first glance, so the instinct is often to chase capacity or registry issues instead of the actual scheduling constraint.
  • A single cluster-wide taint change can silently stop ALL new pod scheduling at once if deployment specs across the cluster were never updated to match it – the blast radius is much larger than one pod.

A pod stuck in Pending isn’t waiting for resources to free up – it may simply have never been invited to any node in the first place.

— Cloud Architect

Related posts:

Why Your Pod “Looks Healthy” But Still Drops Traffic

Why Your Pod “Looks Healthy” But Still Drops Traffic

Kubernetes: Use Helm Charts to Package and Deploy Apps

Kubernetes: Use Helm Charts to Package and Deploy Apps

Kubernetes — CPU Limits Cause Throttling Spikes

Kubernetes — CPU Limits Cause Throttling Spikes

Views: 9

Post navigation

WordPress: Fix wp_remote_get Calls to an External API That Silently Fail Because of a Missing SSL Certificate Verification Setting
SQL Server: Fix a Correlated Subquery in the SELECT List That Silently Runs Once Per Row Instead of a Set-Based CROSS APPLY

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

November 2026
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
30  
« Oct    

Most Viewed Posts

  • How to make theater mode the default for Youtube (1,055)
  • Get the User Name and Domain Name from an Email Address in SQL (981)
  • How to add default value for Entity Framework migrations for DateTime and Bool (977)
  • Get the First and Last Word from a String or Sentence in SQL (858)
  • How to select distinct rows in a datatable in C# (854)
  • How to enable, disable and check if Service Broker is enabled on a database in SQL Server (653)
  • Add Constraint to SQL Table to ensure email contains @ (609)
  • Average of all values in a column that are not zero in SQL (570)
  • How to use Map Mode for Vertical Scroll Mode in Visual Studio (547)
  • Find numbers with more than two decimal places in SQL (475)

Recent Posts

  • C#: Fix a ConditionalWeakTable Entry Vanishing Too Soon
  • Docker: Fix a Stale COPY Layer After Editing .dockerignore
  • Photoshop: Fix a Pattern Overlay Scaling Wrong on a High-DPI File
  • SQL Server: Fix a Computed Column Index Skipped by Implicit Cast
  • HTML5: Fix a required Input Inside a Hidden Tab Blocking Submit
  • SQL Server: Fix ROW_NUMBER Ties Reordering Between Identical Runs
  • C#: Fix an Enum HasFlag Returning True for the Zero Value
  • WordPress: Fix Featured Images Vanishing After a CDN Migration
  • Photoshop: Fix Content-Aware Fill Smearing a Repeating Pattern
  • Asp.Net Core: Fix Antiforgery Tokens Rejected Behind a Reverse Proxy

Most Viewed Posts

  • How to make theater mode the default for Youtube (1,055)
  • Get the User Name and Domain Name from an Email Address in SQL (981)
  • How to add default value for Entity Framework migrations for DateTime and Bool (977)
  • Get the First and Last Word from a String or Sentence in SQL (858)
  • How to select distinct rows in a datatable in C# (854)

Recent Posts

  • C#: Fix a ConditionalWeakTable Entry Vanishing Too Soon
  • Docker: Fix a Stale COPY Layer After Editing .dockerignore
  • Photoshop: Fix a Pattern Overlay Scaling Wrong on a High-DPI File
  • SQL Server: Fix a Computed Column Index Skipped by Implicit Cast
  • HTML5: Fix a required Input Inside a Hidden Tab Blocking Submit

Social

  • ErcanOPAK.com
  • GoodReads
  • LetterBoxD
  • Linkedin
  • The Blog
  • Twitter
© 2026 Bits of .NET | Built with Xblog Plus free WordPress theme by wpthemespace.com