Skip to content
  • Home
  • About
  • No Access
  • Privacy Policy
Bits of .NET
Bits of .NET
  • Asp.Net Core
  • C#
  • SQL
  • JavaScript
  • CSS
  • About
  • ErcanOPAK.com
  • No Access
  • Privacy Policy

Bits of .NET

Daily micro-tips for C#, SQL, performance, and scalable backend engineering.

  • Asp.Net Core
  • C#
  • SQL
  • JavaScript
  • CSS
  • About
  • ErcanOPAK.com
  • No Access
  • Privacy Policy
Kubernetes

Kubernetes Monitoring: The Prometheus + Grafana Stack That Saved Our Production

- 23.02.26 - ErcanOPAK

🚨 The 3 AM Wake-Up Call

Your Kubernetes cluster is down. Users are angry. You have no idea what happened. This is why Fortune 500 companies spend $500K/year on observability.

The Complete Observability Stack

📊 The Three Pillars

  1. Metrics – What’s happening right now (Prometheus)
  2. Logs – What happened in the past (Loki)
  3. Traces – Why requests are slow (Jaeger)

âš¡ Quick Install (Helm)

# Add Prometheus repo
helm repo add prometheus-community \\
  https://prometheus-community.github.io/helm-charts

# Install Prometheus + Grafana stack
helm install monitoring prometheus-community/kube-prometheus-stack \\
  --namespace monitoring --create-namespace

# Get Grafana password
kubectl get secret -n monitoring monitoring-grafana \\
  -o jsonpath="{.data.admin-password}" | base64 --decode

# Port forward to access
kubectl port-forward -n monitoring \\
  svc/monitoring-grafana 3000:80
                

Result: Full monitoring stack in 5 minutes. 50+ pre-built dashboards included.

🎯 Critical Alerts

  • Pod restarts > 5 in 10 min
  • Memory > 90% used
  • CPU throttling detected
  • Disk > 85% full
  • Node not ready > 2 min

📈 Key Metrics

  • Request rate (QPS)
  • Error rate (4xx, 5xx)
  • Latency (p50, p95, p99)
  • Saturation (CPU, RAM)
  • Availability (uptime %)

🔥 Production War Stories

The Memory Leak That Cost $50K

Pod memory slowly climbing. Grafana showed it weeks before the crash. We ignored it. Pod OOMKilled during Black Friday. Lost $50K in sales. Now we have alerts.

The DDoS We Caught in 60 Seconds

Request spike alert fired. Dashboard showed 10,000% traffic increase from one IP range. Blocked at CDN level. Attack neutralized before impacting users. Total downtime: 0 seconds.

Dashboard Purpose When to Check
Cluster Overview Overall health Daily morning check
Node Metrics Hardware utilization Capacity planning
Pod Metrics Application performance During deployments
API Server Kubernetes health When things feel slow

“Before Prometheus: We found out about outages from angry customers. After Prometheus: We fix issues before users notice them.”

— DevOps Lead, SaaS company

Related posts:

Kubernetes: Fix a HorizontalPodAutoscaler That Never Scales Down Because of the Default Stabilizatio...

Kubernetes: Fix a HorizontalPodAutoscaler That Never Scales Down Because of the Default Stabilizatio...

Kubernetes: Liveness vs Readiness Probes - Don't Kill Your Traffic

Kubernetes: Liveness vs Readiness Probes - Don't Kill Your Traffic

Kubernetes: Use PodDisruptionBudget for High Availability

Kubernetes: Use PodDisruptionBudget for High Availability

Views: 14

Post navigation

WordPress Caching: The 4-Layer Strategy That Handles 1 Million Daily Visitors
Docker Multi-Stage Builds: From 1.2GB to 50MB – A Production Story

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

November 2026
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
30  
« Oct    

Most Viewed Posts

  • How to make theater mode the default for Youtube (1,055)
  • Get the User Name and Domain Name from an Email Address in SQL (981)
  • How to add default value for Entity Framework migrations for DateTime and Bool (977)
  • Get the First and Last Word from a String or Sentence in SQL (858)
  • How to select distinct rows in a datatable in C# (854)
  • How to enable, disable and check if Service Broker is enabled on a database in SQL Server (653)
  • Add Constraint to SQL Table to ensure email contains @ (609)
  • Average of all values in a column that are not zero in SQL (570)
  • How to use Map Mode for Vertical Scroll Mode in Visual Studio (547)
  • Find numbers with more than two decimal places in SQL (475)

Recent Posts

  • C#: Fix a ConditionalWeakTable Entry Vanishing Too Soon
  • Docker: Fix a Stale COPY Layer After Editing .dockerignore
  • Photoshop: Fix a Pattern Overlay Scaling Wrong on a High-DPI File
  • SQL Server: Fix a Computed Column Index Skipped by Implicit Cast
  • HTML5: Fix a required Input Inside a Hidden Tab Blocking Submit
  • SQL Server: Fix ROW_NUMBER Ties Reordering Between Identical Runs
  • C#: Fix an Enum HasFlag Returning True for the Zero Value
  • WordPress: Fix Featured Images Vanishing After a CDN Migration
  • Photoshop: Fix Content-Aware Fill Smearing a Repeating Pattern
  • Asp.Net Core: Fix Antiforgery Tokens Rejected Behind a Reverse Proxy

Most Viewed Posts

  • How to make theater mode the default for Youtube (1,055)
  • Get the User Name and Domain Name from an Email Address in SQL (981)
  • How to add default value for Entity Framework migrations for DateTime and Bool (977)
  • Get the First and Last Word from a String or Sentence in SQL (858)
  • How to select distinct rows in a datatable in C# (854)

Recent Posts

  • C#: Fix a ConditionalWeakTable Entry Vanishing Too Soon
  • Docker: Fix a Stale COPY Layer After Editing .dockerignore
  • Photoshop: Fix a Pattern Overlay Scaling Wrong on a High-DPI File
  • SQL Server: Fix a Computed Column Index Skipped by Implicit Cast
  • HTML5: Fix a required Input Inside a Hidden Tab Blocking Submit

Social

  • ErcanOPAK.com
  • GoodReads
  • LetterBoxD
  • Linkedin
  • The Blog
  • Twitter
© 2026 Bits of .NET | Built with Xblog Plus free WordPress theme by wpthemespace.com