DevOps&SRE Library

GKE pod IP exhaustion is one of the few failure modes that gives you no warning before it goes terminal. I recently stepped into a war room where a client’s primary scaling group had flatlined — workloads cordoned, deployments stuck in Pending, and the estimated cost of the stall nearing $15k per hour in lost transaction volume. The culprit wasn’t traffic. It was a /20 subnet that had quietly run out of address space, and a set of GKE allocation defaults nobody had questioned at design time.

A few weeks after publishing the 1M token/s post, I spent a weekend helping my good friend Milko Ilari set up vLLM on his shiny new DGX Spark with Gemma 4. My first in-person reaction was “It’s Champagne” (from the old days of PC Perspective) The Spark is a wild little machine — 128 GB of unified memory in a box you can hold with one hand, running the same Blackwell architecture as the datacenter B200s. But its memory bandwidth is 273 GB/s. The B200s in our cluster do 8,000 GB/s. Almost 30x less.

Watching the numbers on that tiny machine got me thinking. The benchmark I ran on GKE Autopilot with 96 B200 GPUs had reported 4.4% FLOPS utilization. 10.9% memory bandwidth. Tensor cores active 1.5% of the time. The GPUs looked almost idle while pushing a million tokens per second. Was something wrong?

No. And honestly, figuring out why turned out to be more interesting than the benchmark itself.

That first post covers the journey — every optimization, and many failure 🫠. This one covers the physics.

Harmful workload generator for PostgreSQL.

ZeroFS serves S3-compatible buckets as POSIX filesystems over NFS and 9P, and as raw block devices over NBD. All three servers run in a single userspace process. Data is compressed and encrypted before upload.

A powerful and flexible Kubernetes Model Context Protocol (MCP) server implementation with support for Kubernetes and OpenShift.

A native macOS UI for Apple's container platform — think OrbStack/Docker Desktop, but for the Linux-containers-as-lightweight-VMs stack Apple ships for Apple silicon.

OfficeCLI is the first and best Office suite purpose-built for AI agents to read, edit, and automate Word, Excel, and PowerPoint files. Free, open-source, single binary, no Office installation required.

A Ghostty-based macOS terminal with vertical tabs and notifications for AI coding agents

When the storage backend for Stream Router hit hard limits, we needed to redesign its data model and migrate it to a new storage architecture without disrupting live production traffic. We would not have completed the implementation in the time frame we had without AI tools.

Lore is an open source version control system designed for unprecedented scalability of both data and teams. It is optimized for projects that combine code with large binary assets, including games and entertainment, and caters for the needs of developers and artists alike.

Tabularis is an open-source desktop SQL workspace for PostgreSQL, MySQL/MariaDB, SQLite and 12+ more databases like DuckDB, ClickHouse, Redis and Firestore.

HASP is a local secret broker for coding agents.

Agents need credentials to run tests, call APIs, and deploy code. Copying those credentials into prompts, shell history, .env files, or repo-local notes makes the agent faster today and harder to trust tomorrow. HASP keeps secrets in a local encrypted vault and gives commands only the values they are allowed to use at runtime.

The core rule is:

Managed secret values must not enter agent context.

Managing secrets at scale is not just about security — it’s about availability, resilience, and cost efficiency. At BioCatch, we use HashiCorp Vault to manage secrets in our Kubernetes environment, but our initial setup left room for improvements.

Here’s how we redesigned our Vault architecture using Raft storage, Kubernetes automation, and snapshot-based disaster recovery — resulting in a system that is highly available, cost-effective, and production-ready.

In the world of Kubernetes, resource management is more than just creating, deleting, or updating objects. It's an intricate dance involving numerous tools, operators, and users. As our infrastructure grows, it becomes increasingly challenging to maintain control, necessitating the adoption of more advanced and sophisticated resource management and control systems and approaches.

In this article, I will not delve into the basic methods of creating resources, as that is a rather trivial task. Instead, I would like to share my experience in optimizing resource update paths, which have proven to be immensely valuable in managing resources within large and complex Kubernetes clusters, as well as during the development of operators.

Are your Kubernetes applications inexplicably slowing down, even when your nodes seem to have plenty of CPU to spare? You might be experiencing CPU throttling. I recently watched an excellent CNCF video by Dave Chiluk, an engineer at Indeed. This article has some of my takeaways from the video.

The Kubernetes NodeSwap feature, likely to graduate to stable in the upcoming Kubernetes v1.34 release, allows swap usage: a significant shift from the conventional practice of disabling swap for performance predictability. This article focuses exclusively on tuning swap on Linux nodes, diving into the critical Linux kernel parameters that govern swap behavior and how they influence workload performance, swap utilization, and eviction mechanisms.

Kubernetes is powerful, but troubleshooting issues in a live cluster can be painful. In a complex deployment, critical warning signs often hide in thousands of log lines and events. What if we could surface these reliability issues before they take applications down?

Istio is a foundational piece of our architecture, which makes ongoing maintenance and upgrades a challenge. Despite that, we have upgraded Istio a total of 14 times. This blog post will explore how the Service Mesh team at Airbnb safely upgrades Istio while maintaining high availability.

Kubernetes networking doesn't have to be a black box. This guide breaks it down, starting from the fundamentals of Linux networking and container isolation. We then dive into the complete Kubernetes model, explaining everything from Pod IPs and CNI plugins to Services, NetworkPolicy, and Ingress, providing a clear end-to-end map of how connectivity works in your cluster.