DevOps&SRE Library

I'm currently working with a client to build out an agent-based automation system designed to reduce the manual labor associated with weekly, monthly, and ad-hoc operational activities.

Picture this. A developer asks an AI assistant to "scale down staging to save costs." The AI, helpful as always, executes: kubectl scale deployment critical-api --replicas=0 -n production. Wrong namespace. Right outcome, wrong cluster. The API is down.

We recently started migrating Kafka clusters from EC2 to EKS using Strimzi. As soon as we moved the first cluster, we saw persistent disk reads across the brokers and higher latency than we expected on comparable hardware.

Kubernetes has matured into a stable platform, while innovation is increasingly happening in the tools around it.

At this scale, even minor overloads aren't isolated events, they cascade. A brief spike in one part of the system can ripple outward: downstream services time out, retries pile up, and degradation amplifies into broader failure. This blog shares how we built an intelligent load manager that detects overload from multiple signals to keep our databases stable and fair under pressure.

For eight years I ran the SRE team behind a storage system measured in exabytes. Over time, the dashboard I checked every morning shrank to a handful of numbers. These are the seven that told me whether the service was healthy.

Agent skill for beautiful, verifiable architecture, workflow, sequence, data-flow, and lifecycle diagrams—self-contained HTML with motion and crisp export.

NVIDIA Personal AI Router (PAIR) is a local inference router for a group of compatible computers on the same network. It discovers participating nodes, manages supported inference engines, and presents Ollama-compatible and OpenAI-compatible proxy endpoints to applications and agents. Independent requests can be routed to eligible nodes according to engine availability, model availability, and current workload.

WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU.

Ship one musl-linked executable. At runtime, load the user's existing glibc-linked GPU driver. No container, no AppImage, and no second libc in the process.

In high-performance networking, 35 seconds is a lifetime. This was the latency required to iterate through our connection tracking table of 7 million entries at a maximum speed of 200,000 entries per second. At our 16-million-entry peak, this sequential lookup could take up to 80 seconds, leading to Cilium CNI timeouts preventing new pods from starting on affected nodes.

Tailcat is a remix of Tailscale open source pieces to act like netcat, but over Tailscale's data plane, without Tailscale's control plane. Tailscale's data plane (magicsock, internally) gives you point-to-point WireGuard®-encrypted tunnels between two machines with DERP as the NAT-hole-punching communication side channel and the ultimate relay-of-last-resort if NAT traversal fails. Instead of using the Tailscale control plane, all tailcat connection metadata is exchanged out of band, however you want.

You see someone declare a Sev-2 and you wonder: wait, why is that even an incident? Nothing is down. Customers aren't affected. But a manager needed to get their team's problem to the top of another team's priority queue, and the incident process was a reliable way to make it happen.

Data Plane API is a sidecar process that runs next to HAProxy and provides API endpoints for managing HAProxy. It requires HAProxy version 1.9.0 or higher.

Agent Substrate delivers a performant, high density runtime environment for large scale agent deployments. At its core, Agent Substrate maps a larger set of "actors" (applications such as agents) onto a smaller set of ready "workers", relying on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing.

Kubernetes cluster cost optimizer. Connects to a live cluster (or a saved snapshot), collects every scheduling constraint, and uses CP-SAT constraint programming to find the cheapest node fleet that still satisfies all placement rules.

Synthetic monitoring for gRPC server-side streams — open each stream, verify frames actually arrive, and export Prometheus metrics.

k8s-mechanic is a Kubernetes controller that watches your cluster for failures, investigates them automatically, and opens pull requests on your GitOps repository with proposed fixes — all without leaving your cluster. When a Pod is crash-looping, a Deployment is degraded, or a Node goes NotReady, mechanic spawns an in-cluster OpenCode agent that inspects the live cluster, locates the relevant manifests in your GitOps repo, determines the root cause, and opens a PR.