Kubernative by Palark | Kubernetes news and goodies

This summer, CNCF turns 10! (Here’s the original announcement of this organisation formation back in 2015.)

You can celebrate this anniversary by evaluating your contribution stats and seeing your first contribution in the CNCF-related GitHub repositories. Use CNCF ContribCard for that by typing your GitHub user here.

Following Daniel Krook, Senior Director of Developer Experience at CNCF, the community shares their contributor cards on social media (LinkedIn, Bluesky, etc.) using the #cTENcf hashtag now — feel free to join!

#news

CNOE AWS reference implementation

CNOE (Cloud Native Operational Excellence) is an Open Source organisation that focuses on building Kubernetes-based Internal Developer Platforms (IDPs) for enterprises. IDP Builder for spinning up a complete IDP featuring Docker as the only dependency is the most well-known project by CNOE.

Last week, the organisation published a GitHub repo with its AWS reference IDP implementation. It aims to create a production-ready IDP on EKS based on Crossplane, Backstage, Argo CD, Argo Workflows, Keycloak, and other Open Source tools.

UPD: In this YouTube video, Miguel Fontanilla, Platform Engineering Lead at Sennder, demonstrates the CNOE AWS IDP.

#news #tools #AWS

Reddit released ProgressiveDaemonSet for Kubernetes

ProgressiveDaemonSet is a Kubernetes controller and webhook implementation for safe, staged rollouts of DaemonSets. It adds automatic rate-limiting with Pod Scheduling Gates (configurable via annotations) and exposes Prometheus metrics to watch progress in real time.

This project emerged as a solution to the incident that brought half of Reddit offline in November 2024. It was caused by a kube-apiserver memory storm triggered by a one-line DaemonSet rollout. This post shares more details on the reasoning behind ProgressiveDaemonSet and its current implementation. The project is Open Source and available on GitHub.

#tools #news

KubeSphere is not Open Source anymore

KubeSphere is a well-known Kubernetes platform originating from China (created in QingCloud), boasting more than 12k GitHub stars. It also gave a start to a few CNCF projects, such as OpenFunction (currently in the Sandbox) and OpenELB (Archived).

Yesterday, a GitHub issue acknowledging this project is not Open Source was raised. Downloading the Open Source version of KubeSphere or even viewing its documentation became unavailable. It turned out that a commit changing the original project’s license (Apache 2) to the so-called “KubeSphere Open Source License” was made almost a year ago, in September 2024. This new license wasn’t Open Source since it enforced several limitations on the users, such as commercial use or offering SaaS.

Today, Ray Xiaosi ZHOU, the founding member of KubeSphere who just left QingCloud, stated:

This project carries countless late nights and relentless effort from our team. Seeing its reputation affected feels like a blow to everyone who once fought for its success. I understand the company’s reasoning. In recent years, repeated violations of the open-source license—by third parties repackaging and monetizing the project—have caused tangible impact on QingCloud’s interests. While the source code remains available under open-source norms, discontinuing the out-of-the-box distributions is, in my view, a challenging adjustment for today’s collaborative open-source ecosystem.

#news

Here goes our latest bunch of interesting Kubernetes-related articles recently spotted online:

1. "How I Survived the Great Kubernetes Exodus: Migrating EKS Cluster from v1.26 to v1.33 on AWS" by Ukeme David Eseme.

So when it was time to migrate a clients 3-4 years old Amazon EKS cluster from v1.26 to v1.33, I knew it wouldn’t just be a version bump—it would be a battlefield. This cluster wasn't just any cluster—it was a complex ecosystem running critical healthcare applications with: 46 Custom Resource Definitions (CRDs) across multiple systems, 7 production domains with SSL certificates, Critical data in PostgreSQL databases, Zero downtime tolerance for production services, Complex networking with Istio service mesh, Monitoring stack with Prometheus and Grafana…

2. "Debugging the One-in-a-Million Failure: Migrating Pinterest’s Search Infrastructure to Kubernetes" by Samson Hu, Shashank Tavildar, Eric Kalkanger, and Hunter Gatewood (Pinterest).

While migrating Pinterest’s search infrastructure — which powers core experiences for millions of users monthly — to Kubernetes, we faced a challenge in the new environment: one in every million search requests took 100x longer than usual. This post chronicles our investigation, uncovering an elusive interaction between our memory-intensive search system and a seemingly innocent monitoring process. The journey involves profiling search systems, debugging performance issues, Linux kernel features, and memory management.

3. "How we tracked down a Go 1.24 memory regression across hundreds of pods" by Nayef Ghattas, Datadog.

Our story begins while the new version was being rolled out internally. Shortly after deploying it to one of our data-processing services, we noticed an unexpected memory usage increase. We observed the same pattern, a ~20% increase in memory usage, across multiple environments before pausing the rollout. To confirm our suspicions, we conducted a bisect in the staging environment, which pointed directly to the Go 1.24 upgrade as the culprit.

4. "Production-Grade Pain: Lessons From Scaling Kubernetes on EKS" by Aditya Chowdhry, Probo.

Using AWS’s managed Kubernetes offering (EKS) initially simplified our infrastructure management, but as our application grew in scale and complexity, we faced several unexpected challenges in Scaling (Cluster Autoscaler Wasn’t Enough), Networking (Ingress Wars: AWS ALB vs. NGINX), and Application Behavior (Pod Sizing Matters; Graceful Termination; HPA Tuning).

5. "Kubernetes Monitoring — A Complete Solution, Part 8: Logging with VictoriaLogs" by Ryan Jacobs.

Part 8 in a series of posts where we’ll stand up an entire monitoring stack on my home Talos Linux cluster. [..] VictoriaLogs, which is made by the same team as VictoriaMetrics, only stores its data in a local directory, which can be backed by whatever your CSI provides in Kubernetes, and even plays well with NFS just like VM does.

6. "K8sGPT for Kubernetes troubleshooting: How AI helps in different cases" by Evgeny Torin, Palark.

In this article, I will explain what K8sGPT is, how to install it and connect to AI, and which features it offers. I will also share some examples of the output you can expect from this tool and what diagnostics it can perform. Throughout the preparation of this overview, I tested different AI integrations available as well as a number of models (including a local one). All of my examples will be backed up by commands and detailed logs.

#articles

Cloud Native Summit Munich 2025 (ex-KCD Munich) happened last week, and all its talks are now available on YouTube. This playlist features 38 videos, and here you can find the full schedule of the 2-day conference, which includes descriptions for all of these talks.

#events #video

The next Kubernetes release, 1.34, is scheduled for 27th August. The earliest article covering the upcoming changes was just published on the project’s blog. Its feature highlights include:

- An alpha version of KYAML, a new YAML subset that was designed for Kubernetes and aims to be a safer and less ambiguous;
- Improved tracing for kubelet and API Server;
- Structured parameters for Dynamic Resource Allocation (DRA) becoming stable;
- ServiceAccount tokens for image pull authentication moving to beta;
- PreferSameZone and PreferSameNode traffic distribution for Services moving to beta.

UPD: Even better (more detailed) overview of new K8s v1.34 features can be found in this excellent article by Nigel Douglas from Cloudsmith.

#news #releases #articles

Open Source Summit is a big offline event organised by The Linux Foundation for everyone involved in Open Source, featuring Cloud & Containers as one of its tracks. Till the end of this year, we can expect four such events around the world, including three of them in Asia(!):

- Open Source Summit India @ Aug 5, Hyderabad;
- Open Source Summit Europe @ Aug 25–27, Amsterdam;
- Open Source Summit Korea @ Nov 4–5, Seoul;
- Open Source Summit Japan @ Dec 8–10, Tokyo.

#events

The latest CNCF projects' velocity report was published. It names the following projects as the most actively developed (out of 231 hosted in CNCF) during the last year (from July 1st, 2024, to July 1st, 2025):

1. Kubernetes
2. OpenTelemetry
3. Prometheus
4. Argo
5. Backstage
6. Meshery
7. Cilium
8. Envoy
9. gRPC
10. Keycloak

If we compare it with the previous Top 10 for this period (July 1st, 2023, to July 1st, 2024), the most significant difference is:
- Meshery: 11th (a year ago) → 6th place (now);
- Istio: 9th → 15th;
- Prometheus: 5th → 3th.

Other prominent changes in Top 100 include:
- Podman Container Tools debuting at 18th place and CloudNativePG at 29th;
- OpenFGA: 34th → 21st;
- Headlamp: 99th → 50th;
- Buildpacks: 41st → 60th.

You can find all the latest stats on CNCF projects (contributors, commits, PRs, issues, etc.) in this public spreadsheet.

#news #cncfprojects

New Bitnami catalog limitations

The newly announced changes to the Bitnami public catalog of Helm charts and images continue its evolution of becoming more commercial. The authors state that starting August 28th, 2025, “Bitnami will continue to offer a limited subset of free, latest-version images intended for development use.”

Other users — e.g. those who need support for security updates, specific versions of an application, etc. — will need to subscribe to the commercial Bitnami Secure Images offering for that.

#news

“Have you tried turning it off and on again?..” Well, that’s surely not an approach we really want, but if it’s all you can do for some [nasty] app in Kubernetes, don’t hesitate to look at this workaroundish tool 🫠

Restart-operator is a Kubernetes operator that allows you to define schedules to restart specific workloads — e.g., suffering from memory leaks or needing to apply configuration changes — automatically via rolling updates. It comes with:

- Standard cron-style definitions for the schedules;
- Support for Deployments, StatefulSets, and DaemonSets as workloads, as well as support for K8s namespaces;
- Tracking the status of restarts performed.

Language: Go | License: MIT | 27 ⭐️

▶️ GitHub repo
💬 Reddit announcement

#tools

According to the State of Tech Talent 2025, published last month by The Linux Foundation (LF Research and LF Education):

- The most significant understaffing persists in the following fields: AI/ML engineering (68%), cybersecurity and compliance (65%), FinOps and cost optimisation (61%), cloud computing (59%), and platform engineering (56%);
- 71% of organisations consider certifications important when recruiting new talent;
- 85% prioritize portfolios of practical work in hiring decisions and see Open Source contributions as proof of technical and collaboration skills;
- 94% expect that AI will deliver significant value in core activities, increasing the need for a skilled workforce.

Find more insights by reading the full report here.

#reports #career

Amazon EKS announced it now supports Kubernetes clusters with up to 100,000 nodes. The authors name massive AI/ML workloads as a possible use case for such setups, which can fit 1.6m AWS Trainium chips or 800k NVIDIA GPUs in a single K8s cluster.

It’s been a significant effort in AWS involving a comprehensive set of improvements aimed at achieving such a scale. Here’s what the engineers did:

- Re-architecting etcd (its new design is illustrated in this post). It involved switching from a Raft-based consensus backend to an internal component built at AWS, moving BoltDB from network-attached Amazon EBS volumes to in-memory storage (tmpfs), and choosing an optimal partitioning scheme.
- Tuning API servers by elaborating specific configurations, enabling strongly-consistent reads from cache and streaming list responses, and using CBOR (Concise Binary Object Representation) encoding for custom resources.
- Optimizing cluster controllers' performance, enhancing Karpenter, scaling the cluster network, and introducing SOCI (Seekable OCI) fast pull for container image pulls.

Find more details, including the resulting benchmark charts, in this blog post.

#news #AWS

If you often render, validate, and debug Kubernetes manifests, you’ll surely find this new tool helpful.

kat simplifies working with manifests in your terminal by invoking their generators (such as Helm, Kustomize, CUE, KCL, etc.), displaying the resulting resources and providing several convenient features:

- browsable list structure for the Kubernetes resources, fuzzy search and filtering;
- live reload of the displayed data;
- built-in validation based on external tools, such as Kubeconform and Kyverno;
- customisable keybindings, profiles, themes, and even plugins.

P.S. We also can’t conceal the fact that the cat at the helm used for the project’s logo is almost as cute as our channel’s platypus, Pal 😂

Language: Go | License: Apache 2.0 | 61 ⭐️

▶️ GitHub repo
💬 Reddit announcement

#tools #cli

We haven’t shared any Kubernetes-related articles for a while. Filling this gap with some of the latest interesting reads:

1. “Kubernetes is not just for Black Friday” by Thibault Martin.

I’ve always ruled out Kubernetes as too complex machinery designed for large organizations who face significant surges in traffic during specific events like Black Friday sales. I thought Kubernetes had too many moving parts and would work against my objectives. I was wrong. Kubernetes is not just for large organizations with scalability needs I will never have. Kubernetes makes perfect sense for a homelabber who cares about having a simple, sturdy setup.

2. “Exploring Cloud Native projects in CNCF Sandbox. Part 4: 13 arrivals of 2024 H2” by Dmitry Shurupov, Palark.

Familiarise yourself with the following recently added CNCF projects: Ratify, Cartography, HAMi, KAITO, Kmesh, Sermant, LoxiLB, OVN-Kubernetes, Perses, Shipwright, KusionStack, youki, OpenEBS!

3. “Kubernetes List API performance and reliability” by Ahmet Alp Balkan.

We use Kubernetes beyond officially supported/tested scale limits by running more than 5,000 nodes and over a hundred thousand of pods in a single cluster. In these large scale setups, expensive “list” calls on the Kubernetes API are the achilles heel of the control plane reliability and scalability. In this article, I’ll explain which list call patterns pose the most risk, and how recent and upcoming Kubernetes versions are improving the list API performance.

4. “Kubernetes Networking from Packets to Pods” by Luca Cavallin.

The TCP/IP model, which powers the modern internet, is composed of four primary layers: [..] Understanding this layered approach is fundamental, as every network packet in a Kubernetes cluster adheres to this model. We'll explore this entire ecosystem in three parts: the foundational technologies that make it all possible, the core Kubernetes model itself, and finally, advanced topics and practical guides.

5. “What Would a Kubernetes 2.0 Look Like” by Matthew Duggan.

Some common trends have emerged, where mistakes or misconfiguration arise from where Kubernetes isn't opinionated enough. Even ten years on, we're still seeing a lot of churn inside of ecosystem and people stepping on well-documented landmines. So, knowing what we know now, what could we do differently to make this great tool even more applicable to more people and problems?

6. “Rootless container builds on Kubernetes” by Spyros Trigazis, CERN.

In this post, we will present 3 options (podman/buildah, buildkit and kaniko) for building container images in Kubernetes pods as non-root with containerd 2.x as runtime. Further improvements can be made using kata-containers, firecracker, gvisor or others but the complexity increases and administrators have to maintain multiple container runtimes.

#articles

Wondering what LEGO can assemble artfully, besides their well-known blocks? Well, let’s talk about Terraform resources in Kubernetes!

To make this real, this company has recently released Kube Terraform Reconciler (krec), a new Open Source project for platform engineers. It’s a Kubernetes operator for managing Terraform resources, allowing you to:

- Get an infrastructure defined by Terraform workspaces as Kubernetes custom resources and continuously reconciled;
- Specify Terraform backend configuration for workspaces;
- Enable auto-apply for workspaces;
- Use custom providers and modules.

Language: Go | License: Apache 2.0 | 164 ⭐️

▶️ GitHub repo

#tools #IaC

Happy to present our newest digest of the prominent software updates in the Cloud Native ecosystem!

1. OpenEBS, a persistent storage for Kubernetes workloads (a CNCF Sandbox project), released its v4.3.0 with numerous enhancements. They include data-at-rest encryption and IPv6 support in the replicated storage (Mayastor), a new unified plugin for interacting with all supported engines, and a backup garbage collector for LocalPV ZFS.

2. werf, a CLI tool for software delivery to Kubernetes (a CNCF Sandbox project), has recently released v2.38.0 and v2.39.0, bringing several new features. Now, it allows to have external configuration includes (to simplify reusing common templates), use template debugging mode for Helm charts, and keep specific tags while performing automatic cleanup of container images.

3. Argo CD (a CNCF Graduated project) announced v3.1, its next significant update, which is currently available as v3.1.0-rc1 only. This version introduces support for using OCI-compliant container registries as sources for configuration artifacts, support for CLI plugins, scaling resources directly from the UI, and client-side apply migration.

4. OpenTofu, a community-driven Terraform fork (a CNCF Sandbox project), reached v1.10.0 that introduced OCI registry support, native S3 state locking, enhanced planning, global provider cache lock, OpenTelemetry tracing, external key providers, official VS Code extension, Language Server Protocol support, and OpenTofu Registry MCP server.

5. Headlamp, a Web UI for Kubernetes, released 0.31.0 and 0.32.0, featuring tons of improvements. Particularly, the project got a multi-cluster view (experimental), a few new themes and support for plugin-defined themes, support for Node shell, Pod eviction and force deletion, new advanced search, and several new localisations.

6. OpenBao, a community-driven Vault fork, was updated to v2.3 with significant enhancements. Most notably, it now supports tenant isolation using namespaces (it’s available in UI as well), automatic unsealing using the KMIP protocol, and CEL (Common Expression Language) in PKI.

#news #releases

CNCF got a new Executive Director

Yesterday, a few changes in the CNCF and Linux Foundation top management were announced:

- Priyanka Sharma, the Executive Director at the CNCF for the last five years, stepped down.
- Jonathan Bryce became the Executive Director at the CNCF and Executive Director of Cloud & Infrastructure at the Linux Foundation. He has been the Executive Director of the OpenInfra Foundation since 2012, and this role will stay with him as well. Previously this year, OpenInfra joined the Linux Foundation.
- Chris Aniszczyk, who has been the CTO at the CNCF, keeps this position and also became the CTO of Cloud & Infrastructure at the Linux Foundation. This means he will work on more cloud projects at the parent organisation.

#news

Seeing Kubernetes nodes’ resource consumption in your terminal has become much easier with this new project.

kubectl node-resource is a kubectl plugin that shows resource allocation and their actual utilisation for your Kubernetes nodes. This tool offers:

- A simple list view and a summary view. The latter features histograms and distribution buckets for both resource allocation and utilisation;
- Displaying specific resources only and free resources, as well as sorting nodes by resource usage;
- Structured JSON output to integrate this data with other tools;
- Optimised API server querying to ensure support for large K8s clusters.

Language: Go | License: Apache 2.0 | 77 ⭐️

▶️ GitHub repo

#tools #cli

Salesforce makes a U-turn for the official Slack workspaces of Kubernetes and CNCF. Their enterprise accounts will still be available (with not much other details provided at the moment).

#news