TGViewer
Channel Public Channel
CatOps

CatOps

@catops

DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own.

We do not post ads including event announcements. Please, do not bother us with such requests!
Subscribers
5.09K
Photos
94
Videos
5
Links
2.7K

Showing posts older than #2967 · Back to latest

Older Posts 20 shown
Post #2966 1.2K
​​Happy Independence Day of Ukraine! 💛💙

It always starts with small steps, but every step counts. So today, I'd like to get back to small fundraisers that also matter!

Here, a friend of mine is raising funds for a pickup track for the Zaporizhzhia direction.

You can find more info in this Instagram post (in Ukrainian).

You can donate to this Monobank jar:
https://send.monobank.ua/jar/8Mjotj4Uei

#donations #Ukraine
  • 🎉 18
  • ❤ 3
Post #2964 1.32K
​University of Colorado published an article about a study of 7700 employees that shows that Remote workers report the highest well-being.

The study simply analyzed the self-reported data without any judgments on why exactly this is the case, but one of the authors points out that:

The study did not explore why remote workers reported higher well-being, but Johnson points to a growing body of research on autonomy and flexibility.

One explanation is that remote workers have greater control over their work setup and daily schedule, she said.

"If you have control over your environment, you tend to have more positive outcomes," she said.

Working from home can also eliminate many everyday stressors.

"Spending a lot of time in traffic is negatively related to well-being," Johnson said. "There are so many little stressors associated with being in the office."

Those stressors can include arranging child care, hiring help for pets or managing the logistics of getting to and from work, she said.

Johnson said the study points to a broader lesson for employers navigating return-to-office debates. Rather than focusing only on where employees work, organizations may get better results by investing in employee well-being.


I know that this has nothing to do with the immediate topic of systems / platform / reliability engineering, but I still find this article amusing, since we all are doing our jobs in one way or another: in an office or remotely.

Now, you also have a study to point to, if anyone brings up the topic of the remote work and well-being. However, it's important to keep in mind that the employee well-being is not necessary the metric that is being optimized when the RTO directives are issued.

#culture
  • 🔥 4
  • 🤗 1
Post #2963 1.26K
A postmortem for the recent GitHub incident. Regarding the cause:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. 


#postmortem #github
Githubstatus Incident with GitHub.com GitHub's Status Page - Incident with GitHub.com.
  • 👍 12
Post #2962 1.27K
The Hidden Cost of Postgres Constraints at Scale is a description of a bottleneck foreign keys can create in a write-heavy PostgreSQL table. It also has recommendations of what you can do to overcome that.

Now, I would say, it's a corner case for many people. The author works in a company that develops their own time series solution based on Postgres, so for them, it makes sense.

For the vast majority of people, I would argue, it makes more sense to offload the integrity checks to the database, rather than trying to implement those in-house. Of course, it may be different for you, but no one knows your system better than you do.

#db #postgres
Hackernoon The Hidden Cost of Postgres Constraints at Scale Learn why PostgreSQL FOREIGN KEY and UNIQUE constraints become bottlenecks at high ingest rates—and four proven ways to reduce write latency safely.
  • 👍 2
Post #2961 1.5K
Last Sunday we discussed that we all learn anew, how to deal with agentic SDLC.

Here's a great interview with a principal engineer at Adyen and a former Amazonian Heitor Lessa on how he approaches this.

The interview is long and information-dense at times, but it's worth the time!

#ai #youtube
YouTube AWS Veteran: How The New Software Development Life Cycle Works "I need to stop using Opus. This doesn't work." That was Heitor Lessa's conclusion after a refactor cost him 200 million tokens, and it forced him to rebuild the entire agent workflow now available for 1400 engineers. Heitor spent 11 years at AWS, built Lambda…
  • 👍 4
Post #2960 1.43K
Argo For Kubernetes: From Argo CD to Workflows and Image Updater | A complete Guide is a nice article about building a complete CD workflow with Argo products in a home lab.

I would still not call it a "complete guide", because it's only scoped for local development, but it's a nice entry point, if you want to learn more about Argo on a home server, or create a small PoC.

#cicd #kubernetes #argo
The Thought Process Argo For Kubernetes: From Argo CD to Workflows and Image Updater | A complete Guide Learn how to build a fully native, end-to-end Argo Ecosystem on Kubernetes. This step-by-step guide covers setting up Argo CD, Argo Workflows, and Argo Image Updater.
  • 🤝 3
Post #2959 1.33K
An article that asks an important question: what happens when you are given a vibe-coded project by a manager and how to deal with it?

This article is partially behind a paywall, but the question is important nevertheless. Moreover, I feel like in the platform field it's even more visible, since you are handed not only projects by your managers, but also vibe-coded projects by developers.

Here, two things become important. First, "partial work has no value", and the second is that a handover is a handover - it doesn't matter who or what created the code, all the existing production-readiness checks should be there.

#ai #culture
Blog4Ems What happens when your PM learns to vibe code “Can the team just productionise this?”
  • 👍 3
Post #2958 1.3K
Two bundles in one post.

- Hands-on DevOps bundle - these are courses, so if you buy them, make sure to claim them before 2027-09-01!
- Software Engineering books by Manning. Just check if you have those already, because some of these books were parts of other bundles.

#books
Humble Bundle The Hands-On DevOps Bundle Take a hands-on approach to DevOps with our latest software bundle with Packt. Plus, a portion of your purchase helps save whales and dolphins!
  • ❤ 1
Post #2957 1.71K
Post #2955 1.41K
Post #2954 1.43K
An interesting article that explains how Karpenter works.

It does have flaws, though. Apart from useless footnotes, it doesn’t elaborate on how did they address the behavior, only some abstract suggestions, some of which kinda contradict what Karpenter actually is.

Spoiler: their fix is a separate node pool. You can learn it from this thread on Reddit.

Otherwise, it’s a good article and a good thread.

#kubernetes #aws #karpenter
blog.appliedcomputing.io Karpenter's consolidation behaviour is counter-intuitive OK, this story starts off with a colleague messaging me at 9am on a Tuesday morning: “Hey, drmorr, Karpenter is behaving a little bit strangely in this cluster, could you take a quick look?”
  • ❤ 4
  • 👍 1
Post #2953 1.31K
An article from Anthropic on the AI agents evaluation strategies, basically test strategies for AI agents.

It's an interesting read, even if you do not create your own agents. There are also some interesting points in the difference of evaluating a coding agent vs a communication agent (customer support, etc.), as well as examples of the metrics you can use for that.

Besides, you can use the ideas from this article not only for the agents, but also for skills, if you develop any.

#ai #testing
Anthropic Demystifying evals for AI agents
  • 👍 3
Post #2952 1.3K
​​For today's Donations Monday, I'd like to remind you about a big fundraiser by DOU for the 2nd separate corps of the National Guard of Ukraine «Хартія».

The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.

Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ

#donations #Ukraine
  • ❤ 6
Post #2951 1.54K
An interesting article by Coroot on reproducing split brain on CloudNativePG.

Regardless of whether or not you use CloudNativePG, it's a good example of chaos testing. Chaos testing is about an experiment: here they define a clear gap in certainty (what would happen if?) and tested it.

The article also has some recommendations on CloudNativePG configuration, if you're using it.

#databases #postgres #kubernetes
Coroot Reproducing split brain on CloudNativePG | Coroot Blog We partitioned the primary of a default CloudNativePG cluster and measured how long it kept accepting writes after a replacement was promoted, what that did to the data, and which settings shrink the window.
  • 👍 1
  • 🔥 1
Post #2949 1.41K
You likely already know about the recent security incident that involves OpenAI and Hugging Face, when a model escaped its containment to cheat an evaluation test and hacked Hugging Face in the process. Now, postmortems are available from both OpenAI and Hugging Face.

- Initial incident disclosure by Hugging Face
- Postmortem by Hugging Face
- Postmortem by OpenAI

This is quite interesting read. On one hand, in the words of Hugging Face: "AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed. Defending an online platform now means treating the data and model surface as a first-class attack surface, and using AI on defense to keep pace."

But also: "...we had no admission policy rejecting privileged or hostPath pods, and the CSI driver's ClusterRole granted pod creation cluster-wide..."

This highlights not only the need to develop new defense methods and strategies, but also the fact that the good-old multi-layer security approach is important, and old boring things work as well.

#ai #security
huggingface.co Security incident disclosure — July 2026 We’re on a journey to advance and democratize artificial intelligence through open source and open science.
  • 👍 7
  • ❤ 2
Post #2948 1.37K
Being a good communicator is becoming more and more important in the world, where more and more code is generated.

15 principles for managing up is an article with some tips that can help you not only get on a better foot with your manager, but also to become a better colleague in general.

I personally so not like the term “managing up”, but I guess you have to call it somehow.

#culture #mgmt
Weskao 15 principles for managing up When your boss loves you, you have options. Here's how to help your manager get what they need, so you get what you want.
  • 👍 4
Older posts →
Threads Profile ViewerView any public Threads profile without an account.Open ThreadLook →Writing with AI? Make it sound human.Metric37 rewrites AI drafts so they read naturally. Free AI detector, 1,500 words free.Try Metric37 →