TechLead Bits

The Cornell Note Taking System

When I study, I like to take notes and write summaries about what I’ve learned. This helps me rethink what I’ve read and provides a short description of useful ideas that I can revisit from time to time.

Recently, I discovered the Cornell Note-Taking System. It was developed by Walter Pauk, a professor at Cornell University, in the 1950s. The system was initially designed to improve the efficiency of college-level students (I wish someone had shown it to me when I was a student!).

The Cornell Note-Taking System is very simple and consists of three parts: notes, keywords/questions, and summary. Here’s how it works:

📍Divide the paper into two columns: the questions column (1/3 of the width of the page) and the notes column (2/3 of the width of the page). Leave 5-7 lines at the bottom of the page for the summary.
📍During study, make notes in the notes section. It’s recommended to keep one line between different statements and two lines between different chapters. This is how you should work during lectures and readings.
📍Add key terms and relevant questions to the questions column. This helps emphasize important points.
📍Write a summary in your own words to describe what you’ve learned. A good tip is to imagine you’re explaining it to one of your friends.

Various studies show that the Cornell Note-Taking System is more effective than other styles when students need to apply learned knowledge in practice. I’ve tested this approach and can confirm that it works much better than standard note-taking. It requires to rework the material, which improves the level of retention and understanding. Now it’s my primary method for taking notes.

#softskills #productivity

And now there is a more entertaining part of the book review: a sketchnote🔥 to help simplify and remember what the book is about.

#architecture #engineering #booknook #sketchnote

Modern Software Engineering

Finally, I’ve finished reading Modern Software Engineering by D. Farley 📚. It took me longer than expected, as I spent a lot of time on the first two parts of the book, which focus more on general concepts rather than practical advice, making the initial experience a bit tedious for me.

Let's dive into the book to see what it's about. It is divided into four major sections:

Part I: What is Software Engineering?
The author provides a definition of software engineering that I find really great:

Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software.

Each word in this definition is significant. Reflecting on them, the author concludes that we must manage the complexity of the systems we create and continuously learn new things and adapt to them.
Engineering is not just coding—it includes all aspects needed to create software: processes, tools, and culture. We should apply engineering practices everywhere to make effective decisions. And we need metrics to measure the effectiveness of those decisions. The author suggests the following metrics:
📍Stability: change failure rate, recovery time
📍Throughput: lead time (from idea to production), deployment frequency

Part II: Optimize for Learning
To effectively learn and adapt to changes, we need to:
Working iteratively
📍Employing fast, high-quality feedback
📍Working incrementally
📍Being experimental
📍Being empirical
As you can see, these principles align closely with modern agile practices.

Part III: Optimize to Manage Complexity
The fundamental principles for managing complexity include:
📍Modularity
📍Cohesion
📍Separation of concerns
📍Abstraction
📍Loose coupling
While these concepts are familiar to most programmers, the book provides a valuable review with practical examples and techniques for better understanding and application.

Part IV: Tools to Support Engineering in Software
To apply an engineering approach in practice, the following tools are suggested:
📍Testability
📍Deployability
📍Speed
📍Controlling variables
📍Continuous delivery
The author discusses specific tools and how they relate to fundamental principles. For example, achieving testability requires good abstraction, modularity, and separation of concerns.

For me, the book doesn't introduce new concepts but rather reinforces my current understanding of how we should engineer our systems. To conclude this review, I’d like to share a quote from the book that serves as a good rule for daily work:

If our ‘software engineering’ practices don’t allow us to build better software faster, then they aren't really engineering

#architecture #engineering #booknook

Pinterest Ad System Redesign

A recent article on the Pinterest Engineering blog, titled "Redesigning Pinterest’s Ad Serving Systems with Zero Downtime," discusses the complete overhaul of Pinterest's recommendation system.
The ad-serving system is crucial to Pinterest's business, serving as the primary revenue channel. The previous system, known as "Mohawk," was implemented in 2014. However, as the company grew, Mohawk accumulated technical debt and complexity, making incident troubleshooting and bug analysis increasingly challenging.

The following architectural issues were identified as motivations for the reimplementation:
📍High coupling between business and infrastructure logic
📍Lack of modularization and ownership: Code from the same packages and files was owned by different teams.
📍No guarantees of data integrity
📍Unsafe multi-threading: Lack of error handling and potential race conditions.

The new design principles include:
📍Extensibility: Enable the addition of new features and the deprecation of old ones.
📍Separation of Concerns: Organize logic into separate modules, each owned by the appropriate teams.
📍Safety: Ensure the safe use of concurrency and enforce data integrity rules.
📍Development Velocity: Provide well-supported development environments and facilitate easy testing and debugging.

The team spent 2 years implementing the new version of the recommendation system - AdMixer. Rewritten in Java, like other Pinterest services, this new system helps unify the technological stack. It has been in production for three quarters now without significant outages. Additionally, it was designed to run on AWS Graviton instances (ARM architecture) to reduce AWS usage costs (more about that in Multi-Arch Images post ).

What I can say is that a complete system rewrite is a very expensive endeavor. It would be interesting to know the costs of this two-year project, as not all companies can afford such an exercise. An eight-year lifespan is not particularly long for a software system, indicating that serious architectural mistakes and unresolved technical debt prevented the previous system from evolving effectively.

Let's be honest: engineers often prefer to rewrite a system, especially one created by others😀. It’s fun, cool, and easier, and it makes a nice addition to a resume. However, it is far more challenging to build a system with the right architectural principles that support evolutionary rather than revolutionary development (do not confuse it with over-engineering!).

In any case, the Pinterest engineering team did a great job: they implemented a new approach, successfully delivered it to production, accelerated business features delivery, and increased developers satisfaction. I hope the new service incorporates lessons learned and it will be extensible for future growth and business needs without requiring a full rewrite.

#architecture #usecase #refactoring

Structure Eats Strategy

Many are familiar with Conway’s Law, which states that the architecture of a system reflects the communication structure of the organization that created it. If architecture mirrors the organizational structure, then having a solid structure is crucial, right? Jan Bosch explores this idea in his article "Structure Eats Strategy". He posits that many organizations prevent their growth due to ineffective organizational structures, communication, and distribution of responsibilities.

Bosch identifies four fundamental building blocks within any organization:
📍Business (B): Defines business strategy, revenue, and innovations.
📍Architecture (A): Establishes the technology stack and system structure.
📍Processes (P): Outlines activities and ways of working.
📍Organization (O): Defines departments, teams, and responsibilities.

Most organizations operate in an OPAB model (organization-first), meaning that the existing organizational structure dictates processes, leading to an accidental architecture shaped by the company's past.

Bosch argues that the BAPO model (business-first) is significantly more effective. This model begins with defining the business strategy, which shapes the architecture. The architecture then forms the processes and tools, and only after these elements are in place is the organizational structure defined.

The article offers a valuable framework for understanding the relationship between organizational structure and architecture. Structure should be a tool to help the business achieve its goals and guide architecture in the right direction for the future. Unfortunately, there are many examples where existing team boundaries lead to flawed architectural decisions. This results in support overheads, lower maintainability, reinvented wheels, and higher costs for the business.

#architecture #management

Architectural Principles

Whether you set them or not, you and your team already use architectural principles. These principles may not be obvious, but they influence technical decisions, from writing any piece of code to preparing complex designs. The challenge is that each developer may have their own set of principles based on their previous experiences, best practices, or books they have read. This can lead to inconsistent system behavior across different modules, debates during code reviews, and the need for additional approvals to make decisions.

Let’s define what an architectural principle is. Eoin Woods, author of Continuous Architecture in Practice, offers a definition I like: an architectural principle is

a declarative statement made with the intention of guiding architectural design decisions to achieve one or more qualities of a system.

In essence, architectural principles are guidelines that establish a common framework for the team, enabling them to make informed decisions independently of any central authority. These principles not only enhance team responsibility for their actions but also streamline the achievement of business and technical goals. For architectural principles to be effective, it is crucial that the team participates in their development and accepts them.

Relationship Between Goals, Principles, and Decisions:

Goal -> Requirements -> Principles -> Decision

Recommendations for defining Architectural Principles:
📍Simplicity: Ensure that principles are straightforward and do not require additional context for understanding.
📍SMART Criteria: Define principles that are Specific, Measurable, Achievable, Realistic, and Testable.
📍Practical Guidance: Principles should be practical and usable as implementation guidance.
📍Relevance: Focus on the most significant principles that, if not properly defined, could lead to poor decisions.
📍Conciseness: Keep the list short (5-7 items) to reduce cognitive load.

Some examples of good principles:
✔️Degrade Gracefully: Design the system to continue functioning even when some components fail, providing a lower-quality service instead of a total failure.
✔️Self-Healing: Ensure that failed actions are continuously retried until they succeed
✔️Design to Be Monitored: Build systems that are self-diagnosing and can be easily monitored.

Architectural principles set boundaries and guidelines for teams, allowing the freedom to make independent decisions. This helps maintain consistency, improve decision-making, boost autonomy, and achieve technical and business goals.

#architecture #documentation

Flaky Tests Overhaul

If you've got a substantial set of tests, chances are very high that you've encountered situations where some test outcomes fluctuate between runs, even when there are no changes in the code. This inconsistency is called "flakiness".

Teams often may find themselves repeatedly retrying pipelines containing flaky tests (timeouts are also considered a form of flakiness), trying to make the pipeline green. However, this process often results in wasted engineering hours and CI resources. As the code base and number of teams grow, managing flaky tests becomes increasingly challenging, leading to more potential issues and accumulating technical debt.

The Uber engineering team recently published an article detailing their approach to improve CI stability and address the issue of test flakiness.

Key points:
📍Separate service Testopedia is introduced to visualize history of test execution and test performance characteristics
📍Testopedia is language/repo-agnostic. It operates with the term ‘test entity’. Each test entity is uniquely identified by a “fully qualified name” (FQN) that usually includes a full test address in the repo.
📍Tests can be grouped into realms, each realm is owned by some responsible team.
📍Testopedia analyzes test execution stats (including flakiness, reliability, staleness, execution time) , groups problem tests and triggers a JIRA ticket with the deadline to fix.
📍GenAI integration is a future step to auto-generate fixes for flaky tests. It’s under research now.

As a result, the authors noted that implementing the Testopedia approach significantly improved the reliability of CI and reduced the number of retries. If this tool were available as an open-source project, I would certainly give it a try, but unfortunately, it's not.

However, in the absence of such a tool, what steps can we take on our own to address this issue? Here are some suggestions:
📍Visualize pipeline health by implementing simple monitoring of CI statistics, including the number of retries, execution time, and other relevant metrics. To improve something it must be measurable.
📍Treat problematic tests as work items with clear deadlines for resolution.
📍Prioritize CI issues, recognizing them as critical technical debt that will require attention anyway
📍Implement measures to make retries more difficult or even impossible (validations, webhooks, etc.)
📍Clearly define roles and responsibilities for maintaining CI stability; otherwise there is a risk of collective irresponsibility.

#engineering #ci

Dual-Write Problem

In the distributed world, it's often the case that two external systems need to be synchronized and updated simultaneously to maintain consistency. It’s called the dual-write problem. A classic example of this is when data needs to be stored both in a database and in Kafka.

So what's the best solution for this problem? What are common pitfalls and how to avoid them? These questions are addressed in the article 'Solving the Dual-Write Problem` by W.Waldron.

Identified antipatterns:
📍Relying on operation order
📍Wrapping dual writes into a database transaction
📍Retrying failed operations

Possible solutions:
📍Transactional outbox pattern. In this approach, an outbox table is set up in the database. Changes are made to the target tables and the outbox table within the same transaction. A separate process then reads the outbox table rows and sends them to Kafka, retrying if there's a failure.
📍Change Data Capture (CDC) if supported by the database. Modification of p.1
📍Event sourcing. Event sourcing: Every change is recorded in the database as an event. Since each event is written to a single row in a single table, transactions are unnecessary. A separate process can then read these events and send them to Kafka.
📍The listen-to-yourself pattern. Any change is directly sent to Kafka. A separate process can listen to these events and use them to update the database, retrying as needed. Database will be eventually consistent.

The core concept behind these solutions is to divide writes into two separate processes and establish a dependency between them. While this isn't an exhaustive list of all potential solutions, it provides a solid set of practices to begin with.

#architecture #systemdesign #patterns

The Motive

`Please, don’t be a leader, unless you’re doing it for the right reason, and you probably aren’t!` It may sound provocative, but Patrick Lencioni starts his book 'The Motive: Why So Many Leaders Abdicate Their Most Important Responsibilities' with this very idea. Released in 2020, this is one of his latest works where he challenges traditional views on leadership, exploring why individuals should choose to take on leadership roles.

The author defines two primary motives that drive leaders:

1️⃣ Reward: Leaders driven by rewards primarily seek personal gratification and enjoyment in their position. They avoid mundane tasks, focusing solely on activities that interest them. They actually dislike managerial tasks and would likely find greater satisfaction and effectiveness in other areas.

2️⃣ Responsibility: Leaders with a sense of responsibility prioritize the well-being and growth of their team. They take unpleasant tasks, recognizing their duty to their employees and organization.

While most individuals possess elements of both motives, Lencioni argues that one typically dominates, significantly influencing leadership effectiveness. According to him, responsibility-oriented leaders tend to achieve greater success.

Lencioni identifies five crucial activities that reward-oriented leaders tend to avoid:

1️⃣ Developing the leadership team: Taking personal accountability for the growth and development of team members.
2️⃣ Managing subordinates (and making them manage theirs): Providing guidance, sharing knowledge, and offering mentorship.
3️⃣ Having difficult and uncomfortable conversations: Addressing complex and uncomfortable issues promptly and effectively.
4️⃣ Running great team meetings: Facilitating productive discussions and decision-making.
5️⃣ Communicating constantly and repetitively to employees: Keeping the team informed about organizational vision, goals, decisions, and challenges.

For me, it was a valuable review that brought particular aspects into focus. After looking closely at how I lead, I realized that handling tough conversations (point 3) isn't my strongest skill. So, I marked it as an area to improve upon. But it's nice to know that this challenge isn't unique to me, as the book mentions 😉.

So, revise those unpleasant tasks, clarify your motives and be a good leader.

#management #leadership #booknook

Coupling or Uncoupling?

We all start our careers knowing that our code should be loosely coupled and have high cohesion. But can we ever achieve fully uncoupled code in a system? Not really. Coupling determines how freely we can make changes within a system, and different types of coupling may produce totally different effects.

Michael Nygard dives deep into that topic in his Uncoupling talk where he explains what coupling is, how to analyze it and what we as engineers can do to make the systems more resilient to change.

So, let's break it down. The author talks about a few types of coupling:
📍Operational. Consumer cannot run without a provider. For example, a service might fail if it can't access the database.
📍Development. Changes in the producer and consumer need to be synchronized and delivered together
📍Semantic. Changed together because of shared concepts.If a concept changes due to new business requirements, those changes need to be reflected in all downstream systems.
📍Functional. Changed together because of shared responsibility.
📍Incidental. Change together for no good reason. The E-Gate system failure serves as a prime example of incidental coupling, where an update in one system triggered a failure in another due to a shared network.

By strategically adjusting the levels of different types of coupling, we can effectively manage the impact of our changes.

#architecture #systemdesign

Why Don’t People Do What You Expect

As leaders, at any level, we delegate tasks to our subordinates and control their execution. During these times, there's an expectation that everything will be executed exactly as envisioned. Surprisingly, this isn't always the case. In fact, there are cases where some team members consistently fail to complete assigned tasks. This can be frustrating, leading to increased tension between team members and their leader. In the worst-case scenario, it may result in a team member being released from the project or even fired.

But what causes these issues? And is there a way to address them?

Let's consider one of the fundamental assumptions of management: that people always act in the best possible manner (excluding deliberate sabotage, of course). This "best possible manner" depends on numerous factors, ranging from the team atmosphere to personal health and family relationships.

So we must try to understand the reasons behind the unexpected behavior. Generally, these reasons can be grouped into four categories:
1️⃣Don’t understand. The task or its purpose isn't clear, leading to a lack of motivation to do it.
2️⃣Not able to do. There's a lack of skills or competencies necessary to perform the task.
3️⃣Cannot do it. There's a lack of resources, both physical and mental, to carry out the task. Personal issues may decrease the ability to focus on work tasks effectively.
4️⃣Don’t want. The task may not align with the individual's competencies or goals, leading to a lack of motivation to do it.

As leaders, our first step is to identify the correct reason. Often, a one-on-one conversation with appropriate questions can shed light on the situation. In most cases, we have the power to address the problem, help our team members and make the team stronger.

#management #leadership

How Instagram Scaled to 14 Million Users

I enjoy studying various real-life scenarios where technologies and patterns are applied to solve practical problems. These examples often inspire me with new ideas for my day-to-day work.
So today we’ll review a really nice video that offers valuable insights into Instagram's growth journey and techniques they use to serve 14 million users.

Key principles of Instagram architecture:
✔️ Keep things simple
✔️ Don’t reinvent the wheel
✔️ Go with already proven and solid technologies

Instagram initially used several Django instances, a PostgreSQL database on EC nodes, and Nginx load balancers. When the database grew too large, they split the data into multiple shards.

One of the toughest challenges was generating IDs and determining the correct shard to handle each ID. The Instagram team decided to use the Snowflake ID approach with some modifications:
- 41 bits for unix time in ms
- 13 bits for shard id
- 10 bits for the auto-increment sequence

Other interesting aspects of Instagram's architecture include:
- Their notification system (e.g., for likes, comments, new posts) is powered by a Gearman job server with around 200 workers to handle these tasks.
- Files are stored on S3 servers located worldwide to keep data close to users
- The technology stack also includes Apache Solr, Memcache, Pingdom, PagerDuty, and Sentry.

To me, the system seems well thought-out and carefully built, focusing on simplicity and scalability. Keeping it simple is a great architectural principle—it makes maintenance easier and the system more resilient.

#architecture #scaling #usecase

What We Can Learn from UK E-Gate Failure

In the beginning of May, the UK E-Gate system experienced a 4-hour outage. This system handles automatic passport control, with over 270 automated gates spread across 15 airports and rail ports in the UK. These gates use biometric data and advanced facial recognition technology to allow entry into the country. The failure caused major airports to cease operations, leading to delays for passengers attempting to cross the UK border.

The outage was not the result of a cyber attack but rather a network issue. The Home Office was conducting a software update that exceeded the data limits outlined in the contract, prompting the network provider to shut down the service. This network was also used for a connection between the gates and the database used for passport verifications.

Dave Farley addressed potential architectural flaws in the E-Gate system in a recent video and offered recommendations for designing distributed systems with failure in mind.

So how E-Gate architecture can be improved to prevent such failures:
- Add redundancy (reserved network channel)
- Cache copies of the data
- Limit the ways things can fail (e.g., use local network instead)

If something can go wrong, it will go wrong. And we as engineers must be prepared for potential failures, designing systems that can handle problems instead of trying to make them perfect.
Here are some general recommendations to consider when building distributed systems:
✔️Assume things can go wrong
✔️ Limit the blast radius of failure
✔️ Use Chaos Engineering techniques

Just a few days ago on May 30th, the E-Gate system encountered another failure🤦‍♂️, resulting in disruptions at railway stations and leaving passengers queued for approximately 4 hours. It appears that the fundamental issues with the system remain unresolved.

#architecture #reliability #resilience #usecase

Caching CheatSheet

In the past weeks, there have been two big articles discussing caching strategies:
- Mastering Caching in Distributed Applications
- 9 Caching Strategies for System Design Interviews

So it's a good time to review the patterns we use.

Let’s start with the definition of what caching is. I really like the following one:

Caching is the action of storing data in a temporary medium where the data is either cheaper, faster, or more optimal to retrieve rather than retrieving it from its original storage

Caches can be local or distributed.

5 implementation strategies:
✔️Cache-Aside (Lazy-Loading): The app controls reading and writing to the cache. If data's in the cache, it's used; if not, it's fetched from storage and added to the cache.
✔️Write-Through. Both the cache and the database get updated together within the same transaction when you write data. Reading happens only from the cache.
✔️Write-Around. Data is written directly to storage. Usually, it's combined with Cache-Aside for reading.
✔️Write-behind (write-back). Data is first written to the cache and then sent to the datastore asynchronously. The cache product usually handles syncing with the datastore.
✔️Read-Through. The cache is the main place to get data from. If it's not there, it's fetched from storage. Unlike Cache-Aside, the cache product, not the app, decides when and how to fetch data.

Cache invalidation can be performed time-based or event-based.

Cache-eviction strategies:
- Least Recently Used (LRU)
- First In First Out (FIFO)
- Least Frequently Used (LFU)
- Time To Live (TTL)
- Random Replacement

Promising paths for further development in that field:
- AI and Machine Learning-Driven Caching. It can enhance caching mechanisms by predicting data usage patterns and preemptively caching data based on the needs.
- In-Memory Data Grids. They not only cache data but also provide a range of data processing capabilities, real-time analytics, and decision-making directly within the cache layer.

#architecture #patterns

Multi-Arch Images

Public cloud providers actively promote migrating to ARM instances, offering potential cost savings of up to ~20%. For instance, AWS suggests their m7g ARM instances priced at $8.617 per hour, compared to $10.644 per hour for equivalent AMD instances. While the performance of ARM instances may vary depending on the application's loading profile, many benchmarks indicate quite positive outcomes:
- ARM vs Intel on Amazon’s cloud
- Faster and Cheaper: ARM Graviton vs Intel and AMD x86 AWS EC2
- Performance Analysis for Arm vs x86 CPUs

As pragmatic engineers who love cost-effective solutions, we see the value in exploring instances with different architectures to optimize both cost and performance.

The concept sounds promising, but what technical preparations are needed for that?

Let's delve into some theory.

By default, when building containers, they are targeted for the same architecture as the local CPU. As a result, running AMD-built containers on ARM nodes, and vice versa, causes errors. This is where multi-arch images come into play.

A multi-arch image looks like a single image with a single tag, but it has a collection of images designed for multiple architectures. How does this function? Each Docker image is defined by a manifest—a JSON file containing all the information about the image. It includes references to each of its layers, their corresponding sizes, the hash of the image, its size and also the platform it's intended for.

For multi-arch images the manifest contains a list of manifest, enforcing the container runtime to select the appropriate image based on the underlying platform architecture.

Presently, two primary manifest standards exist: the Docker manifest version 2 and the OCI Image Index Specification.
Personally, I prefer the OCI standard as it presents a more generic approach to work with multi-arch images.

Many articles explain how to create these images, so I won't cover that here. Just remember, making multi-arch images requires adoption from the build process and a bit more storage in the registry.

In conclusion, using multi-arch images can lower operational costs in public clouds, but it requires making extra changes to integrate them into CI/CD processes. Do you need that in your project? It depends 😉

#engineering

Channel photo updated

Leaders Eat Last

`Leaders Eat Last` by S. Sinek is highly recommended by various management sources, so let’s check what it has to offer.

In my experience as a leader, I've often relied on the gut feeling for certain practices. For instance, I've long believed in the importance of offline communication—the handshakes, conversations over coffee (or something stronger), and discussions in informal settings like smoking rooms. These interactions significantly enhance collaboration in ways that virtual meetings, even with cameras, cannot replicate.

How does this relate to the book? It proves my observations

Key points:

📌Safety: One of the fundamental human needs is the need for safety. When we feel secure within a team or company, we can work effectively towards achieving its goals. Otherwise, resources are diverted towards personal protection. The author refers to this concept as "The Circle of Safety."

📌Endorphins, dopamine, serotonin, and oxytocin: These chemical elements continue to influence our behavior, just as they did thousands of years ago. Understanding their principles enables us to stimulate the desired behaviors.

📌Leadership: Every team, whether formal or informal, has a leader. This is inherent in our nature and evolutionary processes. While self-organizing teams are often discussed, there is inevitably a leader within each.

📌Abstraction: We tend to emotionally react more strongly to a specific story about an individual death or disaster than to a report about hundreds of people dying in a news headline. This is due to abstraction, which can greatly skew our perception of the world. This section of the book presents shocking psychological research that even scientifically explain one of the most horrific events in modern history—the Holocaust.

📌Based on information above the author outlines the following lessons for leaders:

✔️ A good corporate culture can cultivate good leaders, while bad leaders can produce a bad culture
✔️ The values of a leader strongly influence the culture of the team
✔️ Integrity is essential for building trust
✔️ Personal relationships matter
✔️ Lead people, not numbers
✔️ Being a leader is not a reward; it's a responsibility

The book presents much more interesting examples and psychological research on the principles of human behavior. Sometimes it delves into philosophical reflections on the state of modern leadership and its future. Even with that, I definitely recommend the book.

#booknook #management #leadership

Queues for Kafka

`Kafka Queues: Now and in the Future` breaks down how consumer groups work today and what updates are coming in future Kafka versions. Future is the most interesting part actually.

So, Kafka is exceptionally effective for streaming large volumes of data through topics and partitions. Consumer groups manage this data by dividing the workload based on the number of partitions. While this method offers high performance, it also presents notable limitations:
- Consumers are exclusively assigned specific partitions.
- Parallel data processing is restricted by the number of partitions in a topic
- There's no option for individual acknowledgment or message redelivery
- Messages cannot be marked as rejected
- Message processing times can vary unevenly

So, KIP-932 has been introduced to enhance Kafka's queue capabilities. This proposal suggests a 'share group' with a `share-partition` abstractions.
Key changes:
- A share-partition can be assigned to any number of consumers, removing the scaling limitation by the number of partitions.
- Each message has a state: available for processing, acquired, acknowledged, or rejected.
- Consumers receive available messages from shared partitions, transitioning the message state to acquired
- Messages not acknowledged within a specific timeframe are returned to the available state.
- If the number of delivery attempts exceeds a defined threshold, a message is marked as rejected, halting further delivery attempts
- Consumed messages are not guaranteed to be ordered

Currently, the KIP is in the Accepted state and is planned for release in Kafka 4.0.

#news #architecture #kafka

The Price of the Decision

Let’s talk about the financial implications of our technical choices. Understanding this aspect is a key; the less we spend on our solution, the more funds we retain for the business.

Consider a simple example to illustrate the significant impact of a single decision:

Imagine we're integrating with a Data Lake through Kafka, leveraging Debezium to stream data from the database. Our project is relatively small, with approximately 100 tables for streaming. Opting for AWS to meet our infrastructure requirements, we'll utilize MSK (Kafka) service and leverage its well-defined pricing model in our calculations. Also we need AWS recommendations for broker sizes.

Alright, we'll intentionally skip all other requirements and opt for two diametrically opposed implementation options:
1️⃣Stream data from all tables to a single topic
2️⃣Stream each table to a separate topic

Option 1
Let’s take standard replication factor 3 and 10 partitions for the topic.
1 env cost:
- 1 topic * 10 partitions * 3 replication factor = 30 partitions
- Take the smallest possible configuration: 3 kafka.t3.small brokers
Price: $4.1 per month
Nice!
Usually we have more than 1 environment (dev, QA, SIT, UAT and other environment types), let’s say 10 envs in total.
10 env cost:
- 10 topics * 10 partitions * 3 replication factor = 300 partitions
- Take the smallest possible configuration: 3 kafka.t3.small broker
Price: $4.1 per month

Option 2
Let’s take standard replication factor 3 and 5 partitions for the topic as we have less data per topic.
1 env cost:
- 100 topics * 5 partitions * 3 replication factor = 1500 partitions
- Take the smallest possible configuration: 6 kafka.t3.small brokers
Price: $8.21 per months
10 env cost:
- 1000 topics * 5 partitions * 3 replication factor = 15 000 partitions
- Take the smallest possible configuration: 6 kafka.m5.4xlarge brokers
Price: $1209.6 per month

$4.1 vs $1209.6
And this price doesn't even account for any load; it's just for having all those partitions.

As techleads, it's our responsibility to make cost-effective decisions.

#architecture #tradeoffs

The Cafe on the Edge of the World

I stumbled upon 'The Cafe on the Edge of the World: A Story About the Meaning of Life' by John Strelecky while browsing the marketplace for more books to read. Although it wasn't originally on my reading list, I decided to give it a try after it was suggested along with other books I ordered.

The main character finds himself lost on his journey and stumbles upon a café in the middle of nowhere with an unusual name. What's even more intriguing is that the menu includes thought-provoking questions:
- Why are you here? (sometimes it’s “Why am I here?”)
- Do you fear death?
- Are you fulfilled?
Throughout the book, he spends time discussing these questions with the café's inhabitants, uncovering new insights about his own life.
While you shouldn't expect revolutionary ideas and simple answers, the book serves as a great opportunity to pause and reflect on our own lives—what we're doing, why we're doing it, and what we truly want to do, what makes us happy.

It's also a time for reflection for me. Consequently, I've enrolled in a sketchnoting course😲. So, hopefully, in a little while, my summaries will be illustrated😀.

#booknook #productivity #offtop