If you want to secure a long-lasting and successful career in data engineering, it's not enough to merely acquire proficiency in using a particular tool. Instead, you should strive to understand its inner workings at a fundamental level as well as the first principles that underly it.
"But how?" you may ask. "There are so many tools out there: Spark, Trino, BigQuery, Snowflake, etc."
That's certainly true. However, it might surprise you to discover that all of these systems share strikingly similar foundations.
For instance, they all depend on some variation of the MapReduce model to process data. They all require data shuffling between nodes for tasks like joining or grouping. They all rely on column-oriented data formats. They are all susceptible to issues such as skewed keys, the small object problem, uneven partitioning, and so on.
The key, naturally, lies in the details. What sets each tool apart is a unique set of trade-offs that its developers have chosen to make it particularly suited to address specific use cases. For instance, Trino terminates queries that exceed memory limits to prevent costly disk spills, as one of its primary objectives is low latency. Snowflake automatically handles data partitioning for a more user-friendly experience but relinquishes fine-grained control from end users. Spark offers maximum user control but may come across as a more complex tool, and so forth.
Nonetheless, if you dig into how these tools move data around you'll discover that they are not that different after all. Plus their functionalities continue to overlap and converge over time.
Therefore, my advice is to run an 'EXPLAIN' or equivalent command for every query you write and invest time in understanding the resulting output. Ensure you grasp how each part of your query maps to a specific stage within a physical plan. Use this knowledge to debug your queries. I can assure you that the expertise and experience acquired this way will be transferable to other similar tools or data warehouse vendors.
Individual tools may come and go at a rapid pace, but fundamental principles endure and change far less frequently.
Source https://www.linkedin.com/posts/izeigerman_dataengineering-activity-7110648980732080128-DKmn
Post #19
278
- 🐳 1