What is the most cost-effective solution for data engineering?

The answer doesn't lie in a specific vendor or technology, but in understanding the use cases and setting up optimal infrastructure. This involves leveraging best practices in software and appropriately sizing hardware. In essence, there's a strong correlation between a team's skill set for a particular stack and cost-tracking initiatives.

Often, open source solutions can be more expensive than commercial offerings. Consider a scenario where a team of 10 data engineers struggles with infrastructure configuration and daily on-call duties for an open-source solution to keep pipelines operational, versus just 2 data engineers efficiently managing the same with a commercial offering.

For fun, I tried to find evidence that Snowflake is cheaper than Databricks, but I couldn't. This seems like a great niche for Snowflake's SEO optimization.
I suspect Snowflake should cost roughly the same for the same compute instance, but with Data Warehouse as a Service, there's often less focus on right optimization and query patterns.

Have you seen any evidence that Snowflake is cheaper than Databricks? How would you choose the right solution?

https://www.linkedin.com/posts/dmitryanoshin_dataengineering-dataanalytics-surfalytics-activity-7143663492347117568-JrB1