What fundamental axioms and unchangeable principles exist in data engineering and data modeling?
Consider Euclidean geometry as an example. It's an axiomatic system, built on universal "true statements" that define the entire field. For instance, "a line can be drawn between any two points" or "all right angles are equal." From these basic axioms, all other geometric principles can be derived.
So, what are the axioms of data engineering and data modeling?
I asked ChatGPT about that and it gave this list:
▪️ Data exists in multiple forms and formats
▪️ Data can and should be transformed to serve the needs
▪️ Data should be trustworthy
▪️ Data systems should be efficient and scalable
Classic ChatGPT, pretty standard, pretty boring 🥱. Yes, these are universal and fundamental rules, but what can we learn from them?
Here is what I'd call axioms for myself:
🔹 Every table should have a primary key which is unique and not empty (dbt tests for life 🙂)
🔹 Every column should have strong types and constraints (storing data as STRING or JSON is ouch)
🔹 Data pipelines should be idempotent (I don't want to deal with duplicates and inconsistencies)
🔹 Every data transformation has to be defined in code (otherwise what are we doing here)
Post #444
4.26K
- 👍 4
- ❤ 1