В этом блоге я публикую свои выводы и мнения на работу в Data:
— Data Integration
— Database engines
— Data Modeling
— Business Intelligence
— Semantic Layer
— DataOps and DevOps
— Orchestrating jobs & DAGs
— Business Impact and Value
Post #81
326
I have updated Wheely's production workloads to a new version of dbt==0.21.0
Along with major improvements to performance, stability, and speed we now have:
– A dbt build command for multi-resource runs
– Handling for column schema changes in incremental models
– Defining configs in all the places you’d expect
An average prod job definition looks like: dbt seed + dbt run + dbt snapshot + dbt test
Now with single dbt build command its going to be really simplified and convenient, building resource by resource, from left to right across your DAG.
New on_schema_change parameter enables additional control when incremental model columns change. Possible strategies are:
– ignore (default): new column will not appear in your target table.
– fail: Triggers an error message when the source and target schemas diverge .
– append_new_columns: Append new columns to the existing table.
– sync_all_columns: Adds any new columns to the existing table, and removes any columns that are now missing.
But nothing comes all good, I’ve faced with a couple of bugs these days:
– One that broke my materialization macro with new dispatch logic
– Serializable isolation violation (unrelated to dbt)
I will describe them in next posts.
Meanwhile read more about upcoming dbt v1.0 in December 2021 !
#dbt #release
dbt Community Forum Release: dbt Core v0.21 (Louis Kahn) Updates [Sep 27] v0.21.0 (final) is available for production use. [Sep 27] v0.21.0-rc2 is available. It includes small bug fixes and bumps to schema versions for changed metadata artifacts. [Sep 20] v0.21.0-rc1 is available for prerelease testing. 🔔 Who… Along with major improvements to performance, stability, and speed we now have:
– A dbt build command for multi-resource runs
– Handling for column schema changes in incremental models
– Defining configs in all the places you’d expect
An average prod job definition looks like: dbt seed + dbt run + dbt snapshot + dbt test
Now with single dbt build command its going to be really simplified and convenient, building resource by resource, from left to right across your DAG.
New on_schema_change parameter enables additional control when incremental model columns change. Possible strategies are:
– ignore (default): new column will not appear in your target table.
– fail: Triggers an error message when the source and target schemas diverge .
– append_new_columns: Append new columns to the existing table.
– sync_all_columns: Adds any new columns to the existing table, and removes any columns that are now missing.
But nothing comes all good, I’ve faced with a couple of bugs these days:
– One that broke my materialization macro with new dispatch logic
– Serializable isolation violation (unrelated to dbt)
I will describe them in next posts.
Meanwhile read more about upcoming dbt v1.0 in December 2021 !
#dbt #release
