Airbyte deployed Clickhouse destination which I already use to gather data from multiple sources.
By default it replicates all the data as JSON blobs (all the attributes inside one String field)
To get it flattened you either have to do it yourself or use Airbyte normalization.
1. Flattening manually with JSON functions
JSONExtract(_airbyte_data, 'id', 'UInt64') as id➕ Could be tricky and exhausting if you have a lot of attributes.
➖ Works extremely fast
2. Airbyte normalization (= dbt underneath 😉)
➕ It will flatten all your data automatically
Technically it is auto-generated dbt project
➖ Still a little bit buggy and looks like a work in progress.
I almost managed to get it done, but I use Yandex’ Managed Clickhouse which demands you use SSL / Secure connection.
Unfortunately, Airbyte’s dbt profiles.yml is hard-configured to secure: False at the moment.
I might create a PR to fix this when I have some time.
#airbyte #clickhouse #dbt