Let’s examine some integrations I currently use

UNLOADing customer/partner/chauffeur labels to S3 bucket files

Let’s say we want to exchange some data with other teams be it Backend, Financial Service, Frontend. You may also treat it as publishing data for whom it may concern, so there could be multiple consumers.

Could be easily automated it with dbt + Redshift UNLOAD:

– Create a model with desired attributes and rows
– Materialize it as a view
– Add dbt post-hook with UNLOAD script
– Schedule it on a daily/hourly basis

As a result:

– No actual excessive data is stored – materialized='view'
– Export is always up-to-date as it is scheduled with dbt jobs
– UNLOAD is performed only with production runs
– For consumers snapshot or history is available
– Multiple file formats supported: CSV, JSON, Parquet