Подготовил конспект статьи от Shopify о сетапе Airflow на 10 тысяч DAG'ов со 150 тысячами запусков в день. Сэкономит вам время на прочтении и поможет освежить в памяти в будущем.
#briefly #airflow Airflow: scaling out recommendations by Shopify
https://telegra.ph/Airflow-scaling-out-recommendations-by-Shopify-06-03
What's inside:
— Cloud Storage vs Network File System.
— Metadata retention policy.
— Manifest file.
— Consistent distribution of load.
— Concurrency management.
— Using different execution environments.
Origin: Lessons Learned From Running Apache Airflow at Scale
Post #471
3.24K
Forwarded from How to DWH with Python
Telegraph Airflow: scaling out recommendations by Shopify Shopify runs over 10k DAGs. 150k runs per day. Over 400 tasks at a given moment on average. This is a brief overview of their approach. Link to source article. Fast file access Problem: reading DAGs files from Google Cloud Storage (through GCSFuse as a filesystem…- 🔥 10
- 👍 4