👍ETL and data integration with Airbyte
The key component of any data pipeline is the extraction of data. Once data is extracted it needs to be loaded and transformed (ELT). Make it easier with Airbyte, an open source data integration platform that aims to standardize and simplify the extraction and loading process. Airbyte works like an ELT, extracting raw data and uploading it to destinations. Airbyte also allows you to perform data transformation, separating them from the EL phases. It simplifies the process by creating connectors between data sources and data destinations. Airbyte is a plugin-based system where you can quickly create your own custom connector using the CDK.
If there are not enough ready-made 170+ connectors and 25+ destinations, you can develop your own. Airbyte has a built-in scheduler that provides different clock rates, AirFlow and dbt integration is supported. It is available on the K8s platform, includes octavia-cli with a YAML template for deployment, and supports near real-time CDC.
However, the framework is still in alpha and does not support IAM role-based authentication in AWS services. No built-in support for Prometheus, although Open Telemetry has been recently added. Airbyte does not provide support for user access control and replaying a specific job execution instance. And also the work can slow down with 2000+ simultaneous tasks.
https://airbyte.com/