Visual ETL with VDP
VDP (Visual Data Preparation) is an open source visual data ETL tool for optimizing the end-to-end visual data processing pipeline. It involves extracting unstructured visual data from pre-built data sources such as cloud/local storage or IoT devices, transforming it into parsable structured data using Vision AI models, and loading the processed data into repositories, applications, or other destinations.
VDP streamlines the end-to-end visual data processing pipeline by eliminating the need for developers to create their own connectors, model service platforms, and ELT automation tools. With VDP, visual data integration becomes easier and faster. The VDP is released under the Apache 2.0 license and is available for local and cloud deployment on Kubernetes. VDP is built from a data management perspective to optimize the end-to-end flow of visual data with a transformation component that can import Vision AI models from different sources in a flexible way. Building an ETL pipeline becomes like assembling from ready-made blocks, as in a children's constructor. And high performance is provided by a Go backend with Triton Inference Server with powerful NVIDIA GPU architectures supporting TensorRT, PyTorch, TensorFlow, ONNX and Python.
VDP is also in line with MLOps, allowing one-click import and deployment of ML/DL models from GitHub, Hugging Face, or cloud storage managed by version control tools such as DVC or ArtiVC. CV Task's standardized output formats simplify data warehousing, while pre-built ETL connectors provide advanced data access through integration with Airbyte.
VDP supports different usage scenarios: synchronous for real-time inference and asynchronous for on-demand workload. The scalable API-based microservice design is developer-friendly through seamless integration with the modern data stack. NoCode/Low Code interfaces lower the barrier to entry into the technology, giving the Data Scientist and analyst independence from data engineering.
https://github.com/instill-ai/vdp