CocoIndex released first stable version of its engine for building data for long-acting agents - those that serve RAG, knowledge graphs, memory, and context in production systems.
CocoIndex is a specialized open-source incremental ETL (Extract, Transform, Load) engine for building AI systems. It is used to automate data processing and instant indexing into vector databases or knowledge graphs.
➡️ What are the Change?
The tool is popular for its ability to update information in real time: as soon as the source data changes, CocoIndex recalculates only the necessary parts of the index, eliminating the need for a full and costly re-indexing of the entire database.
➜ Main change of v1 is the complete abandonment of the DSL
The entire pipeline is now described by ordinary asynchronous Python functions that call each other. The engine continues to track changes and materialize target states, but does so via the native Python API, rather than a separate type system.
The authors were inspired by the thesis of Jeff Dean and Bill Dally from GTC 2026: agents work about 50 times faster than a human, but rely on tools designed for human pace.
Nightly rebuilds of indexes in this logic become a problem - a need for an engine that synchronizes derived data with the source incrementally, reprocessing only changed chunks and overwriting only changed rows.
➜ In addition to the abandonment of the DSL, the release brought three more changes
☞ Firstly, the engine uses Python's own type system: PIL.Image, pyarrow.Table, torch.Tensor, and any class from an imported library can be passed directly to functions without wrappers and bidirectional conversion.
☞ Secondly, Postgres is no longer needed - the engine's state is stored in a single local file. Postgres remains a full-fledged target, it just ceased to be a mandatory dependency.
☞ Thirdly, sources and targets are created at runtime: you can mount a separate target for each tenant, build a topology from the rows of a configuration table, or connect a Kafka topic via a feature flag.
➜ Core is still in Rust
All the hot logic for detecting and applying changes lives there. At the Python level, a decorator connects a function to change tracking, and a separate flag caches its result by the hash of the arguments and code: a change in a helper invalidates only those callers that actually depend on it.
➜ Contract of managed targets has been preserved
The developer declares what a table, graph, or directory should look like, and CocoIndex itself performs create/alter/drop for containers and insert/update/delete for content, including deleting orphaned objects when the schema changes. If you stop declaring an entity, it disappears from the target.
The contract works identically for Postgres, LanceDB, Neo4j, Kafka, S3, and regular files on disk.
Examples of pipelines, from embedding code in LanceDB and processing PDFs to building a knowledge graph from conversations, are in the repository on GitHub, alongwith Documentation and You Tube.
#ML #ETL #RAG #Agents #СocoIndex
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore