Tongyi Lab (Alibaba) released VimRAG, a framework for agent-based RAG that works with text, images, and videos.
The project improves on last year's VRAG-RL and addresses the challenge of multimodal RAG: visual data are token-heavy but semantically sparse, and the classic ReAct history clutters the context with noise and triggers repeated useless queries to the search engine.
➡️ How it works?
Instead of a log of observations, VimRAG models reasoning as a dynamic directed acyclic graph. Each vertex stores a sub-query, an agent's action, a text summary, and a bank of visual tokens.
Edges capture logical dependencies between steps. Such a graph allows the agent to distinguish a dead-end branch from a new hypothesis and avoid getting stuck in repetitive search loops.
➜ Graph-Modulated Visual Memory Encoding operates on top of the graph.
Visual token budgets are allocated based on outgoing degree in the graph, exponential temporal decay (simulating forgetting), and recursive feedback from descendants.
Key fragments receive high resolution, while secondary frames are compressed or discarded. For videos, VLM's ability to tie content to a timeline (extracting key frames by timestamps) is utilized.
➜ Third component is Graph-GPO.
GGPO constructs a critical path from the root to the answer and imposes a gradient mask, excluding dead-end nodes from positive examples and valuable retrievals from negative ones.
On training curves, this leads to faster convergence than the base GSPO without pruning.
➜ Tests
☞ VimRAG outperforms Vanilla RAG, ReAct, VideoRAG, UniversalRAG, and MemAgent on 9 benchmarks (HotpotQA and SQuAD to SlideVQA, MMLongBench, LVBench, and XVBench).
☞ On Qwen3-VL-8B-Instruct, the average score rises from 43.6 to 50.1, and on the 4B version, from 40.6 to 45.2.
However, the average path length is lower than in ReAct and Mem1: structured memory consumes fewer actions per response.
➜ In repository:
☞ the VRAG-RL training framework, a demo of VRAG on the test Qwen2.5-VL-7B-VRAG via vLLM (requires A100 80GB);
☞ a demo on the API Qwen3.5-Plus via DashScope (with visualization of the reasoning DAG and extended rizomics).
The search engine is built on FAISS and supports GVE-3B/7B and Qwen3-VL-Embedding-2B/8B embeddings. Images, PDFs (via conversion), and segmented videos can be indexed.
The VimRAG training code will be released after Alibaba's internal review.
Arxiv, Model, GitHub
#AI #ML #RAG #VRAG #TongyiLab
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore