An open-source robot simulation framework that for the first time combines high speed and photorealistic vision.
🟢 How it actually works?
Using 3D Gaussian Splatting, integrated as a drop-in renderer in vectorized simulators (e.g., IsaacGym), GaussGym enables training visuomotor policies based on RGB images at speeds exceeding 100,000 steps per second — even on a single RTX 4090.
- Create training worlds from iPhone videos, datasets (GrandTour, ARKit), or generative videos (e.g., via Veo)
- Automatically build physically accurate scenes using VGGT and NKSR — no manual 3D modeling required
- Train navigation and locomotion policies directly from pixels, then transfer them to the real world without fine-tuning (zero-shot sim2real) — the authors have already demonstrated a robot climbing 17-cm steps
- Support for depth, motion blur, camera randomization, and other realistic effects for better transfer
All of this is fully open: code, demo, models, and even ready-made datasets on HuggingFace.
GaussGym erases the trade-off between speed and realism in robotics, making robot training from images truly scalable.
🤖 Data Science, ML & Big Data with @DataXplore