Beyond the Frame: How MIT & NVIDIA’s FoundationMotion is Teaching AI to Truly “See” Movement

Led by MIT and joined by research teams from NVIDIA, the University of Michigan, UC Berkeley, and Stanford, FoundationMotion was published on arXiv and aims to address the scarcity of high-quality, richly annotated motion data. The team built a fully automated data-generation pipeline that produces video clips, object tracks, and natural language descriptions across seven dimensions, creating a dataset of 467,000 video clips with accompanying question-answer pairs.

After fine-tuning the open-source NVILA-Video-15B model on this data, the system achieved 91.5% accuracy in autonomous-driving scene understanding, outperforming Gemini-2.5-Flash at 84.1% and Qwen-2.5-VL-72B at 83.3%. The approach emphasises data quality over model size and includes four stages: video preprocessing, object detection and tracking with Qwen2.5-VL and SAM 2, language description generation with GPT-4o-mini, and automatic question-answer generation.

MIT and NVIDIA have committed to open-sourcing the code and datasets, potentially enabling broader improvements in motion understanding across domains including autonomous driving, robotics, and healthcare. According to arXiv.