NewtPhys: A New Benchmark for Newtonian Physics Understanding in Foundation Models

A new research paper introduces NewtPhys, a novel 4D physically annotated dataset aimed at assessing how well foundation models comprehend Newtonian physics. Unlike previous benchmarks that often use synthetic or semi-synthetic scenes and focus on high-level events, NewtPhys is constructed from multiview images of real-world scenes, augmented with physics-grounded simulations. This approach provides dense, fine-grained annotations across timesteps, encompassing 3D forces and amodal per-pixel quantities related to physics, tracking, semantics, and geometry.

The creators of NewtPhys utilized this dataset to conduct a systematic evaluation of 56 Vision-Language Models (VLMs), including 54 open-weight and 2 closed-source models, alongside 10 Vision Foundation Models (VFMs). The findings indicate that these models exhibit limitations in their low-level physics reasoning abilities. The researchers suggest that NewtPhys can serve as a valuable resource for future research in physics-grounded vision and for developing advanced physics-aware evaluation methods. Code and datasets are publicly available.