MindTopo Benchmark Evaluates Topological Spatial Reasoning in Foundation Models

What changed

Researchers have introduced MindTopo, a novel benchmark designed to evaluate the topological spatial reasoning capabilities of foundation models. Unlike existing benchmarks that primarily focus on metric or viewpoint-dependent spatial properties, MindTopo assesses models on five key topological properties: continuity, separation, order, enclosure, and knots. These properties are grounded in cognitive science and formal topology, reflecting how humans understand spatial relationships that remain invariant even when objects are continuously deformed. The benchmark is structured around two cognitive levels: 'reasoning' and 'planning'. Reasoning tasks require models to identify topological relations or predict how they change when a scene is altered. Planning tasks, on the other hand, instantiate foundation models as closed-loop agents that must select actions within an interactive environment to build, preserve, or modify topological structures. MindTopo comprises 11,030 instances across 13 procedurally generated task types, offering controllable difficulty. The evaluation included 14 multimodal large language models (MLLMs), with results indicating that all models performed better on reasoning tasks than on planning tasks. Notably, even the top-performing model fell significantly short of observed human performance. For the Qwen3-VL-2B-Instruct model, both supervised fine-tuning and reinforcement learning improved reasoning capabilities more than planning. Furthermore, agent configurations augmented with image and video generation, including three video generative models, were studied in planning settings. While generated observations could retain local cues and reach plausible endpoints, audited rollouts did not consistently adhere to environment dynamics or preserve topology across transitions, highlighting a gap between understanding topological relations and acting upon them effectively.

Why it matters for builders

MindTopo offers AI builders a crucial new tool to assess and enhance a fundamental aspect of spatial intelligence that has been largely neglected in current evaluations. By providing a benchmark that specifically targets topological intuition, developers can now identify weaknesses in their models related to understanding spatial invariants. This is essential for building more capable AI agents that can interact with the physical world, from robotics to augmented reality, where understanding how objects can be deformed without losing their essential topological properties is key.

Practical impact

Builders can leverage the MindTopo benchmark to test their MLLMs and agent configurations. The benchmark's structure, with its distinct reasoning and planning levels, allows for targeted improvements. For instance, developers can focus on fine-tuning models for planning tasks if their agents struggle with action selection based on topological constraints. The inclusion of procedurally generated tasks with controllable difficulty means that models can be trained and evaluated across a wide range of complexity. The findings suggest that current training methods, including supervised fine-tuning and reinforcement learning, yield greater improvements in reasoning than planning, indicating an area for further research and development in agent training strategies. The benchmark also provides insights into the limitations of current generative models in maintaining topological consistency during action sequences, guiding efforts to improve their world modeling and predictive capabilities.

Caveats and source limits

The primary source is a research paper detailing the MindTopo benchmark and initial evaluations. Specific benchmark results for all 14 MLLMs are presented, but the exact performance metrics and comparisons are detailed within the paper itself. The paper notes that the best-performing model is still far below human performance, indicating significant room for improvement across the board. The source does not provide pricing information for any models or tools, nor does it detail specific release dates for the benchmark itself, other than its publication date. The benchmark is presented as a research contribution, and its availability for broader use is implied through the mention of a website and code repository, though direct links to download or access the dataset are not explicitly detailed in the excerpt. The performance of video generative models in planning settings is also discussed, with limitations noted in their ability to reliably preserve topology.