Diagram-MMU: New Benchmark for Scientific Diagram Understanding

What changed

Researchers have introduced Diagram-MMU, a novel multi-modal benchmark specifically designed to assess the capabilities of Multimodal Large Language Models (MLLMs) in understanding and processing scientific diagrams. This benchmark aims to evaluate MLLMs on three core tasks: diagram-to-code parsing (D2C-P), diagram-to-code editing (D2C-E), and diagram question answering (DQA). It also incorporates agentic settings to evaluate how models can act within a scientific writing context. Diagram-MMU features a dataset of 3,744 curated diagrams and 18,305 human-validated questions spanning six scientific domains, including charts, planar geometry, 3D shapes, graphs, chemistry, and circuit diagrams. Notably, it is the first benchmark to support diagram-to-code tasks for chemistry and circuit diagrams. The evaluation uses multiple metrics, including object-level F1 scores for fine-grained element grounding, CrystalBLEU for TikZ syntax correctness, image-level similarity, and answer accuracy for DQA. The benchmark also includes 16 controllable evaluation settings, focusing on foundational abilities and agentic capabilities such as context utilization, tool use, state management, and planning. A key innovation is the inclusion of a TikZ search tool, implemented as an MCP server, to help MLLMs selectively access relevant references.

Why it matters for builders

Diagram-MMU provides a standardized way for AI builders to measure and enhance the performance of their MLLMs in a crucial area of scientific research and communication. The ability to convert scientific diagrams into editable code, such as LaTeX TikZ, is a key feature for tools aimed at scientific writing and collaboration, like OpenAI Prism. By evaluating models on parsing, editing, and question answering related to diagrams, this benchmark helps identify specific weaknesses and strengths, guiding the development of more capable AI assistants for researchers.

Practical impact

Builders can leverage Diagram-MMU to benchmark their MLLMs against a comprehensive set of tasks and domains. The evaluation results indicate that while models show strong reasoning capabilities for diagram question answering (up to 86% accuracy), they struggle significantly with diagram-to-code parsing (31-57% object-level F1). This suggests a need for improved perception and coding abilities. The benchmark also highlights that agentic capabilities can improve editing performance but may degrade question answering if not implemented carefully. Claude-4.6 Opus demonstrated consistent improvement across all tasks under agentic settings, suggesting its robust tool use. Developers can use these insights to focus on enhancing code generation from visual inputs and refining agentic strategies for scientific diagram manipulation.

Caveats and source limits

The source paper provides a detailed description of the Diagram-MMU benchmark and initial evaluation results on 12 MLLMs. However, specific pricing for accessing or using the benchmark, or details on its integration into specific development platforms, are not provided. The evaluation results are based on the authors' assessment, and independent benchmark results from third-party researchers are not yet available. The benchmark is presented as a research contribution, and its availability as a production-ready tool or API is not specified.