Diagram-MMU: A New Benchmark for Evaluating Multimodal LLMs on Scientific Diagrams

What changed

Researchers have developed Diagram-MMU, a novel benchmark specifically designed to assess the performance of Multimodal Large Language Models (MLLMs) in understanding and parsing scientific diagrams. The benchmark comprises 3.7k curated diagrams and over 18.3k human-validated questions spanning six distinct domains. It evaluates MLLMs across three key tasks: converting diagrams to code (e.g., LaTeX TikZ), editing diagrams based on code, and answering questions about diagrams. The benchmark also includes agentic settings for each task.

Why it matters for builders

This benchmark is crucial for AI builders working on MLLMs intended for scientific domains. It provides a concrete evaluation framework for a capability that is becoming increasingly important for tools that assist in scientific writing and collaboration, such as OpenAI Prism. The focus on diagram-to-code generation highlights a specific area where current MLLMs often struggle, offering a clear target for improvement.

Practical impact

Evaluations conducted using Diagram-MMU on 12 MLLMs indicate that while models can generally reason well over diagrams, they face significant challenges in accurately parsing and editing them into code. Diagram-to-code tasks proved more difficult than diagram question answering. Interestingly, under agentic settings, most models showed improved parsing and editing performance but a decline in question answering accuracy. Claude-4.6 Opus was noted as an exception, demonstrating consistent improvement across all three tasks.

Caveats and source limits

The provided source is a research paper introducing the benchmark. Specific details on the exact performance metrics for each of the 12 evaluated MLLMs, beyond the general observation of challenges in diagram-to-code tasks and the specific mention of Claude-4.6 Opus, are not detailed in the excerpt. The benchmark itself is presented as a new creation, and its long-term impact and adoption remain to be seen. The source does not provide information on the specific technical implementation details of the benchmark or the models evaluated, nor does it offer pricing or release dates for any associated tools.