IdeaAMBIG: A Benchmark for Implementation Gaps in AI Research

What changed

A new benchmark, IdeaAMBIG, has been developed to assess the 'codification readiness' of research-method specifications in AI. This readiness is defined by whether a specification provides enough detail for a competent implementer or coding agent to construct the intended method without making unsupported assumptions. The benchmark comprises 660 instances, including 163 real-world gaps sourced from reproducibility reports and GitHub issues, and 497 synthetic gaps.

IdeaAMBIG is designed to evaluate three core capabilities: codification-readiness assessment, defect localization (identifying the ambiguity in a specification), and clarification action generation (proposing how to resolve the ambiguity). The benchmark's design allows for evaluating these capabilities both with and without the annotated defect.

Why it matters for builders

For AI builders, this benchmark highlights a significant hurdle in the research-to-implementation pipeline: the gap between a research idea and its actionable specification. By providing a standardized way to measure how well AI models can understand and resolve these ambiguities, IdeaAMBIG can drive the development of more robust AI coding agents and tools, ultimately accelerating the adoption of new research findings.

Practical impact

Testing across 13 large language models (LLMs) revealed varying performance. The best-performing model achieved a Macro Defect Recovery Rate of 9.6% on real-world instances for defect localization. However, when provided with the annotated defect, the same model showed an 80.6% Macro Clarification Action Success Rate. An oracle study demonstrated that supplying the correct resolution significantly improved the codification-ready rate from 14% to 98%. The research indicates that defect localization is currently the primary bottleneck, while clarification generation is more effective when the defect is known.

Caveats and source limits

The provided source is a preprint of a research paper introducing the IdeaAMBIG benchmark. Specific details regarding the exact methodologies for generating synthetic gaps, the precise definitions of 'competent implementer' or 'coding agent,' and the full list of 13 LLMs evaluated are not fully detailed in the excerpt. Performance metrics are presented as Macro Rates, and further context on their interpretation might be beneficial. The benchmark focuses on specification readiness and does not directly evaluate the implementation of the method itself.