RATIO Benchmark for Scientific Literature Retrieval

What changed

Maayan Sharon and Tom Hope have introduced RATIO (Retrieval Across Typed Ideation Operations), a novel benchmark designed to evaluate retrieval systems for scientific literature, specifically focusing on its role in inspiring scientific ideation. Unlike traditional benchmarks that primarily assess topical relevance or direct answers to queries, RATIO categorizes relevance based on three distinct "ideation moves":

• Address: Retrieves potential approaches or insights that directly respond to a stated problem.
• Broaden: Surfaces formulations or concepts at a more general or abstract level.
• Specify: Identifies concrete instantiations or narrower formulations of a concept.

RATIO is constructed from millions of full-text scientific papers across computer science literature. The construction methodology extends discourse-marker distant supervision, previously used for classification, to corpus-scale retrieval. This involves identifying explicit discourse markers (e.g., "to address this problem," "more generally," "as an example") within papers to mine sentence pairs that represent these ideation moves. The process includes multiple stages of supervision and validation, combining manual analysis, rule-based expansion, LLM generation, and extensive human and LLM vetting to create high-precision evaluation sets. Experiments conducted with retrievers demonstrate that while operation-specific fine-tuning significantly improves performance for each of the three operations, there remains substantial room for further advancement.

Why it matters for builders

For AI builders, RATIO provides a specialized framework for developing and refining retrieval systems that go beyond simple keyword matching. By enabling systems to understand and retrieve literature based on its potential to inspire specific types of ideation—addressing problems, broadening perspectives, or specifying details—developers can create more sophisticated AI assistants for scientific research. This can lead to tools that not only find relevant papers but also actively contribute to the creative and problem-solving process in scientific discovery.

Practical impact

Builders can leverage RATIO to train and evaluate retrieval models that are specifically tuned for scientific inspiration. The benchmark's focus on typed ideation operations suggests that fine-tuning retrievers on these specific tasks can yield substantial improvements. This means developers can explore building systems that can identify not just related work, but work that offers a specific type of inspiration, such as a novel solution to a problem, a more general theoretical framework, or a concrete example applicable to a new domain. The availability of a large-scale benchmark with a clear construction methodology also opens avenues for research into new retrieval techniques tailored for scientific discourse.

Caveats and source limits

The source indicates that while operation-specific fine-tuning substantially boosts retriever performance, absolute performance remains limited, suggesting that current models still have considerable room for improvement in capturing these nuanced ideation moves. The benchmark is constructed from computer science literature, and its direct applicability to other scientific domains may require adaptation. The source does not provide specific details on the LLMs used for vetting or generation, nor does it offer independent benchmark results beyond the authors' experiments. The publication date is in the future (2026), indicating this is a research proposal or early work. The source does not mention availability of code or models for the benchmark itself.