New Benchmark for Evaluating LLMs in EHR Information Retrieval

What changed

Researchers have introduced the Benchmark for Retrieving Information in EHRs (BRIE), a novel evaluation dataset designed for Large Language Models (LLMs) used in clinical settings. Traditional benchmarks for evaluating LLM-based clinical assistants, which are increasingly integrated into Electronic Health Record (EHR) systems, are often manually curated, expensive to update, and quickly become obsolete due to rapid technological advancements. BRIE addresses these limitations by employing a scalable framework that automatically generates question-answer pairs directly from longitudinal EHR notes. This generator was validated by nineteen clinicians, leading to the creation of BRIE, which is designed to be continuously maintainable. The benchmark aims to facilitate rigorous and up-to-date evaluations of clinical LLMs, particularly for tasks requiring information synthesis across multiple documents and patient encounters. Initial evaluations using BRIE across nine LLMs and five inference strategies revealed that state-of-the-art systems frequently omit clinically important information, especially when complex synthesis is required.

BRIE supports evaluations that static benchmarks cannot, such as generating multiple answers to reflect variations in clinician reasoning, which allows for a more robust assessment of performance. Furthermore, its design allows for continuous refreshing of benchmark content to prevent data leakage and ensure ongoing relevance. The framework enables the creation of datasets that can be regenerated from new patient data, demonstrating its capacity to retain difficulty without manual clinician intervention over time.

Why it matters for builders

For AI builders developing clinical LLM applications, BRIE offers a more dynamic and reliable evaluation tool. The ability to automatically generate and continuously update benchmark data means that performance assessments will remain relevant as LLM technology and clinical documentation practices evolve. This is crucial for ensuring the safety and efficacy of LLMs deployed in healthcare, where errors can have significant consequences. Builders can leverage BRIE to identify specific failure modes, such as the omission of critical information, and to refine their models for better performance in complex, multi-document synthesis tasks.

Practical impact

Developers can utilize BRIE to test and validate their LLM-based clinical assistants. The benchmark's automated generation process and continuous refresh capability mean that it can serve as an ongoing monitoring tool for deployed systems. By identifying instances where models omit clinically important information, particularly across multiple patient encounters, builders can focus on improving the models' synthesis and retrieval capabilities. The framework's support for multiple reference answers also allows for a more nuanced understanding of model performance, accounting for the inherent variability in clinical reasoning. This enables a more accurate assessment of model readiness for real-world clinical deployment.

Caveats and source limits

The primary source is a research paper detailing the development and initial application of the BRIE benchmark. While the generator was validated by nineteen clinicians, the full extent of its performance across a wider range of clinical specialties and EHR system types is not detailed. The paper focuses on the benchmark's creation and its utility in identifying common LLM failure modes, particularly information omission. Specific details on the computational resources required for benchmark generation or regeneration are not provided. The benchmark's effectiveness in evaluating LLMs for tasks beyond information retrieval, such as clinical note generation or decision support, is also outside the scope of this paper. The source does not provide pricing information for any LLM services or tools used in the evaluation.