CorporateBench: New Q&A Benchmark for Enterprise LLM Evaluation

What changed

Researchers have introduced CorporateBench (CB), a novel Q&A benchmark specifically designed to evaluate Large Language Models (LLMs) on their ability to process and reason over enterprise-scale document collections. Existing benchmarks often fall short due to companies' reluctance to share internal data and the oversimplification of synthetic datasets. CB aims to bridge this gap by offering a human-validated, multi-task Q&A benchmark that simulates the conditions found in corporate communication networks. The evaluation corpora within CB surpass 230,000 documents, providing a scale that approaches real-world enterprise scenarios. CB assesses LLMs across two key dimensions: information extraction and knowledge base querying. It achieves this through four synthetically generated firms, varying in size from 12 to 10,000 employees. Each corpus is derived from a temporally evolving knowledge base, ensuring logical consistency across hundreds of thousands of documents, even as the information evolves over time. Initial evaluations on five LLMs using CB have revealed a decline in performance as the input size approaches realistic scales, highlighting a common failure mode for current LLMs in enterprise contexts.

Why it matters for builders

CorporateBench provides LLM developers with a much-needed metric for assessing corporate communication reasoning capabilities. This benchmark is crucial for understanding how well LLMs can handle the complex, interconnected information typical of enterprise environments, which often involves synthesizing data from numerous documents. By offering a more realistic evaluation framework, CB helps developers identify and address limitations in LLM performance related to long-context understanding and multi-document reasoning, ultimately leading to more robust and capable models for business applications.

Practical impact

LLM developers can utilize CorporateBench to benchmark their models against a realistic enterprise data simulation. The benchmark's design, which emphasizes cross-document logical consistency and temporal evolution, allows for rigorous testing of information extraction and knowledge base querying skills. Builders can leverage the insights gained from CB evaluations to fine-tune models for improved performance in enterprise settings, particularly in scenarios requiring deep understanding of large, dynamic knowledge bases. The availability of the data and code on Hugging Face (https://huggingface.co/datasets/epiq-ai-labs/corporatebench) facilitates direct integration into development and testing workflows.

Caveats and source limits

The CorporateBench benchmark is based on synthetically generated firms and documents, although efforts have been made to ensure realism and logical consistency. While the benchmark aims to mimic enterprise conditions, it does not use real-world proprietary data due to non-disclosure agreements. The initial evaluation involved five LLMs, and further testing across a broader range of models would provide more comprehensive insights into their performance characteristics on this benchmark. The source does not provide specific performance numbers for the evaluated LLMs, only a general observation of decreasing performance with increasing input size.