CausalArena: A New Benchmark for Causal Discovery

What changed

Researchers have introduced CausalArena, a new benchmark designed to address inconsistencies and complexities in evaluating causal discovery methods, particularly in the era of foundation models. Existing evaluation practices often differ significantly in their choice of structural causal models (SCMs), including graph families, data generation mechanisms, and evaluation protocols. This lack of standardization makes it difficult to compare results across studies and can lead to misleading conclusions, especially as causal discovery foundation models (CDFMs) emerge. CDFMs' performance might be influenced by the overlap between their pretraining environments and the test SCMs, rather than solely reflecting their causal discovery capabilities.

CausalArena aims to provide a unified and evolvable framework. It comprises three main SCM families:

• Synthetic SCMs: These offer controlled breadth across various graph structures, mechanisms, noise distributions, dimensions, and intervention settings.
• Semantic operational SCMs: These ground causal structures in human-auditable, real-world processes, offering a basis for introducing new environments in future benchmark versions.
• Formula-grounded scientific SCMs: These anchor causal mechanisms in explicit scientific equations, facilitating diagnosis of successes and failures at the mechanism level.

All SCM families compile to a common executable specification and share a standardized observational and interventional evaluation protocol. Additionally, public real-world datasets with published graphs are included for external validity checks. The initial release includes 1,200 executable SCM specifications (1,000 synthetic, 100 semantic operational, and 100 formula-grounded scientific), with a balanced half of each family made public for initial evaluation. The remaining audited SCMs are reserved for held-out leaderboard evaluation and will be released progressively.

Experiments conducted using CausalArena across classical, neural, and pretrained methods reveal substantial ranking shifts. Performance on one benchmark regime does not reliably transfer to others, highlighting the importance of benchmark diversity and the challenge of pretraining-evaluation overlap. The benchmark also allows for analysis of performance based on SCM family, graph properties, mechanism types, sample size, intervention protocols, and semantic or scientific settings.

Why it matters for builders

CausalArena's unified protocol and diverse SCM families offer builders a more reliable way to assess and compare causal discovery algorithms. By moving beyond single, potentially biased benchmarks, builders can gain a clearer understanding of how different methods perform across a wider range of scenarios. This is particularly important for those developing or utilizing CDFMs, as it helps to disentangle true causal discovery ability from pretraining-evaluation overlap. The benchmark's design also supports the development of more robust and generalizable causal inference tools.

Practical impact

Builders can leverage CausalArena to rigorously test their causal discovery implementations. The benchmark's standardized interface and protocol simplify the process of evaluating methods, allowing for direct comparisons against existing classical, neural, and pretrained models. The availability of diverse SCM types—synthetic, semantic, and formula-grounded—enables fine-grained analysis of method strengths and weaknesses. Developers can use these insights to identify areas for improvement, select the most appropriate algorithms for specific applications, and contribute to the advancement of causal discovery research by testing against new SCMs as they are released.

Caveats and source limits

The source material indicates that the initial public release of CausalArena includes a balanced half of each generated SCM family, with the remaining audited SCMs reserved for held-out leaderboard evaluation. The full benchmark suite and its progressive updates will be released over time. The source also notes that performance can depend on factors like sample size and intervention protocols, and that strong performance on synthetic SCMs does not guarantee transferability to semantic, scientific, or real-world datasets. Specific pricing, API details, or direct implementation guidance beyond the benchmark's existence are not provided in the source.