GameHorizon Suite: Evaluating AI Gameplay Across Multiple Horizons

What changed

The GameHorizon Suite is presented as a novel, unified data and evaluation framework aimed at assessing AI models' gameplay capabilities across multiple temporal horizons. It addresses shortcomings of existing benchmarks, such as narrow game coverage, lack of language instructions, and reliance on high-variance online rollouts. The suite comprises three core components: GameHorizon-Annotator, GameHorizon-Data, and GameHorizon-Bench.

GameHorizon-Annotator is an automated pipeline designed for scalable annotation of multi-horizon instructions within gameplay. This pipeline generates a hierarchical set of natural-language instructions, ranging from short-horizon operations to medium-horizon goals and long-horizon strategies, by abstracting fine-grained actions.

Utilizing the annotator, GameHorizon-Data has been constructed. This dataset is described as the first large-scale AAA gameplay dataset featuring temporally aligned videos, player actions, and multi-horizon instructions. It contains 5,000 hours of recordings from 21 different games, gathered by 100 expert human players, surpassing previous datasets in scale and scope.

GameHorizon-Bench provides a framework for reproducible evaluation through both offline and stepwise online testing. The offline track offers standardized questions across three primary tasks and diagnostic variants, enabling consistent assessment. The online track complements this by verifying if offline performance translates to actual gameplay and by pinpointing failures to specific steps in long-horizon gameplay.

Based on this suite, the researchers evaluated 47 models, performing over one million model invocations. This evaluation revealed a distinct hierarchy of task difficulty and significant variations in model capabilities.

Why it matters for builders

GameHorizon offers AI builders a standardized and comprehensive platform to evaluate and compare the performance of their models in complex gaming environments. By providing a unified benchmark that spans diverse games and temporal scales, it allows for more accurate assessments of capabilities like visual understanding, instruction decomposition, goal planning, and action control. This can guide development efforts towards more robust and versatile game-playing AI.

Practical impact

Builders can leverage the GameHorizon Suite to test their models against a broad spectrum of gameplay challenges. The availability of the dataset, annotator, and benchmark is intended to facilitate future research and development in game AI. The suite's design, supporting both dedicated game agents and general-purpose models, allows for cross-family comparisons. The researchers plan to release the dataset, annotator, and benchmark, enabling developers to integrate these resources into their evaluation pipelines and potentially identify areas for improvement in their AI agents.

Caveats and source limits

The primary source is a research paper detailing the GameHorizon Suite. While it describes the components and evaluation performed, specific details regarding the release timeline or accessibility of the dataset, annotator, and benchmark are not provided beyond the intention to release them. Independent benchmarks or comparisons with other existing game AI evaluation suites are not detailed in the provided excerpt. The excerpt focuses on the introduction and components of the suite, with further details on specific task formulations or diagnostic variants within GameHorizon-Bench being trimmed.