SWEGraph v3 Toy Sample: 800 Coding Agent Tasks Released

What changed

A free, toy sample of the SWEGraph v3 dataset has been made available on Hugging Face by williamTLmiller. This sample contains 800 agent tasks, representing a fraction of the full SWEGraph v3 dataset, which aims to normalize SWE-bench, SWE-smith, and BigCodeBench into structured proof chains. Each task is linked to its repository, issue, fix, diff hunks, tests, and evaluation task, along with licensing information and container images for execution. The sample focuses on metadata, providing task definitions, licenses, difficulty tiers, and execution evidence, but the actual code text for tasks with non-permissive licenses is withheld. The dataset is structured in plain CSV format and is intended for schema exploration and small-scale training runs, not for full model training.

Key specifications of the toy sample:

• Total Agent Tasks: 800
• Sources: 420 from SWE-bench train, 380 from SWE-smith.
• Difficulty Distribution: 275 easy, 327 medium, 119 hard, 79 very hard.
• Code Text Status: 797 tasks include permissive code, 3 tasks withhold code due to license issues.
• Execution Evidence: 380 tasks (all SWE-smith) have upstream reported execution evidence; 420 SWE-bench tasks have only a patch.
• Languages: Primarily Python (743 tasks), with some Go (57 tasks).
• Tests: Includes 2,367 FAIL_TO_PASS and 164 PASS_TO_PASS test rows.
• Issue Dates: SWE-bench train issues range from May 2, 2014, to August 21, 2023.

The sample includes all 800 tasks from the full package's training split, deterministically drawn using a salted SHA-256 hash of the instance ID. Tasks exceeding a spreadsheet cell limit for test lists are excluded from this toy sample. Identity and contamination groups are preserved, with member lists trimmed to those present in the sample.

Why it matters for builders

This release offers developers a tangible resource for building and evaluating AI agents focused on software engineering tasks. The structured nature of the data, following a clear proof chain from issue to fix, allows for precise measurement of agent capabilities in bug fixing and code generation. The decontaminated aspect is crucial for reliable benchmarking, preventing models from overfitting to specific training examples. Builders can use this dataset to test their agent architectures, fine-tune models for specific coding tasks, and develop more robust evaluation methodologies.

Practical impact

Developers can leverage this toy sample to experiment with loading and processing structured software engineering tasks. The dataset's CSV format and clear schema make it accessible for initial integration into agent pipelines. Builders can use the provided metadata to simulate agent execution, analyze task complexity, and develop strategies for handling different types of software bugs. The inclusion of execution evidence for a subset of tasks allows for direct comparison of agent-generated solutions against known outcomes. For those interested in the full dataset, contact information for licensing the complete SWEGraph v3 is provided.

Caveats and source limits

The primary limitation of this release is its nature as a "toy sample." It contains only 800 tasks and omits the actual code text for a portion of these tasks due to licensing restrictions. The full dataset, which includes all code text and a larger number of tasks, is private and licensed separately. Furthermore, the "eval_tasks.csv" and "heldout_eval.csv" files are header-only, as these belong to the paid held-out benchmark and are not included in this free sample. Therefore, this toy sample is not suitable for comprehensive model training or rigorous benchmarking of production-ready agents.