dataset_info, configs

dataset_info

configs

features splits download_size dataset_size name dtype repo string name dtype instance_id string name dtype base_commit string name dtype patch string name dtype test_patch string name dtype problem_statement string name dtype requirements string name dtype interface string name dtype repo_language string name dtype fail_to_pass string name dtype pass_to_pass string name dtype issue_specificity string name dtype issue_categories string name dtype before_repo_set_cmd string name dtype selected_test_files_to_run string name num_bytes num_examples test 23284899 731 120086340 416515483

config_name data_files default split path test external_hf_v2.csv

Dataset Summary

SWE-Bench Pro is a challenging, enterprise-level dataset for testing agent ability on long-horizon software engineering tasks.

See the related evaluation Github: https://github.com/scaleapi/SWE-bench_Pro-os

Dataset Structure

We follow SWE-Bench Verified (https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified) in terms of dataset structure, with several extra fields.

Data Fields

repo (string): Repository identifier - one of 11 repository classes

instance_id (string): Unique identifier for each instance (65-120 characters)

base_commit (string): Git commit hash of the base version (40 characters)

patch (string): The golden code patch/diff (1.44k - 180k characters)

test_patch (string): Test cases related to the patch (325 - 322k characters)

problem_statement (string): Description of the issue being addressed (419 - 8.04k characters)

requirements (string): Project requirements or dependencies (124 - 6.7k characters, may be null)

interface (string): API or interface specifications (1 - 12.2k characters, may be null)

repo_language (string): Programming language of the repository - one of 4 language classes

fail_to_pass (string): Test cases that should pass after patch application (10 - 155k characters)

pass_to_pass (string): Test cases that should continue passing (2 - 532k characters)

issue_specificity (string): Specificity of the issue (12-77 characters)

issue_categories (string): Categories or tags for the issue type