SWE-bench_Pro/README.md

dataset_info, configs

dataset_info

configs

features splits download_size dataset_size name dtype repo string name dtype instance_id string name dtype base_commit string name dtype patch string name dtype test_patch string name dtype problem_statement string name dtype requirements string name dtype interface string name dtype repo_language string name dtype fail_to_pass string name dtype pass_to_pass string name dtype issue_specificity string name dtype issue_categories string name num_bytes num_examples dev 4783179 225 name num_bytes num_examples test 44121927 2294 name num_bytes num_examples train 367610377 19008 120086340 416515483

config_name data_files default split path dev data/dev-* split path test data/test-* split path train data/train-*

Dataset Summary

SWE-Bench Pro is a challenging, enterprise-level dataset for testing agent ability on long-horizon software engineering tasks.

See the related evaluation Github: https://github.com/scaleapi/SWE-bench_Pro-os

Dataset Structure

Data Instances

An example datum contains:

{
"repo": "repository_name",
"instance_id": "unique_identifier_65_to_120_chars",
"base_commit": "40_char_git_commit_hash",
"patch": "code_patch_diff_1.44k_to_180k_chars",
"test_patch": "test_cases_325_to_322k_chars",
"problem_statement": "issue_description_419_to_8.04k_chars",
"requirements": "project_dependencies_124_to_6.7k_chars",
"interface": "api_specifications_1_to_12.2k_chars",
"repo_language": "programming_language",
"fail_to_pass": "tests_that_should_pass_after_patch_10_to_155k_chars",
"pass_to_pass": "tests_that_should_continue_passing_2_to_532k_chars",
"issue_specificity": "specificity_level_12_to_77_chars",
"issue_categories": "issue_type_categories_and_tags"
}

Data Fields

repo (string): Repository identifier - one of 11 repository classes instance_id (string): Unique identifier for each instance (65-120 characters) base_commit (string): Git commit hash of the base version (40 characters) patch (string): The actual code patch/diff (1.44k - 180k characters) test_patch (string): Test cases related to the patch (325 - 322k characters) problem_statement (string): Description of the issue being addressed (419 - 8.04k characters) requirements (string): Project requirements or dependencies (124 - 6.7k characters, may be null) interface (string): API or interface specifications (1 - 12.2k characters, may be null) repo_language (string): Programming language of the repository - one of 4 language classes fail_to_pass (string): Test cases that should pass after patch application (10 - 155k characters) pass_to_pass (string): Test cases that should continue passing (2 - 532k characters) issue_specificity (string): Specificity level of the issue (12-77 characters) issue_categories (string): Categories or tags for the issue type