Update README.md
This commit is contained in:
Jeff Da
@@ -13,17 +13,19 @@ dataset_info:
|
||||
dtype: string
|
||||
- name: problem_statement
|
||||
dtype: string
|
||||
- name: hints_text
|
||||
- name: requirements
|
||||
dtype: string
|
||||
- name: created_at
|
||||
- name: interface
|
||||
dtype: string
|
||||
- name: version
|
||||
- name: repo_language
|
||||
dtype: string
|
||||
- name: FAIL_TO_PASS
|
||||
- name: fail_to_pass
|
||||
dtype: string
|
||||
- name: PASS_TO_PASS
|
||||
- name: pass_to_pass
|
||||
dtype: string
|
||||
- name: environment_setup_commit
|
||||
- name: issue_specificity
|
||||
dtype: string
|
||||
- name: issue_categories
|
||||
dtype: string
|
||||
splits:
|
||||
- name: dev
|
||||
@@ -47,123 +49,48 @@ configs:
|
||||
- split: train
|
||||
path: data/train-*
|
||||
---
|
||||
### Dataset Summary
|
||||
SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution.
|
||||
The dataset was released as part of [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770)
|
||||
## Want to run inference now?
|
||||
This dataset only contains the problem_statement (i.e. issue text) and the base_commit which can represents the state of the codebase before the issue has been resolved. If you want to run inference using the "Oracle" or BM25 retrieval settings mentioned in the paper, consider the following datasets.
|
||||
[princeton-nlp/SWE-bench_oracle](https://huggingface.co/datasets/princeton-nlp/SWE-bench_oracle)
|
||||
[princeton-nlp/SWE-bench_bm25_13K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_13K)
|
||||
[princeton-nlp/SWE-bench_bm25_27K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_27K)
|
||||
[princeton-nlp/SWE-bench_bm25_40K](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_40K)
|
||||
[princeton-nlp/SWE-bench_bm25_50k_llama](https://huggingface.co/datasets/princeton-nlp/SWE-bench_bm25_50k_llama)
|
||||
### Supported Tasks and Leaderboards
|
||||
SWE-bench proposes a new task: issue resolution provided a full repository and GitHub issue. The leaderboard can be found at www.swebench.com
|
||||
### Languages
|
||||
The text of the dataset is primarily English, but we make no effort to filter or otherwise clean based on language type.
|
||||
## Dataset Structure
|
||||
### Data Instances
|
||||
An example of a SWE-bench datum is as follows:
|
||||
|
||||
instance_id: (str) - A formatted instance identifier, usually as repo_owner__repo_name-PR-number.
|
||||
patch: (str) - The gold patch, the patch generated by the PR (minus test-related code), that resolved the issue.
|
||||
repo: (str) - The repository owner/name identifier from GitHub.
|
||||
base_commit: (str) - The commit hash of the repository representing the HEAD of the repository before the solution PR is applied.
|
||||
hints_text: (str) - Comments made on the issue prior to the creation of the solution PR’s first commit creation date.
|
||||
created_at: (str) - The creation date of the pull request.
|
||||
test_patch: (str) - A test-file patch that was contributed by the solution PR.
|
||||
problem_statement: (str) - The issue title and body.
|
||||
version: (str) - Installation version to use for running evaluation.
|
||||
environment_setup_commit: (str) - commit hash to use for environment setup and installation.
|
||||
FAIL_TO_PASS: (str) - A json list of strings that represent the set of tests resolved by the PR and tied to the issue resolution.
|
||||
PASS_TO_PASS: (str) - A json list of strings that represent tests that should pass before and after the PR application.
|
||||
|
||||
[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)Follow this markdown format, put it so i can copy pasteEditmarkdown---
|
||||
dataset_info:
|
||||
features:
|
||||
- name: repo
|
||||
dtype: string
|
||||
- name: instance_id
|
||||
dtype: string
|
||||
- name: base_commit
|
||||
dtype: string
|
||||
- name: patch
|
||||
dtype: string
|
||||
- name: test_patch
|
||||
dtype: string
|
||||
- name: problem_statement
|
||||
dtype: string
|
||||
- name: hints_text
|
||||
dtype: string
|
||||
- name: created_at
|
||||
dtype: string
|
||||
- name: version
|
||||
dtype: string
|
||||
- name: FAIL_TO_PASS
|
||||
dtype: string
|
||||
- name: PASS_TO_PASS
|
||||
dtype: string
|
||||
- name: environment_setup_commit
|
||||
dtype: string
|
||||
splits:
|
||||
- name: dev
|
||||
num_bytes: 4783179
|
||||
num_examples: 225
|
||||
- name: test
|
||||
num_bytes: 44121927
|
||||
num_examples: 2294
|
||||
- name: train
|
||||
num_bytes: 367610377
|
||||
num_examples: 19008
|
||||
download_size: 120086340
|
||||
dataset_size: 416515483
|
||||
configs:
|
||||
- config_name: default
|
||||
data_files:
|
||||
- split: test
|
||||
path: data/test-*
|
||||
---
|
||||
|
||||
# Dataset Card for SWE-bench
|
||||
# Dataset Card for Software Repository Patch Dataset
|
||||
|
||||
## Dataset Summary
|
||||
|
||||
SWE-bench Pro is a dataset that tests systems' ability to solve GitHub issues automatically. This dataset is the public set only.
|
||||
This dataset contains software repository patches and associated metadata for automated program repair and code analysis research. The dataset includes patch information, test cases, problem statements, requirements, interface specifications, and various metadata fields across multiple programming languages and repositories.
|
||||
|
||||
The dataset was released as part of SWE-Bench Pro paper.
|
||||
|
||||
## Dataset Details
|
||||
|
||||
### Dataset Description
|
||||
|
||||
- **Curated by:** Scale AI
|
||||
- **Language(s) (NLP):** English (primarily)
|
||||
|
||||
## Uses
|
||||
|
||||
### Direct Use
|
||||
|
||||
This dataset is designed for evaluating systems' ability to automatically resolve GitHub issues by generating appropriate code patches.
|
||||
|
||||
### Out-of-Scope Use
|
||||
|
||||
This dataset is not meant as training data.
|
||||
## Dataset Structure
|
||||
|
||||
### Data Instances
|
||||
|
||||
An example of a SWE-bench datum is as follows:
|
||||
An example datum contains:
|
||||
```json
|
||||
{
|
||||
"instance_id": "repo_owner__repo_name-PR-number",
|
||||
"patch": "The gold patch that resolved the issue",
|
||||
"repo": "The repository owner/name identifier from GitHub",
|
||||
"base_commit": "The commit hash before the solution PR is applied",
|
||||
"hints_text": "Comments made on the issue prior to the solution PR",
|
||||
"created_at": "The creation date of the pull request",
|
||||
"test_patch": "A test-file patch contributed by the solution PR",
|
||||
"problem_statement": "The issue title and body",
|
||||
"version": "Installation version for running evaluation",
|
||||
"environment_setup_commit": "Commit hash for environment setup and installation",
|
||||
"FAIL_TO_PASS": "JSON list of tests resolved by the PR",
|
||||
"PASS_TO_PASS": "JSON list of tests that should pass before and after PR"
|
||||
}
|
||||
"repo": "repository_name",
|
||||
"instance_id": "unique_identifier_65_to_120_chars",
|
||||
"base_commit": "40_char_git_commit_hash",
|
||||
"patch": "code_patch_diff_1.44k_to_180k_chars",
|
||||
"test_patch": "test_cases_325_to_322k_chars",
|
||||
"problem_statement": "issue_description_419_to_8.04k_chars",
|
||||
"requirements": "project_dependencies_124_to_6.7k_chars",
|
||||
"interface": "api_specifications_1_to_12.2k_chars",
|
||||
"repo_language": "programming_language",
|
||||
"fail_to_pass": "tests_that_should_pass_after_patch_10_to_155k_chars",
|
||||
"pass_to_pass": "tests_that_should_continue_passing_2_to_532k_chars",
|
||||
"issue_specificity": "specificity_level_12_to_77_chars",
|
||||
"issue_categories": "issue_type_categories_and_tags"
|
||||
}
|
||||
```
|
||||
|
||||
## Data Fields
|
||||
|
||||
repo (string): Repository identifier - one of 11 repository classes
|
||||
instance_id (string): Unique identifier for each instance (65-120 characters)
|
||||
base_commit (string): Git commit hash of the base version (40 characters)
|
||||
patch (string): The actual code patch/diff (1.44k - 180k characters)
|
||||
test_patch (string): Test cases related to the patch (325 - 322k characters)
|
||||
problem_statement (string): Description of the issue being addressed (419 - 8.04k characters)
|
||||
requirements (string): Project requirements or dependencies (124 - 6.7k characters, may be null)
|
||||
interface (string): API or interface specifications (1 - 12.2k characters, may be null)
|
||||
repo_language (string): Programming language of the repository - one of 4 language classes
|
||||
fail_to_pass (string): Test cases that should pass after patch application (10 - 155k characters)
|
||||
pass_to_pass (string): Test cases that should continue passing (2 - 532k characters)
|
||||
issue_specificity (string): Specificity level of the issue (12-77 characters)
|
||||
issue_categories (string): Categories or tags for the issue type
|
||||
Reference in New Issue
Block a user