iHore/SWE-bench_Verified
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add eval.yaml (#2)

Browse Source 
- Add eval.yaml (8db12b107a1abfd10808f023ed233890b4d7cbae)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
This commit is contained in:
Ofir Press
2026-02-23 17:42:36 +00:00
committed by system
co-authored by Niels Rogge
parent 99450355ca
commit 880e32832a
1 changed files with 11 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
eval.yaml
+11
View File
@@ -0,0 +1,11 @@
name: SWE-Bench Verified
description: SWE-bench Verified is a curated, human-annotated subset of 500 issues from the original SWE-bench, designed to accurately evaluate AI models on real-world software engineering tasks. It fixes issues in the original dataset by ensuring all tasks are solvable, well-defined, and properly tested by human developers.
evaluation_framework: swe-bench-verified
tasks:
- id: swe_bench_%_resolved
config: default
split: test
- id: swe_bench_average_cost
config: default
split: test