iHore/SWE-bench_Pro
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add eval.yaml (#4)

Browse Source 
- Add eval.yaml (0bffd3f4825f713e220b52e99eb3209fd98b4162)


Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
This commit is contained in:
Jeff Da
2026-02-23 20:54:47 +00:00
committed by system
co-authored by Niels Rogge
parent 2dd05cab15
commit 7ab5114912
1 changed files with 8 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
eval.yaml
+8
View File
@@ -0,0 +1,8 @@
name: SWE-Bench Pro
description: SWE-Bench Pro is an advanced, 1,865-task benchmark for evaluating AI coding agents on realistic, complex, long-horizon software engineering tasks. It improves upon the original SWE-Bench by focusing on non-trivial patches across 41+ repositories, aiming to address data contamination, task and complexity.
evaluation_framework: swe-bench-pro
tasks:
- id: SWE_Bench_Pro
config: default
split: test