diff --git a/eval.yaml b/eval.yaml new file mode 100644 index 0000000..30b2a0e --- /dev/null +++ b/eval.yaml @@ -0,0 +1,8 @@ +name: SWE-Bench Pro +description: SWE-Bench Pro is an advanced, 1,865-task benchmark for evaluating AI coding agents on realistic, complex, long-horizon software engineering tasks. It improves upon the original SWE-Bench by focusing on non-trivial patches across 41+ repositories, aiming to address data contamination, task and complexity. +evaluation_framework: swe-bench-pro + +tasks: + - id: SWE_Bench_Pro + config: default + split: test \ No newline at end of file