From 7ab5114912baf22bb098818e604c02fe7ad2c11f Mon Sep 17 00:00:00 2001 From: Jeff Da Date: Mon, 23 Feb 2026 20:54:47 +0000 Subject: [PATCH] Add eval.yaml (#4) - Add eval.yaml (0bffd3f4825f713e220b52e99eb3209fd98b4162) Co-authored-by: Niels Rogge --- eval.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 eval.yaml diff --git a/eval.yaml b/eval.yaml new file mode 100644 index 0000000..30b2a0e --- /dev/null +++ b/eval.yaml @@ -0,0 +1,8 @@ +name: SWE-Bench Pro +description: SWE-Bench Pro is an advanced, 1,865-task benchmark for evaluating AI coding agents on realistic, complex, long-horizon software engineering tasks. It improves upon the original SWE-Bench by focusing on non-trivial patches across 41+ repositories, aiming to address data contamination, task and complexity. +evaluation_framework: swe-bench-pro + +tasks: + - id: SWE_Bench_Pro + config: default + split: test \ No newline at end of file