Ofir PressandNathan Habib
91aa3ed51b
- Update eval.yaml (f969e1b0b82b7791849e2aa8331931061c574b32) Co-authored-by: Nathan Habib <SaylorTwift@users.noreply.huggingface.co>
11 lines
523 B
YAML
11 lines
523 B
YAML
name: SWE-Bench Verified
|
|
description: SWE-bench Verified is a curated, human-annotated subset of 500 issues from the original SWE-bench, designed to accurately evaluate AI models on real-world software engineering tasks. It fixes issues in the original dataset by ensuring all tasks are solvable, well-defined, and properly tested by human developers.
|
|
evaluation_framework: swe-bench
|
|
|
|
tasks:
|
|
- id: swe_bench_%_resolved
|
|
config: default
|
|
split: test
|
|
- id: swe_bench_average_cost
|
|
config: default
|
|
split: test |