14 well-known public benchmarks, converted into 22 individually-valid
EvalPort suites — 8,012 test
cases total, every one of them passing openeval.validate.validate_suite().
This is the reference set of "real" suites for anyone building or testing
an EvalPort-compatible runner, adapter, or tool: pull any file below and
you have a spec-conformant suite backed by a real, cited, license-cleared
benchmark, not a synthetic example.
# Validate a single suite against the real EvalPort validator
pip install evalport-sdk
python3 -c "
import json
from openeval.validate import validate_suite
suite = json.load(open('benchmarks/gsm8k/gsm8k.json'))
print(validate_suite(suite).valid)
"
# Or validate every suite in the hub at once (the same check CI runs)
python3 benchmarks/_tools/validate_all.pyOnce evalport run (the CLI runner, see the main repo README) lands, this
whole hub becomes runnable directly:
evalport run benchmarks/gsm8k/gsm8k.json --provider openai --model gpt-4o-mini --dry-run| Benchmark | Suite file(s) | Cases | License | Type |
|---|---|---|---|---|
| GSM8K | gsm8k/gsm8k.json |
500 | MIT | Math word problems |
| ARC-Easy | arc-easy/arc-easy.json |
500 | CC-BY-SA-4.0 | Science MC |
| ARC-Challenge | arc-challenge/arc-challenge.json |
500 | CC-BY-SA-4.0 | Science MC (harder) |
| BoolQ | boolq/boolq.json |
500 | CC-BY-SA-3.0 | Yes/no reading comprehension |
| HellaSwag | hellaswag/hellaswag.json |
500 | MIT | Commonsense sentence completion |
| WinoGrande | winogrande/winogrande.json |
500 | CC-BY | Pronoun resolution |
| CommonsenseQA | commonsenseqa/commonsenseqa.json |
500 | MIT | Commonsense MC |
| PIQA | piqa/piqa.json |
500 | AFL-3.0 | Physical commonsense |
| TruthfulQA | truthfulqa/truthfulqa.json |
817 | Apache-2.0 | Truthfulness (open-ended) |
| MMLU | 6 files, one per subject | 1,044 | MIT | Academic knowledge MC |
| HumanEval | humaneval/humaneval.json |
164 | MIT | Python code generation |
| MBPP | mbpp/mbpp.json |
300 | CC-BY-4.0 | Python code generation |
| SQuAD 2.0 | squad2/squad2.json |
500 | CC-BY-SA-4.0 | Reading comprehension (+ unanswerable) |
| DROP | drop/drop.json |
500 | CC-BY-SA-4.0 | Discrete reasoning over paragraphs |
| BBH | 3 files, one per task | 687 | MIT | Hard multi-step reasoning |
| Total | 22 suite files | 8,012 |
Every benchmark's license was verified from its primary source (dataset
card or repo LICENSE) before inclusion — see LICENSES.md
for the full attribution table, licensing notes, and one candidate
(LAMBADA) that was deliberately excluded over unclear redistribution
rights.
Every suite is a single JSON file conforming to the EvalPort EvalSuite
schema (version, id, name, test_cases, graders, metadata). Two
grader architectures are used, depending on whether a benchmark's grading
logic is constant across all cases or varies per case:
- Shared suite-level graders (GSM8K, ARC, BoolQ, HellaSwag, WinoGrande,
CommonsenseQA, PIQA, TruthfulQA, MMLU, BBH) — a single grader definition
referenced by id from every test case, used when every case in the suite
is graded the same way (e.g.
exact_matchon a letter/number/boolean, or a fixedsemantic_similaritythreshold). - Per-case inline graders (HumanEval, MBPP, SQuAD 2.0, DROP) — each test
case embeds its own grader dict directly, used when grading parameters
genuinely vary per case (a unique unit-test harness per coding problem, a
unique expected substring per QA pair). This is a deliberate architecture
choice driven by reading the SDK's own validator:
validate_suite'sDANGLING_REFERENCEcheck only fires for string-type grader references in test cases, so a shared suite-level grader has no way to hold per-case-varying parameters — inline dict graders are the only spec-valid way to express this.
Every suite also carries a consistent metadata block for provenance and reproducibility:
"metadata": {
"openeval": {"source": "evalport-benchmarks"},
"evalport.source": "<link to the dataset's own source>",
"evalport.source_license": "<verified license>",
"evalport.original_paper": "<arXiv link>",
"evalport.converted_by": "evalport-benchmarks/_tools/convert_hf_dataset.py",
"evalport.conversion_date": "<date this suite was generated>",
"evalport.case_count": <int, derived from the actual written cases>
}All suites in this hub — except _tools/convert_jsonl.py-based ones, none
of which exist yet — are produced by one reusable script,
_tools/convert_hf_dataset.py, rather than
20 one-off conversion scripts. It pulls each benchmark from HuggingFace
Datasets, applies benchmark-specific field mapping (documented per-handler
in the script itself and per-benchmark in each subdirectory's README),
validates every suite against the real openeval.validate.validate_suite()
before writing it to disk, and refuses to write anything that fails
validation.
cd _tools
python3 convert_hf_dataset.py gsm8k # convert one benchmark (uses its default --limit)
python3 convert_hf_dataset.py mmlu --limit 50 # override the case limit
python3 convert_hf_dataset.py all # regenerate the entire hub
python3 validate_all.py # validate every suite under benchmarks/ (the CI gate)Adding a new benchmark means adding one handler function + one REGISTRY
entry to convert_hf_dataset.py, not a new script.
Most suites cap at 500 cases (a size chosen to keep suite files small and git-diff-friendly while still being a meaningful sample) even when the source split is larger; TruthfulQA (817) and HumanEval (164) are included in full since their native splits are already smaller than the default cap. Every README linked in the index above states the exact case count and, for capped suites, how many cases were available upstream.
Every suite here is redistributed well beyond a direct git clone of this
repo — mirrors, package managers, someone's copy pasted into their own
project. Nothing about the EvalPort document format itself proves a suite
you downloaded from one of those is byte-for-byte what this repo actually
published; metadata.source is a claim, not a proof. Starting with the
first release cut after 1.0.0-rc.4, CI signs every file in this directory
with a keyless Sigstore signature (resolves
Discussion #8 —
see spec/SPEC.md's Extension Mechanism → Suite/ResultSet Signing for the
full convention) and attaches the signature bundles to that GitHub
Release.
To verify a suite you downloaded actually matches what this repo signed:
pip install sigstore
# 1. Download the suite itself, e.g. benchmarks/gsm8k/gsm8k.json
# 2. Download that release's benchmark-signatures-<tag>.tar.gz asset from
# https://github.com/adhabnr-ux/evalport/releases and extract it
# 3. Verify:
python3 spec/tools/verify_signature.py verify gsm8k.json \
--bundle gsm8k.json.sigstore.json \
--cert-identity-regex '^https://github\.com/adhabnr-ux/evalport/\.github/workflows/ci\.yml@refs/tags/.*$' \
--cert-oidc-issuer https://token.actions.githubusercontent.comExit code 0 means the file is exactly what this repo's release CI
published. This is entirely optional — every suite here is just as
spec-valid unsigned as signed, and _tools/validate_all.py (above) is
what actually confirms a suite is well-formed. Signing only answers "did
this specific file change since the release," a different and narrower
question than "is this a valid EvalPort suite."
- Verify the benchmark's license permits data redistribution — check the
HuggingFace dataset card's "Licensing Information" section or the
source repo's
LICENSEfile directly. Don't assume from a benchmark's popularity or how another framework classifies it. - Add a handler function to
_tools/convert_hf_dataset.py(or_tools/convert_jsonl.pyif the source isn't HF-hosted) plus aREGISTRYentry. - Run it, confirm
valid=Truein the script's own output. - Add an entry to this table, a per-benchmark
README.mdin the new subdirectory (source/license/citation/grader rationale/case count — match the shape of any existing benchmark README), and an entry to LICENSES.md. - Run
_tools/validate_all.pyto confirm the whole hub — not just your new suite — still passes.