Skip to content

evals.md

Latest commit

 

History

History
97 lines (72 loc) · 4.44 KB

evals.md

File metadata and controls

97 lines (72 loc) · 4.44 KB
Raw

Evaluation harness

laya.evals turns a labelled dataset into a repeatable score, and a baseline into a pass/fail gate, so a quality change is a reviewable diff instead of a hand-check.

The metric math and the dataset parser are pure Python plus numpy and never import torch, so they run with no weights. Running a dataset against a checkpoint needs the checkpoint and takes its normal load time.

Quickstart

# check the format without a model
laya-evals validate research/evals/fixture.jsonl

# score a labelled set on one checkpoint, with thresholds and a baseline
laya-evals run data.jsonl --model english --device cpu \
    --min-accuracy 0.8 --max-ece 0.05 --slice language \
    --json report.json --markdown report.md

# compare a saved report to a baseline
laya-evals compare report.json --baseline baseline.json --tolerance choice_accuracy=0.02

laya eval ... is the same thing through the main CLI, so laya eval validate data.jsonl works too.

Exit codes: 0 on success, 1 when a threshold or a baseline tolerance fails, 2 on a usage error. run prints the overall metrics and any requested slices to stdout, and writes the full report and a Markdown summary when --json / --markdown are given.

Dataset format

One JSON object per line (JSONL). Blank lines and lines starting with # are ignored.

field required meaning
state yes text, email, ticket or JSON document to decide on
questions yes a Laya question dict, exactly as Router.predict accepts
expected yes ground truth keyed by question id: a label for choice, a number for score, true/false for noul
tags no strings to slice by
language no a code to slice by
model no force a checkpoint for this row; --model overrides it

research/evals/dataset.template.jsonl has a commented example.

Metrics

Each metric is computed per answer where it applies and aggregated over the dataset:

metric applies to meaning
choice_accuracy choice fraction whose chosen label matches
noul_accuracy noul fraction whose boolean (probability >= 0.5) matches
score_mae score mean absolute error
score_within_<tol> score fraction within an absolute tolerance
ece any answer with a confidence expected calibration error, 15 bins, computed on answer["answer_confidence"], the calibrated probability Laya reports on every answer type
mean_confidence any answer with a confidence mean reported answer["answer_confidence"]
latency_p50_ms, latency_p95_ms per request wall time, informational

Add ScoreWithin(0.25) to the evaluator list for a tolerance metric; the default set is choice_accuracy, noul_accuracy, score_mae, mean_confidence, plus ece.

Slices

compare and run report overall numbers and, for --slice language|model|qid|tag, the same metrics per slice value, so a regression in one language or one question is visible without reading the aggregate.

Baseline and CI gate

  • Keep the dataset, a baseline report (--json output you have reviewed), and the tolerances together, committed, so a change is a reviewable diff. --tolerance METRIC=VALUE is the maximum absolute drift allowed for that metric.
  • laya-evals run ... --baseline baseline.json --tolerance ... exits non-zero on drift, so it drops into CI unchanged. laya.evals.EvalReport.compare and assert_regression expose the same logic for tests.

Two CI surfaces use this:

  • a weight-free job in .github/workflows/ci.yml runs tests/test_evals.py and tests/test_evals_api.py, so metric math, dataset parsing and the CLI are covered on every PR without downloading a checkpoint;
  • .github/workflows/evals.yml runs weekly, before a release and on demand: it evaluates the English checkpoint on the MASSIVE English suite and compares to research/results/eval_english_51_languages.json with the tolerances in research/evals/thresholds.json. It uploads the report as an artifact and does not block a PR.

The harness is deterministic for a fixed checkpoint revision, so a report is reproducible. run records the dataset, model and device in the report's config block.

Adding the real labelled set

Drop a JSONL in research/evals/ and a reviewed baseline beside it, then point a workflow (or research/evals/check_regression.py) at both. The format is the same as the fixture; nothing in the harness knows about MASSIVE.