TB empty-gold fairness: pure_refusal-only gate under full catalog

[Dominic Nguyen (datduyng)]

Summary

Empty-gold TB negatives (expectedActions: []) are scored as zero actions across the full loaded tool catalog. Label-fair Q&A/definition negatives were not zero-action-scorer-fair: models correctly fire chat/help/history, tanking negative pass.

This PR ships a pipeline fairness contract so empty gold is only accepted for hard abstains:

• Synthesizer + quality-verifier prompts: empty-gold kind is pure_refusal only (leave-alone / don't / never / cancel of the target; no questions, no alternate task).
• negativeFairness module: path-keyed 1:1 negativeAssessments (no regex path parse); fail-closed on missing/duplicate paths; fairEmptyGold only when kind is in the allowlist (TRANSLATION_BENCH_FAIR_EMPTY_GOLD_KINDS).
• Partial generation: when complete coverage is optional, emitted caseCount / coverage metadata match the cases actually produced (so validation can succeed).
• No gold-row hand edits and no empty-gold scorer loosening.

Measured 3-model eval (honest)

Regenerated draft under the new policy (998/1000 rows; 2 slots exhausted data-quality, unrelated to fairness), then evaluated azure/gpt-5.6-{sol,terra,luna}.

Metric	Result
Rows / empty-gold negs	998 / 998
Kind mix	100% pure_refusal
Unfair rate (LLM audit)	0%
Overall pass	36.8%
Negative pass (zero fire on empty gold)	13.9%
Neg fire-on-empty	47.9%
Positive pass	59.8%
Pos abstention FNR	3.2%
Tool / param	83.5% / ~71.6%
Eval cost	~$190

Finding: fairness regen fixed the dataset (kind mix + unfair rate), but did not produce a large negative-pass lift. Checkpoint inspection shows models often emit the refused action itself on clear hard-abstains (negation-following failure). Translator/prompt work is out of scope here.

Test plan

• pnpm run build in ts/packages/benchmarks
• pnpm run jest-esm --testPathPattern='(negativeFairness|datasetGenerator)\\.spec\\.js' (43/43)
• Smoke + full regen fairness audit (0% unfair, 100% pure_refusal)
• 3-model headless eval on regenerated draft; scale-metrics recorded

---

Infra: extract cross-process TPM limiter + run config to core

Follow-on refactor moving the run harness's rate limiter and config loader into @typeagent/benchmarks as tested, reusable modules.

Interface (one owner per concern):

• createRateLimiter(tpmLimits, { dbPath, estTokensPerCall?, maxWaitMs?, onWait? }) → { disabledFor, run, close }. dbPath is required so callers own persistence; no hidden default DB. Optional knobs are orthogonal.
• runConfig.ts is pure: reads a JSON file (schema-validated), merges batch over base, derives per-model concurrency. No env, no process globals — runtime overrides are commander flags prop-drilled by the local runners (TB_* env removed).
• Credential env (OPENAI_*/AZURE_*) is intentionally kept as the aiclient contract; documented in AGENTS.md.

Correctness:

• TPM is enforced by a shared SQLite ledger over a rolling 60s window (BEGIN IMMEDIATE, WAL): concurrent awaited run() calls reserve then settle to actual token usage, staying within quota across processes.
• In-flight claims survive purge to STALE_MS (purge guarded on pending = 0); settle falls back to insert if the claim row was purged.
• Fail-closed: throws on missing positive token estimate; omits non-positive tpm limits.

Tests: translationBench.rateLimiter.spec + translationBench.runConfig.spec (17 cases) — passthrough for unquota'd models, budget admit/throttle, cross-instance shared budget, estimate fallbacks, batch merge, concurrency derivation, malformed-json handling.

Test plan (infra)

• pnpm run build + pnpm run prettier clean in ts/packages/benchmarks
• pnpm run jest-esm --testPathPattern='translationBench.(rateLimiter|runConfig).spec.js' (17/17)
• Commander parse + resolveRunnerConfig smoke against real config.local.json

[Copilot]

Pull request overview

Adds a fairness contract ensuring empty-gold translation-bench negatives represent only pure refusals.

Changes:

• Adds path-keyed LLM fairness assessments and enforcement.
• Updates generation/verifier prompts and gold-action validation.
• Supports concurrent generation and accurate partial-result metadata.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.AUTOGEN.md	Refreshes generated documentation metadata.
actionValidation.ts	Normalizes gold actions for schema validation.
benchmark.ts	Uses normalized gold-action validation.
dataQualityVerifier.ts	Enforces semantic negative-fairness assessments.
datasetGenerator.ts	Adds concurrency and partial-generation handling.
generationCandidate.ts	Preserves required parameter objects during cleanup.
index.ts	Exports the fairness module.
negativeFairness.ts	Implements fairness parsing and enforcement.
quality-verifier.prompt.yaml	Defines full-catalog fairness review requirements.
synthesizer.prompt.yaml	Restricts empty-gold negatives to pure refusals.
utteranceDisambiguation.ts	Documents separate negative handling.
translationBench.datasetGenerator.spec.ts	Updates generation fixtures for fairness assessments.
translationBench.negativeFairness.spec.ts	Tests fairness validation and semantic enforcement.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Dominic Nguyen (datduyng)]

Benchmark of record: 1k generation + 3-model eval on the TPM-limiter config (2026-08-10, this branch tip b77c420 — no code changes; run tooling is gitignored local/).

Generation: 905/1000 rows accepted (8 checkpoint-resume rounds; 95 data-quality-gate rejections — probe disagreement + duplicate-utterance dominant, zero infra residual). Trust gate PASS against the frozen source pool.

Eval (5,430 cells = 905 rows × 2 suite cases × 3 models, recomputed from raw cells):

Model	Pass	Exact	FNR	FPR
azure/gpt-4.1	66.7%	59.3%	3.1%	56.9%
azure/gpt-4.1-mini	65.3%	57.9%	5.3%	58.3%
azure/gpt-5.4-nano	67.0%	59.8%	5.1%	54.8%
overall	66.3%	59.0%	4.5%	56.6%

Schema-valid 98.9%. Zero harness/infra unknown-throw cells: 2,450 transient 429 + 164 no-response cells retried to zero via checkpoint-retry (shared Azure account contention, not per-model quota); the 57 remaining error cells are model-content JSON-validation failures, scored as failures.

Review chain: gen-100 full-coverage row review 99/99 PASS; gen-1000 sample review 50/50 PASS; failed-cell adversarial spot-check 21/21 model failures; final adversarial panel (metrics honesty / reproducibility / stale-artifact hunt) PASS after 2 must-fix rounds. Baseline comparison (old gpt-5.6 trio, 74.1%): different models, harness, and dataset — numbers are the new benchmark of record, not an apples-to-apples rerun; deltas documented in eval-summary.txt.

Run dir of record: ts/packages/benchmarks/local/runs/1k-20260807-neg-fairness/ (eval-summary.txt, RUNNER-CHANGES.md, reviews/, logs/, tools/).