Extend key entities metric to audio-native models by weiz9 · Pull Request #180 · ServiceNow/eva

weiz9 · 2026-07-10T17:43:33Z

What

Extends the existing transcription_accuracy_key_entities metric to support audio-native pipelines (S2S / AUDIO_LLM). It stays a single text-to-text judge — no new metric class.

Previously the metric was CASCADE-only: it compared what the user was instructed to say (intended_user_turns) against what the agent's STT transcribed (transcribed_user_turns). Audio-native models expose no reliable STT transcript, so entity understanding was unmeasured for them.

How

The metric now branches on pipeline type:

CASCADE (unchanged): compares intended_user_turns vs transcribed_user_turns (STT output) — transcription accuracy.
Audio-native (new): compares the user's intended entities against the agent's tool-call arguments (from conversation_trace), which are the evidence of what the model actually understood. If the user says "my id is EM123" and the agent calls a tool with "EMP124", the entity is flagged as misheard.

Both paths reuse the same per-entity ratio scoring (correct / total) and per-entity-type sub-metrics. Audio-native uses a new audio_native_prompt judge template; the cascade user_prompt is untouched.

Changes

transcription_accuracy_key_entities.py: removed the CASCADE-only restriction (now all pipeline types), added _build_prompt_and_turns() to select source + prompt via context.is_audio_native, plus _get_audio_native_turns() / _format_intended_user_turns(). Bumped metric version v0.3 → v0.4.
configs/prompts/judge.yaml: added audio_native_prompt under the same metric key.
Tests: added TestPipelineSupport and TestAudioNativeCompute (misheard → 0, correct → 1, verifies the audio-native prompt is used with tool-call evidence).
Bumped metrics_version 2.2.0 → 2.3.0 and regenerated metric_signatures.json.

Validation

Unit tests: 17 passing (existing cascade cases + new audio-native cases); full metrics suite green (423), including the drift/signature test. ruff clean.

Audio-native behavioral check — real AUDIO_LLM run (nowllm-speech), 5 conversations × 3 trials:

Metric now activates on audio-native runs (was silently filtered out before), mean score 0.92, no errors.
Correctly routed to the audio_native_prompt and compared user entities against tool-call arguments.
All 11 incorrect verdicts were genuine mishears, e.g. conf. code ZK3FFW→VK3FFW, 6VORJU→6BORTU, name Thompson→Lee, destination Washington DC→DFW (Dallas). No false positives on correctly-captured entities.

Cascade regression — 3 cascade runs (xai-stt, flux, pulse), 9 trials:

Every generated judge prompt is byte-identical to the prior v0.3 output → cascade input path provably unchanged.
All trials routed to the cascade prompt (never audio-native); identical entity/eval counts; no errors.
7/9 scores identical; 2 differ by ~0.04–0.06 — attributable to LLM judge nondeterminism (identical inputs, judge runs without fixed temperature), not code.

Notes

Open item (minor, non-blocking): the audio-native judge currently treats soft constraints ("under $80", "before 2 PM") as entities and marks them correct when the agent's action satisfies them. Defensible, but broadens scope beyond literal identifier capture — a prompt-tuning tweak if we want it limited to hard identifiers (codes, names, IDs).
Signature coverage gap: the drift signature only hashes judge.{name}.user_prompt, so future text edits to the new audio_native_prompt won't auto-trip the drift test — they'd need a manual version bump. Happy to extend signatures.py to hash all prompt keys if preferred.
Pre-existing mypy errors in this file (MetricScore pydantic defaults under strict) are unchanged; the new code adds none.

Extend transcription_accuracy_key_entities to support S2S / AUDIO_LLM pipelines. Audio-native models expose no reliable STT transcript, so the agent's tool-call arguments (from conversation_trace) become the evidence of what it understood: if the user says "my id is EM123" and the agent calls a tool with "EMP124", the entity is flagged as misheard. The metric now branches on pipeline type: - CASCADE: compares intended vs transcribed user turns (unchanged). - Audio-native: compares intended user entities vs tool-call arguments, using a new audio_native_prompt judge template. Bumps the metric to v0.4, metrics_version to 2.3.0, and regenerates the metric signature fixture.

gabegma

Nice! This looks good! Only a few comments. Have you tested it? Edit: nevermind, just read the description, cool!!

gabegma · 2026-07-17T15:28:51Z

Oh, and yes, could we fix signatures, so that changes in this new prompt are also caught? Thank you!

@gabegma

…ey signatures Review feedback from @gabegma on the audio-native key-entity path: - Redact assistant turns in the judge input (show [Assistant speaks]) so a mis-transcribed agent read-back can't be used to penalize a user entity (e.g. user 'INC462', agent 'Did you say INC463?'). Judge now sees only user turns + tool calls/responses. - Collapse the duplicated user-turns section into a single redacted Conversation block (user turns shown inline), mirroring speech_fidelity. - Drop standalone honorific/title variants from the entity criteria (not entities). - Merge the two non-spoken-tag subsections into one. - signatures.py: hash ALL prompt keys under judge.{name}, not just user_prompt, so edits to audio_native_prompt are caught by the drift test. Bumps metric v0.4 -> v0.5, metrics_version 2.3.0 -> 2.4.0, regenerates signatures.

weiz9 · 2026-07-17T20:32:36Z

Oh, and yes, could we fix signatures, so that changes in this new prompt are also caught? Thank you!

Fixed! signatures.py now hashes all prompt keys under judge.{name}, so audio_native_prompt edits are caught by the drift test.

@gabegma

Address @gabegma's follow-up comments: - Rate each tool call individually (keyed by a per-call id) instead of per user turn. A user may state an entity in one turn and the agent use it several turns later; a single turn may contain multiple tool calls; and the same entity may appear in more than one call. Each labelled 'Tool Call [N]' is now its own rated unit, so a value can be wrong in one call and right in another, and no tool call is skipped when a turn has several. - Drop the now-redundant 'For example' read-back sentence (assistant turns are redacted, so the judge never sees it). - Ignore tool-call args sourced from tool responses / generated ids. - Tests: add correction-across-turns and multiple-tool-calls-in-one-turn cases. Bumps metric v0.5 -> v0.6, metrics_version 2.4.0 -> 2.5.0, regenerates signatures. Validated on a live AUDIO_LLM record: recovers entities that per-turn keying missed and rates each tool call separately.

@gabegma

Address @gabegma's follow-up comments: - Merge the two duplicate task items about ignoring tool-response/generated args. - Drop the interruption 'skipped' handling and the 'skipped' output field — an unspoken entity can't appear in a tool call, so there is nothing to skip. - Example now uses the pre-correction value (EM123) to show it is wrong once the user has corrected it; keep the entity 'value' clean (reasoning goes in analysis). - Replace the tool_call_id->turn_id normalization block with a setdefault fallback at the point of use. - Collapse to a single version bump for the whole PR (metric v0.4, metrics_version 2.3.0) instead of bumping each review round.

weiz9 · 2026-07-21T20:56:49Z

Hi @gabegma, thank you so much for the reviews! I have made the changes accordingly, please take a look when you get the chance and let me know how it looks, thank you!

weiz9 marked this pull request as ready for review July 17, 2026 04:25

gabegma reviewed Jul 17, 2026

View reviewed changes