Extend key entities metric to audio-native models

configs/prompts/judge.yaml

@@ -1071,3 +1071,154 @@ judge:
           "summary": "<1-2 sentence summary for this turn>"
       }}
       ]
+
+    audio_native_prompt: |
+      You are an expert evaluator analyzing whether an AI voice agent correctly **understood** the key entities a user provided during a spoken conversation.
+
+      This is a speech-to-speech / audio-native agent: there is no reliable speech-to-text transcript of the user to inspect. Instead, the evidence of what the agent actually understood is the **arguments it passed to its tool calls**. When a user says "my id is EM123" and the agent then calls a tool with "EMP124", the agent misheard the entity.
+
+      You are given a Conversation containing user turns, tool calls, and tool responses. **The assistant's own spoken turns are redacted** (shown as `[Assistant speaks]`) and deliberately withheld: the agent's speech is itself imperfectly transcribed, so it must never be used as evidence. For example, if the user says "INC462" and the agent asks "Did you say INC463?", that read-back may just be a transcription artifact — do not treat it as the agent mishearing the entity. Judge only what the user said (user turns) against what the agent did (tool-call arguments).
+
+      Your task:
+      1. For EACH user turn in the Conversation, identify all key entities in what the USER SAID.
+      2. For each entity, find whether the agent used it in a tool-call argument and check whether it was captured CORRECTLY.
+      3. Mark each entity as correct or incorrect.
+      4. If the agent never used a given entity in any tool call, there is no evidence to judge understanding — include the entity but mark it as skipped.
+      5. For entities in regions that were likely never spoken aloud (as indicated by interruption tags), still include them in the output but mark them as skipped.
+
+      ## What Counts as an Entity
+      An entity must have a **specific, concrete value** — something that could be passed as an input to a program or tool (not an AI, but a script or database lookup). Ask yourself: could this value be stored in a variable and used programmatically?
+
+      - Names (people, places, organizations): e.g. "John Smith", "Austin", "Delta Airlines"
+      - Specific dates and times: e.g. "December 15th", "3:45 PM" — NOT vague references like "tomorrow morning" or "later today"
+      - Confirmation codes / reference numbers: e.g. "ABC123", "ZK3FFW"
+      - Flight numbers: e.g. "UA 204"
+      - Amounts and prices: use the specific value only, e.g. "$120" — for qualifier phrases like "under $120", only use the specific value
+      - Addresses: e.g. "123 Main Street"
+      - Phone numbers: e.g. "555-867-5309"
+      - Email addresses: e.g. "john@example.com"
+      - Other specific identifiers: seat numbers, loyalty numbers, booking IDs, employee IDs, etc.
+
+      **Not an entity:** vague temporal words ("tomorrow", "next week", "morning"), general descriptors ("the cheap flight", "a long trip"), open-ended qualifiers ("less than an hour", "around noon"), or standalone honorifics/titles ("Mr.", "Ms.").
+
+      ## IMPORTANT: What This Metric Measures
+      This measures whether the agent **understood** the entity the user provided, as evidenced by the value it passed to a tool call. It is NOT a faithfulness or tool-correctness metric. Do NOT penalize:
+      - The agent choosing the wrong tool, or omitting a tool call it should have made
+      - The agent using an entity that came from a tool RESPONSE rather than from the user (only user-provided entities are evaluated here)
+      - Correct values that simply never got reused (mark those skipped, not incorrect)
+
+      Only mark an entity incorrect when the agent demonstrably used a **different value** than the user provided (e.g. user "EM123" → tool call "EMP124", user "March 25th" → tool call "2026-03-26").
+
+      ## Non-Spoken Tags in the User Text
+      The user text may contain non-spoken tags and markers — metadata that was never said aloud and must not be treated as entities. There are two kinds, and they are handled differently:
+
+      - **Audio-direction tags** like [slow], [firm], [annoyed] describe how the words were meant to be spoken. Ignore them entirely; they have no effect on scoring.
+      - **Interruption tags** indicate that part of the user text was likely never spoken, because the user was interrupted or talked over. An entity that was never spoken cannot have been understood, so do NOT penalize entities in these regions — include them but mark them as skipped.
+
+      {interruption_tags_reference}
+
+      **Key principle:** Only evaluate entities that were reasonably expected to have been spoken aloud AND that the agent used in a tool call. If a tag indicates the user was interrupted before or during an entity, mark it skipped and explain why.
+
+      ## Conversation
+      {conversation}
+
+      ## Correctness Criteria
+      - Entity must have been used correctly by the agent — the value passed to the tool call must match what the user said.
+      - Numbers: "150" and "one hundred fifty" are equivalent
+      - Dates: "December 15th" and "Dec 15" are equivalent
+      - Names: Case-insensitive match. Phonetically equivalent spelling variants are acceptable on the **first mention** of a name (e.g., German Meyer/Meier/Maier/Mayer — all sound the same when spoken). Once the user has explicitly disambiguated the spelling (e.g., spelled it out letter by letter, or corrected a prior spelling), subsequent usage must match that clarified spelling exactly. Flag if:
+        - The value the agent used clearly differs (different sound, not just different spelling)
+        - The agent used a different spelling variant **after** the user has explicitly clarified the correct spelling
+      - Non-Latin scripts: A name or identifier rendered in its native writing system (e.g. Kanji, Hangul, Arabic) is equivalent to its romanized/phonetic form if they represent the same spoken entity. For example, "山田" and "Yamada" should be treated as matching.
+      - Phonetic letter spellings: In languages that do not use the Latin alphabet, Roman/Latin letters are spoken using language-specific phonetic equivalents. These are correct, not errors (e.g., in Korean "에이" for "A" or "비" for "B").
+      - Number surface variants: Many languages have systematic spoken-form differences that do not change the underlying value and must not be penalized:
+        - Reversed digit order in spoken form (e.g., German "vierundzwanzig" = four-and-twenty = 24)
+        - Grammatical gender/agreement variants (e.g., Spanish "veintiún" vs "veintiuna" for 21)
+        - Large-unit grouping differences (e.g., Japanese 万-system: "ichi-man go-sen" / 一万五千 = 15,000)
+
+      Important note: The user text will often feature things formatted like "one two three" instead of "123". Your goal is to evaluate semantic equivalence — these are considered equivalent if the value matches what the agent used.
+
+      ## Examples
+
+      **Example Input (Conversation):**
+      Turn 1 - User: My employee id is E M one two three and my name is John Smith.
+      Turn 1 - [Assistant speaks]
+      Turn 1 - Tool Call (lookup_employee): {{'employee_id': 'EMP124'}}
+      Turn 2 - User: Book it for December 15th.
+      Turn 2 - [Assistant speaks]
+      Turn 2 - Tool Call (create_booking): {{'date': '2026-12-15'}}
+      Turn 3 - User: [slow] The code is X X F six O H, with the letter O, [assistant interrupts] not zero.
+      Turn 3 - [Assistant speaks]
+
+      **Example Response:**
+      [
+      {{
+          "turn_id": 1,
+          "entities": [
+          {{
+              "type": "employee_id",
+              "value": "E M one two three",
+              "captured_value": "EMP124",
+              "analysis": "User said EM123 but the agent called lookup_employee with EMP124 — misheard.",
+              "correct": false,
+              "skipped": false
+          }},
+          {{
+              "type": "name",
+              "value": "John Smith",
+              "captured_value": "not used",
+              "analysis": "The name was never passed to a tool call, so there is no evidence of how the agent understood it.",
+              "correct": false,
+              "skipped": true
+          }}
+          ],
+          "summary": "1 entity used (employee_id, incorrect). Name never used — skipped."
+      }},
+      {{
+          "turn_id": 2,
+          "entities": [
+          {{
+              "type": "date",
+              "value": "December 15th",
+              "captured_value": "2026-12-15",
+              "analysis": "Agent passed 2026-12-15 to create_booking — matches December 15th.",
+              "correct": true,
+              "skipped": false
+          }}
+          ],
+          "summary": "1 entity correct."
+      }},
+      {{
+          "turn_id": 3,
+          "entities": [
+          {{
+              "type": "confirmation_code",
+              "value": "X X F six O H",
+              "captured_value": "not used",
+              "analysis": "The agent never used this code in a tool call — no evidence to judge, skipped.",
+              "correct": false,
+              "skipped": true
+          }}
+          ],
+          "summary": "1 entity found; never used by the agent — skipped."
+      }}
+      ]
+
+      ## Response Format
+      Respond with a JSON object. Each turn entry must include the turn_id matching the user turn number shown in the Conversation above:
+      [
+      {{
+          "turn_id": <int: the user turn number from the Conversation>,
+          "entities": [
+          {{
+              "type": "<name|date|time|confirmation_code|flight_number|amount|address|phone|email|employee_id|etc...>",
+              "value": "<entity value as the user said it>",
+              "captured_value": "<the value the agent used in a tool call, or 'not used'>",
+              "analysis": "<brief reason; if skipped, explain why (never used by the agent, or in an interrupted region)>",
+              "correct": <true|false>,
+              "skipped": <true|false>
+          }}
+          ],
+          "summary": "<1-2 sentence summary for this turn>"
+      }}
+      ]

src/eva/__init__.py

@@ -11,4 +11,4 @@
 
 # Bump metrics_version when changes affect metric computation (metrics code,
 # judge prompts, pricing tables, postprocessor).
-metrics_version = "2.2.0"
+metrics_version = "2.4.0"

src/eva/metrics/diagnostic/transcription_accuracy_key_entities.py

@@ -1,4 +1,16 @@
-"""Transcription accuracy key entities metric using LLM-as-judge (entire conversation).
+"""Key entity accuracy metric using LLM-as-judge (entire conversation).
+
+Measures whether the user's key entities (names, dates, confirmation codes,
+amounts, etc.) survived intact through the agent's pipeline. The comparison
+target depends on the pipeline type:
+
+- **CASCADE**: compares what the user was instructed to say (``intended_user_turns``)
+  against what the agent's STT transcribed (``transcribed_user_turns``) — i.e.
+  transcription accuracy.
+- **Audio-native (S2S / AUDIO_LLM)**: there is no reliable STT output to inspect,
+  so the agent's **tool-call arguments** (from ``conversation_trace``) become the
+  evidence of what it understood. If the user says "my id is EM123" and the agent
+  then calls a tool with "EMP124", the agent misheard the entity.
 
 Debug metric for diagnosing model performance issues, not directly used in
 final evaluation scores.
@@ -14,19 +26,24 @@
     parse_judge_response_list,
     resolve_turn_id,
 )
-from eva.models.config import PipelineType
 from eva.models.results import MetricScore
 
 
 @register_metric
 class TranscriptionAccuracyKeyEntitiesMetric(TextJudgeMetric):
-    """LLM-based transcription accuracy metric for key entities only (entire conversation).
+    """LLM-based key entity accuracy metric (entire conversation).
 
-    Evaluates STT transcription accuracy by comparing key entities (names, dates,
-    confirmation codes, amounts, etc.) between what the user was supposed to say
-    (intended_user_turns) and what STT transcribed (transcribed_user_turns).
+    Evaluates whether the user's key entities (names, dates, confirmation codes,
+    amounts, etc.) survived intact through the agent's pipeline. The comparison
+    target depends on the pipeline type:
 
-    Computes the ratio of correctly transcribed entities per turn:
+    - **CASCADE**: compares ``intended_user_turns`` against ``transcribed_user_turns``
+      (STT output) — transcription accuracy.
+    - **Audio-native (S2S / AUDIO_LLM)**: no reliable STT output exists, so the agent's
+      tool-call arguments (from ``conversation_trace``) are used as the evidence of
+      what it understood.
+
+    Computes the ratio of correctly captured entities per turn:
     score = correct_entities / total_entities
 
     Entity types evaluated:
@@ -48,11 +65,12 @@ class TranscriptionAccuracyKeyEntitiesMetric(TextJudgeMetric):
     """
 
     name = "transcription_accuracy_key_entities"
-    version = "v0.3"
-    description = "Debug metric: LLM judge evaluation of STT key entity transcription accuracy for entire conversation"
+    version = "v0.5"
+    description = (
+        "Debug metric: LLM judge evaluation of user key entity accuracy (STT in cascade, tool calls in audio-native)"
+    )
     category = "diagnostic"
     exclude_from_pass_at_k = True
-    supported_pipeline_types = frozenset({PipelineType.CASCADE})
     rating_scale = None  # Custom scoring (not 1-3 scale)
     default_aggregation = "mean"
 
@@ -70,9 +88,7 @@ async def compute(self, context: MetricContext) -> MetricScore:
             MetricScore with aggregated score and per-turn details
         """
         try:
-            turns_to_evaluate = self._get_turns_to_evaluate(context)
-            user_turns_text = self._format_user_turns(turns_to_evaluate, context)
-            prompt = self.get_judge_prompt(user_turns=user_turns_text)
+            prompt, turns_to_evaluate = self._build_prompt_and_turns(context)
 
             response_text = await self._call_judge_raw(prompt, context)
             turn_evaluations = parse_judge_response_list(response_text)
@@ -206,21 +222,78 @@ def _build_per_entity_type_sub_metrics(
             )
         return sub_metrics
 
+    def _build_prompt_and_turns(self, context: MetricContext) -> tuple[str, list[int]]:
+        """Select the comparison source and judge prompt based on pipeline type.
+
+        CASCADE compares intended vs STT-transcribed user turns. Audio-native
+        pipelines (S2S / AUDIO_LLM) have no reliable STT, so the user's intended
+        turns are compared against the agent's tool-call arguments (from
+        ``conversation_trace``), which are the evidence of what it understood.
+
+        Returns:
+            Tuple of (rendered judge prompt, sorted turn IDs to evaluate).
+        """
+        if context.is_audio_native:
+            turn_ids = self._get_audio_native_turns(context)
+            prompt = self.get_judge_prompt(
+                prompt_key="audio_native_prompt",
+                conversation=self._format_redacted_conversation(context),
+            )
+            return prompt, turn_ids
+
+        turn_ids = self._get_turns_to_evaluate(context)
+        prompt = self.get_judge_prompt(user_turns=self._format_user_turns(turn_ids, context))
+        return prompt, turn_ids
+
     @staticmethod
     def _get_turns_to_evaluate(context: MetricContext) -> list[int]:
-        """Return sorted turn IDs present in both tts_text_user and transcript_user."""
+        """Return sorted turn IDs present in both intended and transcribed user turns."""
         return sorted(context.intended_user_turns.keys() & context.transcribed_user_turns.keys())
 
+    @staticmethod
+    def _get_audio_native_turns(context: MetricContext) -> list[int]:
+        """Return sorted intended user turn IDs (ground truth for audio-native pipelines)."""
+        return sorted(context.intended_user_turns.keys())
+
     @staticmethod
     def _format_user_turns(turn_ids: list[int], context: MetricContext) -> str:
-        """Format user turns for the prompt."""
+        """Format expected-vs-transcribed user turn pairs for the cascade prompt."""
         return "\n\n".join(
             f"Turn {tid}:\n"
             f'Expected: "{context.intended_user_turns[tid]}"\n'
             f'Transcribed: "{context.transcribed_user_turns[tid]}"'
             for tid in turn_ids
         )
 
+    @staticmethod
+    def _format_redacted_conversation(context: MetricContext) -> str:
+        """Format a redacted conversation trace for the audio-native judge.
+
+        User turns and tool calls/responses are shown verbatim — the user turns are
+        the entity source, the tool calls are the evidence of what the agent understood.
+        Assistant turns are redacted to ``[Assistant speaks]`` so the judge cannot use
+        the agent's own (possibly mis-transcribed) speech to penalize an entity: e.g. if
+        the user says "INC462" and the agent asks "Did you say INC463?", that read-back
+        is an artifact of the agent's transcription, not evidence the user was misheard.
+        Only tool-call arguments and explicit user turns should drive the score.
+        """
+        lines: list[str] = []
+        for entry in context.conversation_trace or []:
+            role = entry.get("role")
+            entry_type = entry.get("type")
+            turn_id = entry.get("turn_id", "?")
+            if role == "user":
+                lines.append(f"Turn {turn_id} - User: {entry.get('content', '')}")
+            elif role == "assistant":
+                lines.append(f"Turn {turn_id} - [Assistant speaks]")
+            elif entry_type == "tool_call":
+                params = entry.get("parameters", {})
+                lines.append(f"Turn {turn_id} - Tool Call ({entry.get('tool_name', 'unknown')}): {params}")
+            elif entry_type == "tool_response":
+                resp = entry.get("tool_response", {})
+                lines.append(f"Turn {turn_id} - Tool Response ({entry.get('tool_name', 'unknown')}): {resp}")
+        return "\n".join(lines)
+
     async def _call_judge_raw(self, prompt: str, context: MetricContext) -> str | None:
         """Call LLM judge and return raw response text.
 

src/eva/metrics/signatures.py

@@ -64,14 +64,19 @@ def _source_hash(cls: type) -> str:
 def _prompt_hash_for_metric(cls: type[BaseMetric]) -> str | None:
     """Return the prompt template hash for judge metrics, or None.
 
-    All judge metrics in this codebase use `judge.{name}.user_prompt`.
-    A judge metric without a corresponding template raises KeyError —
-    that's a configuration bug we want surfaced.
+    Hashes *every* prompt template under ``judge.{name}`` (sorted by key), not just
+    ``user_prompt``, so a metric that ships multiple prompts — e.g. a per-pipeline
+    variant like ``audio_native_prompt`` — has all of them covered by drift detection.
+    A judge metric without any templates raises KeyError — that's a configuration
+    bug we want surfaced.
     """
     if not issubclass(cls, TextJudgeMetric | AudioJudgeMetric):
         return None
-    template = get_prompt_manager().get_template(f"judge.{cls.name}.user_prompt")
-    return hash_prompt_template(template)
+    templates = get_prompt_manager().prompts.get("judge", {}).get(cls.name)
+    if not isinstance(templates, dict):
+        raise KeyError(f"No judge prompts found for metric {cls.name!r}")
+    combined = "\n".join(templates[key] for key in sorted(templates) if isinstance(templates[key], str))
+    return hash_prompt_template(combined)
 
 
 def compute_all_metric_signatures() -> dict[str, dict[str, str | None]]:

tests/fixtures/metric_signatures.json

@@ -85,9 +85,9 @@
   },
   "TranscriptionAccuracyKeyEntitiesMetric": {
     "name": "transcription_accuracy_key_entities",
-    "prompt_hash": "877dfa61f232",
-    "source_hash": "70e92b2f7221",
-    "version": "v0.3"
+    "prompt_hash": "3ed3a2fe7dcb",
+    "source_hash": "2926af63f4da",
+    "version": "v0.5"
   },
   "TurnTakingMetric": {
     "name": "turn_taking",