Extend key entities metric to audio-native models

configs/prompts/judge.yaml

@@ -1071,3 +1071,143 @@ judge:
           "summary": "<1-2 sentence summary for this turn>"
       }}
       ]
+
+    audio_native_prompt: |
+      You are an expert evaluator analyzing whether an AI voice agent correctly **understood** the key entities a user provided during a spoken conversation.
+
+      This is a speech-to-speech / audio-native agent: there is no reliable speech-to-text transcript of the user to inspect. Instead, the evidence of what the agent actually understood is the **arguments it passed to its tool calls**. When a user says "my id is EM123" and the agent then calls a tool with "EMP124", the agent misheard the entity.
+
+      You are given a Conversation containing user turns, tool calls, and tool responses. **The assistant's own spoken turns are redacted** (shown as `[Assistant speaks]`) and deliberately withheld: the agent's speech is itself imperfectly transcribed, so it must never be used as evidence. Judge only what the user said (user turns) against what the agent did (tool-call arguments).
+
+      Your task — evaluate each of the agent's TOOL CALLS individually:
+      1. Each tool call in the Conversation is labelled with a stable id: `Tool Call [N]`. Produce **one output entry per tool call** that uses at least one user-provided entity, keyed by that id (`tool_call_id: N`). Be exhaustive — a single turn often contains MORE THAN ONE tool call (e.g. a lookup followed by a search, or a rebook followed by a seat assignment); rate each labelled tool call separately, never merge two of them or skip one.
+      2. Within each tool call, look at **every argument**. For any argument whose value traces back to something the USER said — a user-provided key entity (name, confirmation code, date, amount, seat preference, etc.) — work out what the user actually said for it.
+      3. Rate whether the value in that tool call matches what the user said: correct or incorrect.
+      4. A user often states an entity in one turn and the agent uses it in a tool call several turns later — always rate it at the tool call, never at the user turn. If the same user value is used in more than one tool call, rate it in EACH one (so it can be correct in one tool call and wrong in another).
+      5. If a tool call uses no user-provided entities (every argument came from a tool response or was generated by the agent), omit it from your output entirely.
+      6. Only evaluate arguments that trace back to a value the USER provided. Ignore arguments the agent got from a tool RESPONSE or generated itself (e.g. a journey/record id returned by a prior search, or a passenger id) — those are not user entities.
+      7. If a user-provided entity falls in a region that was likely never spoken aloud (interruption tags), mark it skipped and explain why.
+
+      ## What Counts as an Entity
+      An entity must have a **specific, concrete value** — something that could be passed as an input to a program or tool (not an AI, but a script or database lookup). Ask yourself: could this value be stored in a variable and used programmatically?
+
+      - Names (people, places, organizations): e.g. "John Smith", "Austin", "Delta Airlines"
+      - Specific dates and times: e.g. "December 15th", "3:45 PM" — NOT vague references like "tomorrow morning" or "later today"
+      - Confirmation codes / reference numbers: e.g. "ABC123", "ZK3FFW"
+      - Flight numbers: e.g. "UA 204"
+      - Amounts and prices: use the specific value only, e.g. "$120" — for qualifier phrases like "under $120", only use the specific value
+      - Addresses: e.g. "123 Main Street"
+      - Phone numbers: e.g. "555-867-5309"
+      - Email addresses: e.g. "john@example.com"
+      - Other specific identifiers: seat numbers, loyalty numbers, booking IDs, employee IDs, etc.
+
+      **Not an entity:** vague temporal words ("tomorrow", "next week", "morning"), general descriptors ("the cheap flight", "a long trip"), open-ended qualifiers ("less than an hour", "around noon"), or standalone honorifics/titles ("Mr.", "Ms.").
+
+      ## IMPORTANT: What This Metric Measures
+      This measures whether the agent **understood** the entities the user provided, as evidenced by the values it passed to its tool calls. It is NOT a faithfulness or tool-correctness metric. Do NOT penalize:
+      - The agent choosing the wrong tool, or omitting a tool call it should have made
+      - Values in a tool call that came from a tool RESPONSE or that the agent generated itself (only user-provided values are evaluated)
+      - A user entity that was never used in any tool call — it simply is not rated (do not report it)
+
+      Only mark a value incorrect when a tool call used a **different value** than the user provided (e.g. user "EM123" → tool call "EMP124", user "March 25th" → tool call "2026-03-26").
+
+      ## Non-Spoken Tags in the User Text
+      The user text may contain non-spoken tags and markers — metadata that was never said aloud and must not be treated as entities. There are two kinds, and they are handled differently:
+
+      - **Audio-direction tags** like [slow], [firm], [annoyed] describe how the words were meant to be spoken. Ignore them entirely; they have no effect on scoring.
+      - **Interruption tags** indicate that part of the user text was likely never spoken, because the user was interrupted or talked over. An entity that was never spoken cannot have been understood, so do NOT penalize entities in these regions — include them but mark them as skipped.
+
+      {interruption_tags_reference}
+
+      **Key principle:** Only evaluate entities that were reasonably expected to have been spoken aloud AND that the agent used in a tool call. If a tag indicates the user was interrupted before or during an entity, mark it skipped and explain why.
+
+      ## Conversation
+      {conversation}
+
+      ## Correctness Criteria
+      - Entity must have been used correctly by the agent — the value passed to the tool call must match what the user said.
+      - Numbers: "150" and "one hundred fifty" are equivalent
+      - Dates: "December 15th" and "Dec 15" are equivalent
+      - Names: Case-insensitive match. Phonetically equivalent spelling variants are acceptable on the **first mention** of a name (e.g., German Meyer/Meier/Maier/Mayer — all sound the same when spoken). Once the user has explicitly disambiguated the spelling (e.g., spelled it out letter by letter, or corrected a prior spelling), subsequent usage must match that clarified spelling exactly. Flag if:
+        - The value the agent used clearly differs (different sound, not just different spelling)
+        - The agent used a different spelling variant **after** the user has explicitly clarified the correct spelling
+      - Non-Latin scripts: A name or identifier rendered in its native writing system (e.g. Kanji, Hangul, Arabic) is equivalent to its romanized/phonetic form if they represent the same spoken entity. For example, "山田" and "Yamada" should be treated as matching.
+      - Phonetic letter spellings: In languages that do not use the Latin alphabet, Roman/Latin letters are spoken using language-specific phonetic equivalents. These are correct, not errors (e.g., in Korean "에이" for "A" or "비" for "B").
+      - Number surface variants: Many languages have systematic spoken-form differences that do not change the underlying value and must not be penalized:
+        - Reversed digit order in spoken form (e.g., German "vierundzwanzig" = four-and-twenty = 24)
+        - Grammatical gender/agreement variants (e.g., Spanish "veintiún" vs "veintiuna" for 21)
+        - Large-unit grouping differences (e.g., Japanese 万-system: "ichi-man go-sen" / 一万五千 = 15,000)
+
+      Important note: The user text will often feature things formatted like "one two three" instead of "123". Your goal is to evaluate semantic equivalence — these are considered equivalent if the value matches what the agent used.
+
+      ## Examples
+
+      The user speaks the id in turns 1–2 (with a correction) but nothing is rated at those turns — each **tool call** is rated on its own. Tool Call [1] is wrong and [2] is correct for the same entity. Tool Call [3] uses only a `record_id` that came from a tool response, so it is omitted entirely.
+
+      **Example Input (Conversation):**
+      Turn 1 - User: My employee id is E M one two three.
+      Turn 1 - [Assistant speaks]
+      Turn 2 - User: Actually, sorry — the id is E M four five six.
+      Turn 2 - [Assistant speaks]
+      Turn 4 - [Assistant speaks]
+      Turn 4 - Tool Call [1] (lookup_employee): {{'employee_id': 'EM455'}}
+      Turn 4 - Tool Response (lookup_employee): {{'employee_id': 'EM455', 'record_id': 'R-9012'}}
+      Turn 6 - [Assistant speaks]
+      Turn 6 - Tool Call [2] (get_record): {{'employee_id': 'EM456', 'record_id': 'R-9012'}}
+      Turn 6 - Tool Call [3] (send_summary): {{'record_id': 'R-9012'}}
+
+      **Example Response:**
+      [
+      {{
+          "tool_call_id": 1,
+          "tool_name": "lookup_employee",
+          "entities": [
+          {{
+              "type": "employee_id",
+              "value": "E M four five six (user corrected it in turn 2)",
+              "captured_value": "EM455",
+              "analysis": "The user corrected the id to EM456 in turn 2; this tool call used EM455 — one character off, incorrect.",
+              "correct": false,
+              "skipped": false
+          }}
+          ],
+          "summary": "Tool call used EM455; the user's (corrected) id is EM456 — incorrect."
+      }},
+      {{
+          "tool_call_id": 2,
+          "tool_name": "get_record",
+          "entities": [
+          {{
+              "type": "employee_id",
+              "value": "E M four five six",
+              "captured_value": "EM456",
+              "analysis": "Matches the user's corrected id. (record_id R-9012 came from the turn-4 tool response, not the user, so it is not rated.)",
+              "correct": true,
+              "skipped": false
+          }}
+          ],
+          "summary": "Tool call used EM456 — correct."
+      }}
+      ]
+
+      Note Tool Call [3] does not appear: its only argument (record_id) came from a tool response, not the user.
+
+      ## Response Format
+      Respond with a JSON array. Output **one entry per tool call that uses at least one user-provided entity**, keyed by the tool call's `[N]` id. Tool calls with no user-provided entities, and turns where the user only speaks, must NOT appear.
+      [
+      {{
+          "tool_call_id": <int: the [N] id of the tool call being rated>,
+          "tool_name": "<the tool's name, for readability>",
+          "entities": [
+          {{
+              "type": "<name|date|time|confirmation_code|flight_number|amount|address|phone|email|employee_id|etc...>",
+              "value": "<what the user actually said for this entity, accounting for any later correction>",
+              "captured_value": "<the value this tool call passed>",
+              "analysis": "<brief reason for the correct/incorrect judgement>",
+              "correct": <true|false>,
+              "skipped": <true|false: true only for entities in an interruption region>
+          }}
+          ],
+          "summary": "<1-2 sentence summary for this tool call>"
+      }}
+      ]

src/eva/__init__.py

@@ -11,4 +11,4 @@
 
 # Bump metrics_version when changes affect metric computation (metrics code,
 # judge prompts, pricing tables, postprocessor).
-metrics_version = "2.2.0"
+metrics_version = "2.5.0"