Skip to content

Evaluation: Metric for prompt adherence #1046

Closed

Description

@AkhileshNegi

Is your feature request related to a problem?
Another metrci we want to add is whether an assistant's answer actually follows its prompt — right language, right tone, answering what it should and refusing what it shouldn't. Need a metric for this.

Describe the solution you'd like
Research and prototype a judge that checks:

  • Language and tone match the prompt.
  • Answers what it should, refuses what it shouldn't.
  • Uses the fallback response when it doesn't know, instead of inventing facts.
  • Holds up against prompt injection.

Try it on a real org's assistant, check cost per question. Output: a scored Adherence to prompt metric that feeds the weighted average.

Activity

  1. changed the title [-]Evaluation: Research adherence-to-prompt metric (instruction following)[/-] [+]Evaluation: Metric for prompt adherence[/+] on Jul 13, 2026
  2. moved this from To Do to In Progress in Kaapi-devon Jul 20, 2026
  3. moved this from In Progress to In Review in Kaapi-devon Jul 21, 2026
  4. moved this from In Review to Closed in Kaapi-devon Jul 24, 2026
  5. closed this as completedby moving to Closed in Kaapi-devon Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions