Skip to content

[STESOL-554] Have eval script use main search function and add sources to eval answers#109

Open
brikin01 wants to merge 1 commit into
mainfrom
stesol-554-eval-script-improvement
Open

[STESOL-554] Have eval script use main search function and add sources to eval answers#109
brikin01 wants to merge 1 commit into
mainfrom
stesol-554-eval-script-improvement

Conversation

@brikin01

@brikin01 brikin01 commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Route retrieval evaluation through the shared arm_kb_search.search() path so eval matches production search behavior
  • Set explicit search candidate depth from requested result count, avoiding accidental coupling to deduplication candidate count
  • Add alternate expected URLs for eval questions where multiple relevant sources satisfy the query.
  • Bump package version to 0.2.0.

@brikin01 brikin01 requested a review from NeethuESim July 7, 2026 20:02
k: int | None = None,
) -> list[dict[str, Any]]:
resolved_k = k or resources.default_k
candidate_depth = max(resolved_k * 20, 100)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brikin01 why is this line added here ? Won't this reduce the number of relevant chunks ?

@brikin01 brikin01 Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line makes the default runtime search use the same candidate-depth calculation as the original evaluate_retrieval.py (see L59 of the original file).

For a requested k=5, that means examining 100 dense and BM25 candidates. Without this line, hybrid_search() calculates its default from deduplication_candidate_count(k), which is 50, resulting in a candidate depth of 1,000.

Reducing the depth from 1,000 to 100 can theoretically exclude a relevant chunk ranked below 100 in the initial retrieval stage, but it does not reduce the requested number of final results (although as mentioned above, we were never actually using a depth of 1000 in this particular eval script). Also, I tested the retrieval evaluation with both values. 100 produced equivalent or slightly better results while requiring less search and reranking work, so 100 seems like the better tradeoff here.

@NeethuESim NeethuESim left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants