[STESOL-554] Have eval script use main search function and add sources to eval answers#109
[STESOL-554] Have eval script use main search function and add sources to eval answers#109brikin01 wants to merge 1 commit into
Conversation
| k: int | None = None, | ||
| ) -> list[dict[str, Any]]: | ||
| resolved_k = k or resources.default_k | ||
| candidate_depth = max(resolved_k * 20, 100) |
There was a problem hiding this comment.
@brikin01 why is this line added here ? Won't this reduce the number of relevant chunks ?
There was a problem hiding this comment.
This line makes the default runtime search use the same candidate-depth calculation as the original evaluate_retrieval.py (see L59 of the original file).
For a requested k=5, that means examining 100 dense and BM25 candidates. Without this line, hybrid_search() calculates its default from deduplication_candidate_count(k), which is 50, resulting in a candidate depth of 1,000.
Reducing the depth from 1,000 to 100 can theoretically exclude a relevant chunk ranked below 100 in the initial retrieval stage, but it does not reduce the requested number of final results (although as mentioned above, we were never actually using a depth of 1000 in this particular eval script). Also, I tested the retrieval evaluation with both values. 100 produced equivalent or slightly better results while requiring less search and reranking work, so 100 seems like the better tradeoff here.
Summary
arm_kb_search.search()path so eval matches production search behavior0.2.0.