Skip to content

[STESOL-554] Have eval script use main search function and add sources to eval answers #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brikin01 wants to merge 1 commit into
base: main
Choose a base branch
from
+17 −41
Open

[STESOL-554] Have eval script use main search function and add sources to eval answers #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions arm_kb_search/resources.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,13 +113,15 @@ def search(
k: int | None = None,
) -> list[dict[str, Any]]:
resolved_k = k or resources.default_k
candidate_depth = max(resolved_k * 20, 100)

@NeethuESim NeethuESim Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brikin01 why is this line added here ? Won't this reduce the number of relevant chunks ?

@brikin01 brikin01 Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line makes the default runtime search use the same candidate-depth calculation as the original evaluate_retrieval.py (see L59 of the original file).

For a requested k=5, that means examining 100 dense and BM25 candidates. Without this line, hybrid_search() calculates its default from deduplication_candidate_count(k), which is 50, resulting in a candidate depth of 1,000.

Reducing the depth from 1,000 to 100 can theoretically exclude a relevant chunk ranked below 100 in the initial retrieval stage, but it does not reduce the requested number of final results (although as mentioned above, we were never actually using a depth of 1000 in this particular eval script). Also, I tested the retrieval evaluation with both values. 100 produced equivalent or slightly better results while requiring less search and reranking work, so 100 seems like the better tradeoff here.

search_results = hybrid_search(
query,
resources.usearch_index,
resources.metadata,
resources.embedding_model,
resources.bm25_index,
k=deduplication_candidate_count(resolved_k),
candidate_depth=candidate_depth,
)
deduped = deduplicate_urls(search_results)[:resolved_k]
formatted = [
Expand Down
9 changes: 6 additions & 3 deletions embedding-generation/eval_questions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,8 @@
{
"question": "What does the Arm Performix Code Hotspots recipe show, and how should I interpret the top hotspots, flame graphs, function tables, CPU usage, and source line attribution?",
"expected_urls": [
"https://developer.arm.com/documentation/110163/latest/About-Arm-Performix/Basic-concepts/Code-Hotspots-recipe/Interpreting-and-using-the-results?lang=en"
"https://developer.arm.com/documentation/110163/latest/About-Arm-Performix/Basic-concepts/Code-Hotspots-recipe/Interpreting-and-using-the-results?lang=en",
"https://learn.arm.com/learning-paths/servers-and-cloud-computing/cpu_hotspot_performix/"
]
},
{
Expand Down Expand Up @@ -1154,7 +1155,8 @@
{
"question": "How do I get started with Arm Performance Studio?",
"expected_urls": [
"https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/ams/"
"https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/ams/",
"https://learn.arm.com/install-guides/ams/"
]
},
{
Expand Down Expand Up @@ -2162,7 +2164,8 @@
{
"question": "How do I tune MySQL?",
"expected_urls": [
"https://learn.arm.com/learning-paths/servers-and-cloud-computing/mysql_tune/"
"https://learn.arm.com/learning-paths/servers-and-cloud-computing/mysql_tune/",
"https://amperecomputing.com/tuning-guides/mysql-tuning-guide"
]
},
{
Expand Down
45 changes: 8 additions & 37 deletions embedding-generation/evaluate_retrieval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,63 +3,34 @@
from __future__ import annotations

import argparse
import os
import sys
from pathlib import Path

from sentence_transformers import SentenceTransformer


REPO_ROOT = Path(__file__).resolve().parents[1]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))

from arm_kb_search import ( # noqa: E402
build_bm25_index,
deduplicate_urls,
deduplication_candidate_count,
embedding_dimension,
hybrid_search,
load_metadata,
load_usearch_index,
load_search_resources,
search,
)
from arm_kb_search.evaluation import evaluate_retrieval, load_eval_rows, print_evaluation # noqa: E402


def sentence_transformer_cache_folder() -> str | None:
return os.getenv("SENTENCE_TRANSFORMERS_HOME") or None


def evaluate(index_path: Path, metadata_path: Path, eval_path: Path, model_name: str, top_k: int) -> int:
metadata = load_metadata(str(metadata_path))
if not metadata:
if not metadata_path.exists() or metadata_path.stat().st_size == 0:
print(f"Metadata not found or empty: {metadata_path}")
return 1

embedding_model = SentenceTransformer(
model_name,
cache_folder=sentence_transformer_cache_folder(),
local_files_only=True,
)
usearch_index = load_usearch_index(
str(index_path),
embedding_dimension(embedding_model),
resources = load_search_resources(
metadata_path=str(metadata_path),
usearch_index_path=str(index_path),
model_name=model_name,
)
bm25_index = build_bm25_index(metadata)
eval_rows = load_eval_rows(eval_path)

def retrieve_urls(question: str, top_k: int) -> list[str | None]:
raw_results = hybrid_search(
question,
usearch_index,
metadata,
embedding_model,
bm25_index,
k=deduplication_candidate_count(top_k),
candidate_depth=max(top_k * 20, 100),
)
results = deduplicate_urls(raw_results, max_chunks_per_url=1)[:top_k]
return [item["metadata"].get("url") for item in results]
return [item.get("url") for item in search(question, resources, k=top_k)]

result = evaluate_retrieval(eval_rows, retrieve_urls, top_k)
print_evaluation(result)
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "arm-kb-search"
version = "0.1.0"
version = "0.2.0"
requires-python = ">=3.10"
dependencies = [
"numpy",
Expand Down
Loading