Add DeepSearchQA benchmark

[Yunnglin]

Summary

• Add a native DeepSearchQA AgentLoop benchmark adapter for the ModelScope google/deepsearchqa dataset.
• Implement Kaggle starter-style LLM judging, valid-sample precision/recall/F1 aggregation, and empty/invalid response health rates.
• Add generated benchmark docs plus DeepSearchQA MCP search/fetch usage guides in English and Chinese.

Validation

• PATH=/Users/yunlin/miniconda3/envs/eval/bin:$PATH make docs-pipeline BENCHMARK="deepsearchqa" FORCE=1
• PATH=/Users/yunlin/miniconda3/envs/eval/bin:$PATH make lint
• conda run -n eval python -m py_compile evalscope/benchmarks/deepsearchqa/deepsearchqa_adapter.py evalscope/benchmarks/deepsearchqa/utils.py tests/benchmark/test_agent.py
• conda run -n eval python -m pytest tests/benchmark/test_agent.py::TestAgentBenchmark::test_deepsearchqa -q -p no:warnings

Notes

DeepSearchQA does not bundle a search provider. The benchmark runs through EvalScope AgentLoop and documents how to attach search and fetch MCP servers for web-capable evaluations.

[gemini-code-assist]

Code Review

This pull request introduces the DeepSearchQA benchmark to EvalScope, including comprehensive English and Chinese documentation, metadata configurations, a dataset adapter, utility functions for LLM-based and rule-based evaluation, and end-to-end unit tests. The review feedback highlights several critical issues in the evaluation and parsing utilities that need to be addressed to prevent runtime crashes and incorrect scoring. Specifically, the rule-based fallback scoring and reference splitting functions are prone to type errors and false positives when handling list-type references. Additionally, the JSON extraction and LLM judge response parsing are overly strict and should be refactored to robustly handle common LLM output variations, such as stringified booleans and alternative markdown code block formats. Finally, a potential KeyError in the dataset adapter should be mitigated by using safe dictionary access.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[qoderai]

👋 Review Summary

Thanks for the thorough DeepSearchQA integration – the adapter, utilities, docs, and smoke test are clearly aligned with the benchmark’s goals and the public starter code.

🛡️ Key Risks & Issues

• Rule-based fallback scoring (evalscope/benchmarks/deepsearchqa/utils.py:81): The current rule_fallback_score logic effectively disables hallucination penalties for single-answer questions and can mark perfectly correct set answers as “excessive” when the prediction’s normalized string differs from the reference (e.g., reordering or minor formatting changes). Because RULE mode is used in the DeepSearchQA smoke test and is positioned as a conservative fallback, this may lead to unintuitive precision/recall/F1 values and a mismatch with the docs’ emphasis on penalizing both under-retrieval and excessive answers.
• Judge JSON parsing robustness (evalscope/benchmarks/deepsearchqa/utils.py:96): parse_judge_response and _extract_json_object enforce a very strict JSON shape and do not tolerate extra prose or minor schema variants (e.g., "true"/"false" strings or non-string items in Excessive Answers). Reasonable judge outputs that deviate slightly from the ideal schema will be classified as invalid_auto_rater_response, reducing the effective sample size for metrics and inflating health rates. This could make DeepSearchQA runs surprisingly brittle when swapping judge models or prompts.
• LLM judge concurrency (evalscope/benchmarks/deepsearchqa/deepsearchqa_adapter.py:110): llm_match_score calls self.llm_judge.judge(prompt) directly, without the per-adapter lock pattern used in other agent benchmarks (e.g., WideSearch). In multi-threaded sweeps, if the judge client shares mutable state, this might cause intermittent errors or mixed responses. Adding a simple lock or explicitly guaranteeing thread safety would help keep LLM judging predictable under parallel evaluation.
• Aggregation and health metrics (evalscope/benchmarks/deepsearchqa/utils.py:140): aggregate_official_scores correctly excludes empty/invalid responses from the mean metrics, but there’s no explicit signal of the valid-sample rate beyond the health AggScores. When invalid judge responses are common, users may see reasonable-looking precision/recall/F1 values without realizing they’re based on a small fraction of samples.
• Test coverage (tests/benchmark/test_agent.py:123): The new test_deepsearchqa asserts that answer_type is present in sample_metadata, which is a good end-to-end sanity check, but it doesn’t cover the scoring semantics or the judge JSON parsing. As a result, regressions in rule_fallback_score, aggregate_official_scores, or the LLM judge path could slip through without being detected.

🧪 Verification Advice

• Add focused unit tests around rule_fallback_score for both Single Answer and Set Answer, using synthetic examples where you know the intended precision/recall/F1 and “excessive” behavior, and confirm that RULE-mode scoring matches those expectations.
• Introduce tests for parse_judge_response and _extract_json_object that exercise fenced JSON, bare JSON with surrounding text, and null/missing-node cases, plus common schema variations (boolean-like strings, extra fields), to lock in the desired robustness and error classification.
• Consider a small adapter-level test (possibly with a mocked judge) that runs llm_match_score on a controlled example and verifies both the metric values and the health flags, including how aggregate_official_scores handles empty/invalid responses and reports their rates.
• If DeepSearchQA will be used in multi-threaded sweeps with LLM judging, run a parallelized local evaluation and watch for any flaky behavior or inconsistent judge outputs when a lock is added vs. removed.

💡 Thoughts & Suggestions

• The overall design—reusing AgentLoopAdapter, keeping answer_type hidden during inference, and aligning docs/meta with the benchmark paper—is strong and consistent with other agent benchmarks in EvalScope.
• Tightening the rule-based scoring and making the judge parsing path slightly more tolerant would give users more trust that DeepSearchQA metrics reflect true behavior rather than formatting quirks.
• Extending tests to cover scoring and judge semantics will make future changes safer, especially as new judge models or tool configurations are introduced.

---

🤖 Generated by Qoder • View workflow run

[qoderai]

👋 Review Summary

The DeepSearchQA integration looks thoughtfully aligned with the official benchmark: adapter wiring, metadata, aggregation, docs, and tests are all in good shape and consistent with other agent benchmarks.

🛡️ Key Risks & Issues

• LLM judge prompt construction (utils.py): build_grader_prompt uses GRADER_RATING_OUTPUT_EXAMPLE.format(...) with question, reference, and model response passed directly into the format template. If any of these strings contains { or }, Python’s str.format will treat them as placeholders and can raise KeyError/ValueError. In practice, DeepSearchQA questions and model outputs can legitimately include braces (JSON/code/templates), so this can cause data-dependent crashes or dropped scores when running with judge_strategy='llm'.
• Rule fallback semantics (non-blocking): The rule-based fallback approximates “excessive” answers for set-type questions as a single flag when correct < expected, even if the model didn’t actually hallucinate extra items. This is acceptable as a conservative heuristic and documented as not equivalent to official LLM judging, but it’s worth being aware that precision/recall balance under JudgeStrategy.RULE will not distinguish under-retrieval from hallucinations.

🧪 Verification Advice

• After hardening build_grader_prompt (e.g., escaping braces before .format or avoiding str.format for user/model text), re-run:
  • pytest tests/benchmark/test_deepsearchqa.py -q -p no:warnings
  • pytest tests/benchmark/test_agent.py::TestAgentBenchmark::test_deepsearchqa -q -p no:warnings
• If feasible, add or run a small stubbed-judge test that exercises llm_match_score with questions/answers/responses containing { and } to confirm the judge path no longer fails on such inputs.

💡 Thoughts & Suggestions

• Consider adding a focused unit or integration test for the llm_match_score path using a stub llm_judge so that future changes to the grading prompt or JSON parsing are caught early.
• Overall, this is a solid, well-documented benchmark addition; once the brace-handling issue in the grader prompt is addressed, the DeepSearchQA integration should be robust for both rule and LLM-judge modes.

---

🤖 Generated by Qoder • View workflow run

[qoderai]

👋 Review Summary

This PR adds a native DeepSearchQA agent benchmark (adapter + utils) with LLM-judge and rule-based scoring, plus docs and tests. The overall structure fits well into EvalScope’s AgentLoop and benchmark registry, and the tests cover key helper logic and a basic end-to-end smoke run.

🛡️ Key Risks & Issues

• Fallback rule scoring for set-answer questions currently treats any partial coverage as a false positive by setting excessive = int(expected > 0 and correct < expected) and passing that into _score_from_counts as false_positives. This can penalize models that miss some answers but do not hallucinate extras, and it diverges from the LLM-judge path where the count of Excessive Answers is used as the false-positive term. Because aggregate_official_scores mixes scores across judge strategies, this mismatch may make RULE-based smoke tests and LLM-based runs less directly comparable.
• Judge JSON parsing is intentionally strict: only a bare JSON object or a single ```json fenced block is accepted, and any surrounding text or minor format variation leads to invalid_auto_rater_response. While this matches the current tests and official starter behavior, it is brittle in practice and could yield high invalid rates if a judge model deviates slightly from the expected format.

🧪 Verification Advice

• For RULE-based runs, exercise set-answer questions where the model returns partial but non-hallucinated answer sets and check whether precision/recall/F1 reflect purely recall-side errors or also introduce false positives. Comparing the metrics against a small hand-computed sample can help confirm the intended semantics.
• For LLM-judge runs, try a judge model that tends to add prologues/epilogues or non-standard fences and monitor invalid_auto_rater_response and empty_auto_rater_response rates. If these spike, it may be worth tightening the judge prompt or making the parser slightly more tolerant.
• Run the provided DeepSearchQA tests along with the main CI-lite suite (pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings) on a fresh environment to ensure the new benchmark integrates cleanly across the broader pipeline.

💡 Thoughts & Suggestions

• The adapter and utilities are thoughtfully implemented: they respect the AgentLoopAdapter contract, use metadata to carry answer_type and question, provide a clear max-steps finalization message, and expose detailed correctness and pipeline-health metadata. Once the rule fallback semantics for set answers and the judge JSON parsing strictness are aligned with the desired behavior, this benchmark should be a solid addition for deep research agent evaluation.

---

🤖 Generated by Qoder • View workflow run

[qoderai]

👋 Review Summary

Thanks for adding the DeepSearchQA benchmark and wiring it cleanly into the AgentLoop flow, scoring helpers, docs, and tests. The adapter metadata, RULE fallback scoring, and aggregation logic align well with the described DeepSearchQA protocol.

🛡️ Key Risks & Issues

• LLM judge calls in DeepSearchQAAdapter.llm_match_score are serialized via a single adapter-level threading.Lock. This protects shared judge state but can substantially limit concurrency for DeepSearchQA evaluations, turning the judge into a bottleneck in multi-threaded runs.

🧪 Verification Advice

• For large-scale DeepSearchQA runs that use LLM judging, consider profiling evaluation throughput with and without the adapter-level lock (or with a narrower synchronization strategy) to ensure the judge path doesn’t dominate wall-clock time.

💡 Thoughts & Suggestions

• If the underlying llm_judge implementation is already thread-safe or isolated per adapter instance, we could remove or narrow the global lock to allow concurrent judge requests across samples.
• The DeepSearchQA-specific tests provide good coverage of rule fallback and aggregation; adding focused tests for empty judge responses, malformed Answer Correctness payloads, and the LLM judge path would further increase confidence in scoring correctness for real-world DeepSearchQA runs.

---

🤖 Generated by Qoder • View workflow run

drop