diff --git a/docs/en/benchmarks/deepsearchqa.md b/docs/en/benchmarks/deepsearchqa.md new file mode 100644 index 000000000..4b0b1dda9 --- /dev/null +++ b/docs/en/benchmarks/deepsearchqa.md @@ -0,0 +1,101 @@ +# DeepSearchQA + +## Overview + +DeepSearchQA is a Google DeepMind benchmark for evaluating deep research agents on difficult multi-step information-seeking tasks across the open web. It contains 900 prompts spanning 17 domains and is designed to measure exhaustive answer-set generation rather than single-answer retrieval alone. + +## Task Description + +- **Task Type**: Search-agent factual question answering +- **Input**: A natural-language research question +- **Output**: A single answer or complete answer set, depending on the question +- **Grading**: LLM-as-judge semantic matching against the gold answer and answer type + +## Key Features + +- Tests systematic collation of fragmented information from multiple sources +- Requires entity resolution and de-duplication for set-answer tasks +- Penalizes both under-retrieval and excessive/hallucinated answers +- Uses `problem_category` for analysis metadata; `answer_type` is withheld from the model during inference +- Compatible with EvalScope agent configurations for native or external web-capable agents + +## Agent Tool Configuration + +DeepSearchQA does not hard-code a search provider. By default it runs through EvalScope native AgentLoop without external search tools. To evaluate a web-capable agent, set `TaskConfig.agent_config` and attach the search/fetch tools that should be available to the model. If `NativeAgentConfig.max_steps` is omitted, DeepSearchQA uses its benchmark-level AgentLoop default of 30 steps. + +See the [DeepSearchQA usage guide](https://evalscope.readthedocs.io/en/latest/third_party/deepsearchqa.html) for +runtime examples, MCP search/fetch configuration, and evaluation notes. + +## Evaluation Notes + +- EvalScope loads the ModelScope dataset `google/deepsearchqa` from the `eval` split. +- LLM judge is enabled by default. Official starter code uses Gemini 2.5 Flash with the DeepSearchQA judge prompt, but EvalScope can use any configured judge model for local runs. +- The primary metric is `f1_score`; `precision`, `recall`, and empty/invalid response rates are also reported. +- `JudgeStrategy.RULE` provides a conservative exact/substring fallback for smoke tests and is not equivalent to official LLM judging. + +## Properties + +| Property | Value | +|----------|-------| +| **Benchmark Name** | `deepsearchqa` | +| **Dataset ID** | [google/deepsearchqa](https://modelscope.cn/datasets/google/deepsearchqa/summary) | +| **Paper** | [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) | +| **Tags** | `Agent`, `Knowledge`, `QA`, `Retrieval` | +| **Metrics** | `f1_score`, `precision`, `recall` | +| **Default Shots** | 0-shot | +| **Evaluation Split** | `eval` | + + +## Data Statistics + +| Metric | Value | +|--------|-------| +| Total Samples | 900 | +| Prompt Length (Mean) | 295.54 chars | +| Prompt Length (Min/Max) | 49 / 1007 chars | + +## Sample Example + +*Sample example not available.* + +## Prompt Template + +**Prompt Template:** +```text +{question} +``` + +## Usage + +### Using CLI + +```bash +evalscope eval \ + --model YOUR_MODEL \ + --api-url OPENAI_API_COMPAT_URL \ + --api-key EMPTY_TOKEN \ + --datasets deepsearchqa \ + --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30}' \ + --limit 10 # Remove this line for formal evaluation +``` + +### Using Python + +```python +from evalscope import TaskConfig, run_task +from evalscope.api.agent import NativeAgentConfig + +task_cfg = TaskConfig( + model='YOUR_MODEL', + api_url='OPENAI_API_COMPAT_URL', + api_key='EMPTY_TOKEN', + datasets=['deepsearchqa'], + agent_config=NativeAgentConfig( + strategy='function_calling', + max_steps=30, + ), + limit=10, # Remove this line for formal evaluation +) + +run_task(task_cfg=task_cfg) +``` diff --git a/docs/en/get_started/supported_dataset/agent.md b/docs/en/get_started/supported_dataset/agent.md index 73c83241f..cb92fbbc4 100644 --- a/docs/en/get_started/supported_dataset/agent.md +++ b/docs/en/get_started/supported_dataset/agent.md @@ -9,6 +9,7 @@ Below is the list of supported AGENT benchmarks. Click on a benchmark name for d | `browsecomp` | [BrowseComp](../../benchmarks/browsecomp.md) | `Agent`, `Knowledge`, `QA` | | `claw_eval` | [Claw-Eval](../../benchmarks/claw_eval.md) | `Agent`, `MultiModal`, `MultiTurn` | | `deep_swe` | [DeepSWE](../../benchmarks/deep_swe.md) | `Agent`, `Coding`, `MultiTurn` | +| `deepsearchqa` | [DeepSearchQA](../../benchmarks/deepsearchqa.md) | `Agent`, `Knowledge`, `QA`, `Retrieval` | | `gaia` | [GAIA](../../benchmarks/gaia.md) | `Agent`, `MultiTurn`, `Reasoning` | | `gdpval` | [GDPval](../../benchmarks/gdpval.md) | `Agent`, `Knowledge`, `MultiTurn` | | `general_fc` | [General-FunctionCalling](../../benchmarks/general_fc.md) | `Agent`, `Custom`, `FunctionCalling` | @@ -41,6 +42,7 @@ Below is the list of supported AGENT benchmarks. Click on a benchmark name for d ../../benchmarks/browsecomp.md ../../benchmarks/claw_eval.md ../../benchmarks/deep_swe.md +../../benchmarks/deepsearchqa.md ../../benchmarks/gaia.md ../../benchmarks/gdpval.md ../../benchmarks/general_fc.md diff --git a/docs/en/third_party/deepsearchqa.md b/docs/en/third_party/deepsearchqa.md new file mode 100644 index 000000000..b0412aeef --- /dev/null +++ b/docs/en/third_party/deepsearchqa.md @@ -0,0 +1,133 @@ +# DeepSearchQA + +## Introduction + +[DeepSearchQA](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) evaluates +deep research agents on difficult web information-seeking questions. The ModelScope dataset +[`google/deepsearchqa`](https://modelscope.cn/datasets/google/deepsearchqa/summary) contains 900 examples in the +`eval` split. Each row contains a question, category, gold answer, and answer type (`Single Answer` or `Set Answer`). + +EvalScope implements the ModelScope dataset adapter and the public Kaggle starter-code judging protocol. The official +starter uses Gemini 2.5 Flash with Google Search for answer generation and a Gemini judge. EvalScope does not hard-code +either provider: you can run any OpenAI-compatible model and attach search/fetch tools through AgentLoop MCP servers. + +## Installation + +Install MCP support if the run uses MCP servers: + +```bash +pip install 'evalscope[mcp]' +``` + +`evalscope[mcp]` includes the official `mcp` Python SDK and `mcp-server-fetch`. Fetch is useful for reading known URLs, +but it is not a search engine. A formal DeepSearchQA agent run should also provide a real search MCP server. + +## Search MCP Requirements + +DeepSearchQA questions usually require discovering multiple web pages, resolving aliases, and checking several sources. +The agent should have at least two capabilities: + +- `search` or `web_search`: accepts a query string and returns ranked results with title, snippet, and URL. +- `fetch` or `fetch_url`: accepts a URL and returns readable page text. + +`mcp_server_fetch` only provides the second capability. If it is the only MCP server configured, the model can open URLs +that it already knows, but it cannot reliably discover new sources. + +## Native AgentLoop Example + +The example below uses Fetch MCP plus a remote search MCP endpoint. Replace the search endpoint with your own +ModelScope MCP, Brave/Tavily-backed MCP, or self-hosted MCP server. + +```python +import os +import sys + +from evalscope import TaskConfig, run_task +from evalscope.api.agent import NativeAgentConfig +from evalscope.api.agent.mcp import MCPServerConfigHTTP, MCPServerConfigStdio + +task_config = TaskConfig( + model='YOUR_AGENT_MODEL', + api_url='OPENAI_COMPATIBLE_URL', + api_key=os.getenv('MODEL_API_KEY'), + eval_type='openai_api', + datasets=['deepsearchqa'], + eval_batch_size=1, + limit=10, + judge_strategy='llm', + judge_model_args={ + 'model_id': 'YOUR_JUDGE_MODEL', + 'api_url': 'OPENAI_COMPATIBLE_JUDGE_URL', + 'api_key': os.getenv('JUDGE_API_KEY'), + 'generation_config': {'temperature': 0.0}, + }, + agent_config=NativeAgentConfig( + strategy='function_calling', + max_steps=30, + mcp_servers=[ + MCPServerConfigStdio( + name='fetch', + command=sys.executable, + args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'], + ), + MCPServerConfigHTTP( + name='search', + url='https://mcp.api-inference.modelscope.net//mcp', + headers={'Authorization': 'Bearer '}, + ), + ], + kwargs={ + 'system_prompt': ( + 'Use the available search and fetch tools to research the question. ' + 'Cross-check important facts before answering. When finished, call submit(answer=...) with only the ' + 'final answer or complete answer set.' + ), + }, + ), +) + +run_task(task_config) +``` + +## CLI Example + +When configuring MCP servers through CLI JSON, include each server's `type` field. + +```bash +evalscope eval \ + --model YOUR_AGENT_MODEL \ + --api-url OPENAI_COMPATIBLE_URL \ + --api-key "$MODEL_API_KEY" \ + --datasets deepsearchqa \ + --limit 10 \ + --eval-batch-size 1 \ + --judge-strategy llm \ + --judge-model-args '{"model_id":"YOUR_JUDGE_MODEL","api_url":"OPENAI_COMPATIBLE_JUDGE_URL","api_key":"'"$JUDGE_API_KEY"'","generation_config":{"temperature":0.0}}' \ + --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30,"mcp_servers":[{"type":"stdio","name":"fetch","command":"python","args":["-m","mcp_server_fetch","--ignore-robots-txt"]},{"type":"http","name":"search","url":"https://mcp.api-inference.modelscope.net//mcp","headers":{"Authorization":"Bearer "}}],"kwargs":{"system_prompt":"Use search and fetch tools to research the question. Cross-check important facts before answering. When finished, call submit(answer=...) with only the final answer."}}' +``` + +## Scoring + +The primary metric is `f1_score`; `precision` and `recall` are also reported. EvalScope additionally emits pipeline +health rates from the official starter logic: + +- `empty_model_response` +- `empty_auto_rater_response` +- `invalid_auto_rater_response` + +Empty model responses and empty/invalid judge responses are excluded from the valid-sample mean precision/recall/F1 +denominator and reported separately as rates. Per-sample reviews keep the judge details, including which expected +answers were found and which excessive answers were detected. + +## Notes + +- `function_calling` expects the model to call `submit(answer=...)` when it is done. +- DeepSearchQA adds a max-step finalization turn: if the loop reaches `max_steps` with an empty completion, the model is + asked to answer from the information already gathered. +- For comparable runs, keep the judge temperature at `0.0` and use the same judge model across all systems. + +## Resources + +- [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) +- [ModelScope dataset](https://modelscope.cn/datasets/google/deepsearchqa/summary) +- [Agent MCP guide](../user_guides/agent/native.md#mcp-server-tools) diff --git a/docs/en/third_party/index.md b/docs/en/third_party/index.md index 129065414..506ab30f7 100644 --- a/docs/en/third_party/index.md +++ b/docs/en/third_party/index.md @@ -12,6 +12,7 @@ skillsbench.md toolathlon.md gaia.md wide_search.md +deepsearchqa.md swe_bench.md swe_bench_pro.md tau_bench.md diff --git a/docs/en/user_guides/agent/native.md b/docs/en/user_guides/agent/native.md index 88848c9cd..dd9c92416 100644 --- a/docs/en/user_guides/agent/native.md +++ b/docs/en/user_guides/agent/native.md @@ -99,6 +99,9 @@ pip install evalscope[mcp] `evalscope[mcp]` bundles the official `mcp` Python SDK plus `mcp-server-fetch` (universal HTTP fetching, no API key). Other MCP servers can be installed on demand or run via `uvx` / `npx`. +For search-agent benchmark examples that combine search MCP servers with fetch/browser-style tools, see the +[DeepSearchQA](../../third_party/deepsearchqa.md) and [WideSearch](../../third_party/wide_search.md) guides. + ### Quick start Attach HTTP fetching and a remote MCP tool to a GAIA evaluation: diff --git a/docs/zh/benchmarks/deepsearchqa.md b/docs/zh/benchmarks/deepsearchqa.md new file mode 100644 index 000000000..9c12a3910 --- /dev/null +++ b/docs/zh/benchmarks/deepsearchqa.md @@ -0,0 +1,100 @@ +# DeepSearchQA + +## 概述 + +DeepSearchQA 是 Google DeepMind 推出的一项基准测试，用于评估深度研究智能体在开放网络上执行复杂多步骤信息检索任务的能力。该基准包含 900 个提示，涵盖 17 个领域，旨在衡量模型生成完整答案集合的能力，而不仅仅是单个答案的检索。 + +## 任务描述 + +- **任务类型**：搜索智能体事实型问答 +- **输入**：自然语言形式的研究问题 +- **输出**：单个答案或完整的答案集合（取决于问题） +- **评分方式**：使用大语言模型（LLM）作为评判器，通过语义匹配对比标准答案及答案类型 + +## 核心特性 + +- 测试从多个来源系统性整合碎片化信息的能力 +- 对于答案为集合的问题，要求进行实体消歧和去重 +- 同时惩罚检索不足和过度/幻觉生成的答案 +- 使用 `problem_category` 提供分析元数据；推理过程中对模型隐藏 `answer_type` +- 兼容 EvalScope 智能体配置，支持原生或具备外部网络能力的智能体 + +## 智能体工具配置 + +DeepSearchQA 不硬编码指定搜索引擎。默认情况下，它通过 EvalScope 原生 AgentLoop 运行，不使用外部搜索工具。若要评估具备网络能力的智能体，请设置 `TaskConfig.agent_config` 并附加模型可用的搜索/获取工具。如果未指定 `NativeAgentConfig.max_steps`，DeepSearchQA 将使用其基准级 AgentLoop 默认值 30 步。 + +有关运行示例、MCP 搜索/获取配置和评估说明，请参阅 [DeepSearchQA 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/third_party/deepsearchqa.html)。 + +## 评估说明 + +- EvalScope 从 `eval` 切分中加载 ModelScope 数据集 `google/deepsearchqa`。 +- 默认启用 LLM 评判器。官方起始代码使用 Gemini 2.5 Flash 模型配合 DeepSearchQA 评判提示，但 EvalScope 在本地运行时可使用任意已配置的评判模型。 +- 主要指标为 `f1_score`；同时报告 `precision`、`recall` 以及空/无效响应率。 +- `JudgeStrategy.RULE` 为冒烟测试提供保守的精确匹配/子串匹配回退机制，但不等同于官方的 LLM 评判。 + +## 属性 + +| 属性 | 值 | +|----------|-------| +| **基准测试名称** | `deepsearchqa` | +| **数据集ID** | [google/deepsearchqa](https://modelscope.cn/datasets/google/deepsearchqa/summary) | +| **论文** | [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) | +| **标签** | `Agent`, `Knowledge`, `QA`, `Retrieval` | +| **指标** | `f1_score`, `precision`, `recall` | +| **默认示例数** | 0-shot | +| **评估切分** | `eval` | + + +## 数据统计 + +| 指标 | 值 | +|--------|-------| +| 总样本数 | 900 | +| 提示词长度（平均） | 295.54 字符 | +| 提示词长度（最小/最大） | 49 / 1007 字符 | + +## 样例示例 + +*样例示例不可用。* + +## 提示模板 + +**提示模板：** +```text +{question} +``` + +## 使用方法 + +### 使用 CLI + +```bash +evalscope eval \ + --model YOUR_MODEL \ + --api-url OPENAI_API_COMPAT_URL \ + --api-key EMPTY_TOKEN \ + --datasets deepsearchqa \ + --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30}' \ + --limit 10 # 正式评估时请删除此行 +``` + +### 使用 Python + +```python +from evalscope import TaskConfig, run_task +from evalscope.api.agent import NativeAgentConfig + +task_cfg = TaskConfig( + model='YOUR_MODEL', + api_url='OPENAI_API_COMPAT_URL', + api_key='EMPTY_TOKEN', + datasets=['deepsearchqa'], + agent_config=NativeAgentConfig( + strategy='function_calling', + max_steps=30, + ), + limit=10, # 正式评估时请删除此行 +) + +run_task(task_cfg=task_cfg) +``` diff --git a/docs/zh/get_started/supported_dataset/agent.md b/docs/zh/get_started/supported_dataset/agent.md index a56680a68..70c5c4dd1 100644 --- a/docs/zh/get_started/supported_dataset/agent.md +++ b/docs/zh/get_started/supported_dataset/agent.md @@ -9,6 +9,7 @@ | `browsecomp` | [BrowseComp](../../benchmarks/browsecomp.md) | `Agent`, `Knowledge`, `QA` | | `claw_eval` | [Claw-Eval](../../benchmarks/claw_eval.md) | `Agent`, `MultiModal`, `MultiTurn` | | `deep_swe` | [DeepSWE](../../benchmarks/deep_swe.md) | `Agent`, `Coding`, `MultiTurn` | +| `deepsearchqa` | [DeepSearchQA](../../benchmarks/deepsearchqa.md) | `Agent`, `Knowledge`, `QA`, `Retrieval` | | `gaia` | [GAIA](../../benchmarks/gaia.md) | `Agent`, `MultiTurn`, `Reasoning` | | `gdpval` | [GDPval](../../benchmarks/gdpval.md) | `Agent`, `Knowledge`, `MultiTurn` | | `general_fc` | [General-FunctionCalling](../../benchmarks/general_fc.md) | `Agent`, `Custom`, `FunctionCalling` | @@ -41,6 +42,7 @@ ../../benchmarks/browsecomp.md ../../benchmarks/claw_eval.md ../../benchmarks/deep_swe.md +../../benchmarks/deepsearchqa.md ../../benchmarks/gaia.md ../../benchmarks/gdpval.md ../../benchmarks/general_fc.md diff --git a/docs/zh/third_party/deepsearchqa.md b/docs/zh/third_party/deepsearchqa.md new file mode 100644 index 000000000..70ab615e9 --- /dev/null +++ b/docs/zh/third_party/deepsearchqa.md @@ -0,0 +1,128 @@ +# DeepSearchQA + +## 简介 + +[DeepSearchQA](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) 用于评估 +Deep Research Agent 处理复杂联网问答的能力。ModelScope 数据集 +[`google/deepsearchqa`](https://modelscope.cn/datasets/google/deepsearchqa/summary) 在 `eval` split 中包含 900 道题。 +每条样本包含问题、类别、标准答案和答案类型（`Single Answer` 或 `Set Answer`）。 + +EvalScope 接入 ModelScope 数据集，并对齐公开 Kaggle starter code 的判分协议。官方 starter 使用 Gemini 2.5 Flash +和 Google Search 生成答案，再用 Gemini 评分；EvalScope 不绑定模型或搜索服务，可以使用任意 OpenAI-compatible 模型， +并通过 AgentLoop MCP server 接入搜索和网页抓取工具。 + +## 安装 + +如果运行时需要 MCP server，安装 MCP 依赖： + +```bash +pip install 'evalscope[mcp]' +``` + +`evalscope[mcp]` 自带官方 `mcp` Python SDK 和 `mcp-server-fetch`。Fetch 适合读取已知 URL，但不是搜索引擎。 +正式 DeepSearchQA agent 评测还应配置真实搜索 MCP server。 + +## 搜索 MCP 要求 + +DeepSearchQA 通常需要发现多个网页、解析别名，并交叉验证多个来源。建议至少提供两类工具： + +- `search` 或 `web_search`：输入 query，返回带 title、snippet、URL 的排序结果。 +- `fetch` 或 `fetch_url`：输入 URL，返回可读网页正文。 + +`mcp_server_fetch` 只提供第二类能力。如果只配置 fetch，模型只能打开已经知道的 URL，无法可靠发现新来源。 + +## Native AgentLoop 示例 + +下面示例同时配置 Fetch MCP 和一个远端搜索 MCP endpoint。搜索 endpoint 可替换为 ModelScope MCP、Brave/Tavily +封装 MCP，或自建 MCP server。 + +```python +import os +import sys + +from evalscope import TaskConfig, run_task +from evalscope.api.agent import NativeAgentConfig +from evalscope.api.agent.mcp import MCPServerConfigHTTP, MCPServerConfigStdio + +task_config = TaskConfig( + model='YOUR_AGENT_MODEL', + api_url='OPENAI_COMPATIBLE_URL', + api_key=os.getenv('MODEL_API_KEY'), + eval_type='openai_api', + datasets=['deepsearchqa'], + eval_batch_size=1, + limit=10, + judge_strategy='llm', + judge_model_args={ + 'model_id': 'YOUR_JUDGE_MODEL', + 'api_url': 'OPENAI_COMPATIBLE_JUDGE_URL', + 'api_key': os.getenv('JUDGE_API_KEY'), + 'generation_config': {'temperature': 0.0}, + }, + agent_config=NativeAgentConfig( + strategy='function_calling', + max_steps=30, + mcp_servers=[ + MCPServerConfigStdio( + name='fetch', + command=sys.executable, + args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'], + ), + MCPServerConfigHTTP( + name='search', + url='https://mcp.api-inference.modelscope.net//mcp', + headers={'Authorization': 'Bearer '}, + ), + ], + kwargs={ + 'system_prompt': ( + 'Use the available search and fetch tools to research the question. ' + 'Cross-check important facts before answering. When finished, call submit(answer=...) with only the ' + 'final answer or complete answer set.' + ), + }, + ), +) + +run_task(task_config) +``` + +## CLI 示例 + +通过 CLI JSON 配置 MCP server 时，每个 server 都要写 `type` 字段。 + +```bash +evalscope eval \ + --model YOUR_AGENT_MODEL \ + --api-url OPENAI_COMPATIBLE_URL \ + --api-key "$MODEL_API_KEY" \ + --datasets deepsearchqa \ + --limit 10 \ + --eval-batch-size 1 \ + --judge-strategy llm \ + --judge-model-args '{"model_id":"YOUR_JUDGE_MODEL","api_url":"OPENAI_COMPATIBLE_JUDGE_URL","api_key":"'"$JUDGE_API_KEY"'","generation_config":{"temperature":0.0}}' \ + --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30,"mcp_servers":[{"type":"stdio","name":"fetch","command":"python","args":["-m","mcp_server_fetch","--ignore-robots-txt"]},{"type":"http","name":"search","url":"https://mcp.api-inference.modelscope.net//mcp","headers":{"Authorization":"Bearer "}}],"kwargs":{"system_prompt":"Use search and fetch tools to research the question. Cross-check important facts before answering. When finished, call submit(answer=...) with only the final answer."}}' +``` + +## 评分 + +主指标是 `f1_score`，同时报告 `precision` 和 `recall`。EvalScope 还会按照官方 starter 逻辑输出评测流程健康度指标： + +- `empty_model_response` +- `empty_auto_rater_response` +- `invalid_auto_rater_response` + +空模型回答、空 judge 回复和非法 judge 回复不会进入有效样本 mean precision/recall/F1 的分母，而是单独以 rate 报告。 +每条样本的 review 仍会保留 judge 细节，包括哪些标准答案被命中，以及检测到了哪些额外答案。 + +## 注意事项 + +- `function_calling` 模式要求模型完成后调用 `submit(answer=...)`。 +- DeepSearchQA 增加了 max-step finalization：如果到达 `max_steps` 时 completion 为空，会要求模型基于已收集信息输出最终答案。 +- 为了比较不同模型，建议固定同一个 judge model，并设置 judge temperature 为 `0.0`。 + +## 相关资源 + +- [论文](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) +- [ModelScope 数据集](https://modelscope.cn/datasets/google/deepsearchqa/summary) +- [Agent MCP 配置指南](../user_guides/agent/native.md#mcp-server-tools) diff --git a/docs/zh/third_party/index.md b/docs/zh/third_party/index.md index 76e4051fb..98b713b1a 100644 --- a/docs/zh/third_party/index.md +++ b/docs/zh/third_party/index.md @@ -12,6 +12,7 @@ skillsbench.md toolathlon.md gaia.md wide_search.md +deepsearchqa.md swe_bench.md swe_bench_pro.md tau_bench.md diff --git a/docs/zh/user_guides/agent/native.md b/docs/zh/user_guides/agent/native.md index c6bc15679..5e598d9ea 100644 --- a/docs/zh/user_guides/agent/native.md +++ b/docs/zh/user_guides/agent/native.md @@ -98,6 +98,9 @@ pip install evalscope[mcp] `evalscope[mcp]` 自带 `mcp` Python SDK 和 `mcp-server-fetch`（通用 HTTP 抓取，无需 API key）。其他 MCP server 按需安装或通过 `uvx` / `npx` 即调即用。 +如果要看搜索类 benchmark 如何组合搜索 MCP 与 fetch/browser 类工具，请参考 +[DeepSearchQA](../../third_party/deepsearchqa.md) 和 [WideSearch](../../third_party/wide_search.md) 使用指南。 + ### 快速开始 给 GAIA 评测挂上 HTTP 抓取和远端 MCP 工具： diff --git a/evalscope/benchmarks/_meta/deepsearchqa.json b/evalscope/benchmarks/_meta/deepsearchqa.json new file mode 100644 index 000000000..f13472ea3 --- /dev/null +++ b/evalscope/benchmarks/_meta/deepsearchqa.json @@ -0,0 +1,67 @@ +{ + "meta": { + "pretty_name": "DeepSearchQA", + "dataset_id": "google/deepsearchqa", + "paper_url": "https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf", + "tags": [ + "Agent", + "Knowledge", + "QA", + "Retrieval" + ], + "metrics": [ + "f1_score", + "precision", + "recall" + ], + "few_shot_num": 0, + "eval_split": "eval", + "train_split": "", + "subset_list": [ + "default" + ], + "description": "## Overview\n\nDeepSearchQA is a Google DeepMind benchmark for evaluating deep research agents on difficult multi-step information-seeking tasks across the open web. It contains 900 prompts spanning 17 domains and is designed to measure exhaustive answer-set generation rather than single-answer retrieval alone.\n\n## Task Description\n\n- **Task Type**: Search-agent factual question answering\n- **Input**: A natural-language research question\n- **Output**: A single answer or complete answer set, depending on the question\n- **Grading**: LLM-as-judge semantic matching against the gold answer and answer type\n\n## Key Features\n\n- Tests systematic collation of fragmented information from multiple sources\n- Requires entity resolution and de-duplication for set-answer tasks\n- Penalizes both under-retrieval and excessive/hallucinated answers\n- Uses `problem_category` for analysis metadata; `answer_type` is withheld from the model during inference\n- Compatible with EvalScope agent configurations for native or external web-capable agents\n\n## Agent Tool Configuration\n\nDeepSearchQA does not hard-code a search provider. By default it runs through EvalScope native AgentLoop without external search tools. To evaluate a web-capable agent, set `TaskConfig.agent_config` and attach the search/fetch tools that should be available to the model. If `NativeAgentConfig.max_steps` is omitted, DeepSearchQA uses its benchmark-level AgentLoop default of 30 steps.\n\nSee the [DeepSearchQA usage guide](https://evalscope.readthedocs.io/en/latest/third_party/deepsearchqa.html) for\nruntime examples, MCP search/fetch configuration, and evaluation notes.\n\n## Evaluation Notes\n\n- EvalScope loads the ModelScope dataset `google/deepsearchqa` from the `eval` split.\n- LLM judge is enabled by default. Official starter code uses Gemini 2.5 Flash with the DeepSearchQA judge prompt, but EvalScope can use any configured judge model for local runs.\n- The primary metric is `f1_score`; `precision`, `recall`, and empty/invalid response rates are also reported.\n- `JudgeStrategy.RULE` provides a conservative exact/substring fallback for smoke tests and is not equivalent to official LLM judging.", + "prompt_template": "{question}", + "system_prompt": "", + "few_shot_prompt_template": "", + "aggregation": "mean", + "extra_params": {}, + "sandbox_config": {}, + "category": "agent", + "agent_config": { + "strategy": "function_calling", + "max_steps": 30 + } + }, + "statistics": { + "total_samples": 900, + "subset_stats": [ + { + "name": "default", + "sample_count": 900, + "prompt_length_mean": 295.54, + "prompt_length_min": 49, + "prompt_length_max": 1007, + "prompt_length_std": 141.21, + "target_length_mean": 67.25 + } + ], + "prompt_length": { + "mean": 295.54, + "min": 49, + "max": 1007, + "std": 141.21 + }, + "target_length_mean": 67.25, + "computed_at": "2026-07-21T16:49:54.577389" + }, + "sample_example": {}, + "readme": { + "en": "# DeepSearchQA\n\n## Overview\n\nDeepSearchQA is a Google DeepMind benchmark for evaluating deep research agents on difficult multi-step information-seeking tasks across the open web. It contains 900 prompts spanning 17 domains and is designed to measure exhaustive answer-set generation rather than single-answer retrieval alone.\n\n## Task Description\n\n- **Task Type**: Search-agent factual question answering\n- **Input**: A natural-language research question\n- **Output**: A single answer or complete answer set, depending on the question\n- **Grading**: LLM-as-judge semantic matching against the gold answer and answer type\n\n## Key Features\n\n- Tests systematic collation of fragmented information from multiple sources\n- Requires entity resolution and de-duplication for set-answer tasks\n- Penalizes both under-retrieval and excessive/hallucinated answers\n- Uses `problem_category` for analysis metadata; `answer_type` is withheld from the model during inference\n- Compatible with EvalScope agent configurations for native or external web-capable agents\n\n## Agent Tool Configuration\n\nDeepSearchQA does not hard-code a search provider. By default it runs through EvalScope native AgentLoop without external search tools. To evaluate a web-capable agent, set `TaskConfig.agent_config` and attach the search/fetch tools that should be available to the model. If `NativeAgentConfig.max_steps` is omitted, DeepSearchQA uses its benchmark-level AgentLoop default of 30 steps.\n\nSee the [DeepSearchQA usage guide](https://evalscope.readthedocs.io/en/latest/third_party/deepsearchqa.html) for\nruntime examples, MCP search/fetch configuration, and evaluation notes.\n\n## Evaluation Notes\n\n- EvalScope loads the ModelScope dataset `google/deepsearchqa` from the `eval` split.\n- LLM judge is enabled by default. Official starter code uses Gemini 2.5 Flash with the DeepSearchQA judge prompt, but EvalScope can use any configured judge model for local runs.\n- The primary metric is `f1_score`; `precision`, `recall`, and empty/invalid response rates are also reported.\n- `JudgeStrategy.RULE` provides a conservative exact/substring fallback for smoke tests and is not equivalent to official LLM judging.\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `deepsearchqa` |\n| **Dataset ID** | [google/deepsearchqa](https://modelscope.cn/datasets/google/deepsearchqa/summary) |\n| **Paper** | [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) |\n| **Tags** | `Agent`, `Knowledge`, `QA`, `Retrieval` |\n| **Metrics** | `f1_score`, `precision`, `recall` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `eval` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 900 |\n| Prompt Length (Mean) | 295.54 chars |\n| Prompt Length (Min/Max) | 49 / 1007 chars |\n\n## Sample Example\n\n*Sample example not available.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\n{question}\n```\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n --model YOUR_MODEL \\\n --api-url OPENAI_API_COMPAT_URL \\\n --api-key EMPTY_TOKEN \\\n --datasets deepsearchqa \\\n --agent-config '{\"mode\":\"native\",\"strategy\":\"function_calling\",\"max_steps\":30}' \\\n --limit 10 # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import TaskConfig, run_task\nfrom evalscope.api.agent import NativeAgentConfig\n\ntask_cfg = TaskConfig(\n model='YOUR_MODEL',\n api_url='OPENAI_API_COMPAT_URL',\n api_key='EMPTY_TOKEN',\n datasets=['deepsearchqa'],\n agent_config=NativeAgentConfig(\n strategy='function_calling',\n max_steps=30,\n ),\n limit=10, # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n", + "zh": "# DeepSearchQA\n\n## 概述\n\nDeepSearchQA 是 Google DeepMind 推出的一项基准测试，用于评估深度研究智能体在开放网络上执行复杂多步骤信息检索任务的能力。该基准包含 900 个提示，涵盖 17 个领域，旨在衡量模型生成完整答案集合的能力，而不仅仅是单个答案的检索。\n\n## 任务描述\n\n- **任务类型**：搜索智能体事实型问答\n- **输入**：自然语言形式的研究问题\n- **输出**：单个答案或完整的答案集合（取决于问题）\n- **评分方式**：使用大语言模型（LLM）作为评判器，通过语义匹配对比标准答案及答案类型\n\n## 核心特性\n\n- 测试从多个来源系统性整合碎片化信息的能力\n- 对于答案为集合的问题，要求进行实体消歧和去重\n- 同时惩罚检索不足和过度/幻觉生成的答案\n- 使用 `problem_category` 提供分析元数据；推理过程中对模型隐藏 `answer_type`\n- 兼容 EvalScope 智能体配置，支持原生或具备外部网络能力的智能体\n\n## 智能体工具配置\n\nDeepSearchQA 不硬编码指定搜索引擎。默认情况下，它通过 EvalScope 原生 AgentLoop 运行，不使用外部搜索工具。若要评估具备网络能力的智能体，请设置 `TaskConfig.agent_config` 并附加模型可用的搜索/获取工具。如果未指定 `NativeAgentConfig.max_steps`，DeepSearchQA 将使用其基准级 AgentLoop 默认值 30 步。\n\n有关运行示例、MCP 搜索/获取配置和评估说明，请参阅 [DeepSearchQA 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/third_party/deepsearchqa.html)。\n\n## 评估说明\n\n- EvalScope 从 `eval` 切分中加载 ModelScope 数据集 `google/deepsearchqa`。\n- 默认启用 LLM 评判器。官方起始代码使用 Gemini 2.5 Flash 模型配合 DeepSearchQA 评判提示，但 EvalScope 在本地运行时可使用任意已配置的评判模型。\n- 主要指标为 `f1_score`；同时报告 `precision`、`recall` 以及空/无效响应率。\n- `JudgeStrategy.RULE` 为冒烟测试提供保守的精确匹配/子串匹配回退机制，但不等同于官方的 LLM 评判。\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `deepsearchqa` |\n| **数据集ID** | [google/deepsearchqa](https://modelscope.cn/datasets/google/deepsearchqa/summary) |\n| **论文** | [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) |\n| **标签** | `Agent`, `Knowledge`, `QA`, `Retrieval` |\n| **指标** | `f1_score`, `precision`, `recall` |\n| **默认示例数** | 0-shot |\n| **评估切分** | `eval` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 900 |\n| 提示词长度（平均） | 295.54 字符 |\n| 提示词长度（最小/最大） | 49 / 1007 字符 |\n\n## 样例示例\n\n*样例示例不可用。*\n\n## 提示模板\n\n**提示模板：**\n```text\n{question}\n```\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n --model YOUR_MODEL \\\n --api-url OPENAI_API_COMPAT_URL \\\n --api-key EMPTY_TOKEN \\\n --datasets deepsearchqa \\\n --agent-config '{\"mode\":\"native\",\"strategy\":\"function_calling\",\"max_steps\":30}' \\\n --limit 10 # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import TaskConfig, run_task\nfrom evalscope.api.agent import NativeAgentConfig\n\ntask_cfg = TaskConfig(\n model='YOUR_MODEL',\n api_url='OPENAI_API_COMPAT_URL',\n api_key='EMPTY_TOKEN',\n datasets=['deepsearchqa'],\n agent_config=NativeAgentConfig(\n strategy='function_calling',\n max_steps=30,\n ),\n limit=10, # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```", + "content_hash": "2c4bbb4391c94edf01a50ff18466cdbf", + "needs_translation": false + }, + "updated_at": "2026-07-21T16:49:54.577664", + "translation_updated_at": "2026-07-21T16:49:56" +} diff --git a/evalscope/benchmarks/deepsearchqa/__init__.py b/evalscope/benchmarks/deepsearchqa/__init__.py new file mode 100644 index 000000000..664620f1a --- /dev/null +++ b/evalscope/benchmarks/deepsearchqa/__init__.py @@ -0,0 +1 @@ +from . import deepsearchqa_adapter # noqa: F401 diff --git a/evalscope/benchmarks/deepsearchqa/deepsearchqa_adapter.py b/evalscope/benchmarks/deepsearchqa/deepsearchqa_adapter.py new file mode 100644 index 000000000..51f5fad4c --- /dev/null +++ b/evalscope/benchmarks/deepsearchqa/deepsearchqa_adapter.py @@ -0,0 +1,151 @@ +from typing import Any, Dict, List + +from evalscope.api.benchmark import AgentLoopAdapter, BenchmarkMeta +from evalscope.api.dataset import Sample +from evalscope.api.evaluator import TaskState +from evalscope.api.metric import AggScore, SampleScore, Score +from evalscope.api.registry import register_benchmark +from evalscope.constants import Tags +from .utils import aggregate_official_scores, build_grader_prompt, parse_judge_response, rule_fallback_score + +DEEPSEARCHQA_DATASET_ID = 'google/deepsearchqa' + +DESCRIPTION = """ +## Overview + +DeepSearchQA is a Google DeepMind benchmark for evaluating deep research agents on difficult multi-step information-seeking tasks across the open web. It contains 900 prompts spanning 17 domains and is designed to measure exhaustive answer-set generation rather than single-answer retrieval alone. + +## Task Description + +- **Task Type**: Search-agent factual question answering +- **Input**: A natural-language research question +- **Output**: A single answer or complete answer set, depending on the question +- **Grading**: LLM-as-judge semantic matching against the gold answer and answer type + +## Key Features + +- Tests systematic collation of fragmented information from multiple sources +- Requires entity resolution and de-duplication for set-answer tasks +- Penalizes both under-retrieval and excessive/hallucinated answers +- Uses `problem_category` for analysis metadata; `answer_type` is withheld from the model during inference +- Compatible with EvalScope agent configurations for native or external web-capable agents + +## Agent Tool Configuration + +DeepSearchQA does not hard-code a search provider. By default it runs through EvalScope native AgentLoop without external search tools. To evaluate a web-capable agent, set `TaskConfig.agent_config` and attach the search/fetch tools that should be available to the model. If `NativeAgentConfig.max_steps` is omitted, DeepSearchQA uses its benchmark-level AgentLoop default of 30 steps. + +See the [DeepSearchQA usage guide](https://evalscope.readthedocs.io/en/latest/third_party/deepsearchqa.html) for +runtime examples, MCP search/fetch configuration, and evaluation notes. + +## Evaluation Notes + +- EvalScope loads the ModelScope dataset `google/deepsearchqa` from the `eval` split. +- LLM judge is enabled by default. Official starter code uses Gemini 2.5 Flash with the DeepSearchQA judge prompt, but EvalScope can use any configured judge model for local runs. +- The primary metric is `f1_score`; `precision`, `recall`, and empty/invalid response rates are also reported. +- `JudgeStrategy.RULE` provides a conservative exact/substring fallback for smoke tests and is not equivalent to official LLM judging. +""".strip() # noqa: E501 + + +@register_benchmark( + BenchmarkMeta( + name='deepsearchqa', + pretty_name='DeepSearchQA', + tags=[Tags.AGENT, Tags.KNOWLEDGE, Tags.QA, Tags.RETRIEVAL], + description=DESCRIPTION, + dataset_id=DEEPSEARCHQA_DATASET_ID, + metric_list=['f1_score', 'precision', 'recall'], + few_shot_num=0, + train_split=None, + eval_split='eval', + prompt_template='{question}', + paper_url='https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf', + ) +) +class DeepSearchQAAdapter(AgentLoopAdapter): + """Adapter for the DeepSearchQA deep-research benchmark.""" + llm_judge_default = True + + def __init__(self, **kwargs: Any) -> None: + super().__init__(**kwargs) + self._suppress_doc_sample_example = True + + def record_to_sample(self, record: Dict[str, Any]) -> Sample: + problem = record['problem'] + return Sample( + input=problem, + target=record['answer'] or '', + metadata={ + 'problem_category': record['problem_category'], + 'answer_type': record['answer_type'], + }, + ) + + def extract_answer(self, prediction: str, task_state: TaskState) -> str: + return prediction.strip() + + def build_max_steps_finalization_message(self, sample: Sample) -> str: + return ( + 'You have reached the maximum number of tool-use steps. Based only on the information already gathered ' + 'in this conversation, provide your best final answer now. Return only the answer, with no explanation.' + ) + + def match_score( + self, + original_prediction: str, + filtered_prediction: str, + reference: str, + task_state: TaskState, + ) -> Score: + answer_type = task_state.metadata['answer_type'] + value, metadata = rule_fallback_score(filtered_prediction, reference, answer_type) + return Score( + extracted_prediction=filtered_prediction, + prediction=original_prediction, + value=value, + metadata=metadata, + main_score_name='f1_score', + ) + + def llm_match_score( + self, + original_prediction: str, + filtered_prediction: str, + reference: str, + task_state: TaskState, + ) -> Score: + answer_type = task_state.metadata['answer_type'] + prompt = build_grader_prompt( + question=task_state.input_text, + reference=reference, + answer_type=answer_type, + response=filtered_prediction, + ) + + if not filtered_prediction: + judge_response = '' + value, judge_metadata = {}, { + 'empty_model_response': True, + 'error_message': 'AI response was empty.', + } + else: + judge_response = self.llm_judge.judge(prompt) + value, judge_metadata = parse_judge_response(judge_response) + + score = Score( + extracted_prediction=filtered_prediction, + prediction=original_prediction, + value=value, + explanation=f'LLM judge: {judge_response}', + main_score_name='f1_score', + ) + score.metadata = { + 'source': 'llm_judge', + 'judge_strategy': self.judge_strategy, + 'model': self.llm_judge.model_id, + 'rating_prompt': prompt, + **judge_metadata, + } + return score + + def aggregate_scores(self, sample_scores: List[SampleScore]) -> List[AggScore]: + return aggregate_official_scores(sample_scores) diff --git a/evalscope/benchmarks/deepsearchqa/utils.py b/evalscope/benchmarks/deepsearchqa/utils.py new file mode 100644 index 000000000..f72806854 --- /dev/null +++ b/evalscope/benchmarks/deepsearchqa/utils.py @@ -0,0 +1,267 @@ +import json +import re +from typing import Any, Dict, List, Optional, Tuple + +from evalscope.api.metric import AggScore, SampleScore + +DEEPSEARCH_QA_PROMPT = """ +Your task is to evaluate whether a given "AI Response" for a specific "User Prompt" arrived at the correct answer. + +**Answer Correctness Task** + +* **Purpose:** Assess whether the AI response provides the correct answer(s) based on the provided "Correct Answer" and "Prompt Type". +* **Process:** + * Identify the "Prompt Type": "". + * Refer to the "Correct Answer": "". + * Based on the "Prompt Type", determine if the "AI Response" contains the expected answer(s). + * **'Single Answer'**: Check if the response provides the answer that addresses the user's question. It does not have to match the exact wording of the provided answer. + * **'Set Answer'**: Check if the response includes *each* item from the provided ground truth answers. The order might not matter unless specified otherwise. The response might include more answers than the list. Determine the correctness *only* based on the list first and then check if the response includes answers not in the list. + * **Explanation:** Provide a brief explanation justifying your assessment of answer correctness, referencing specific parts of the AI response and the correct answer. + * **Correctness Details:** Provide a dictionary, one key for each expected answer part, and value is a boolean indicating whether each expected answer part was found. + * For 'Set Answer', this will be a list of attributes, one for each item/part in the "Correct Answer". Each key will be a string indicating the expected answer part, and the value will be a boolean indicating whether that part was found in the response. + * **Excessive Answers:** Provide a list of strings, each indicating an excessive answer part. If the response provides answers that are **not** in the "Correct Answer" list, add these answers as excessive answers. Return an empty list when there's no excessive answers in the response. + + +**Output Format:** + +Your evaluation *must* be structured as a nested JSON dictionary with the following top-level keys: `"Answer Correctness"`. Please return NULL if any of "Prompt", "AI Response" or "Correct Answer" is empty. +The value for `"Answer Correctness"` should be a dictionary containing `"Explanation"` (a string), `"Correctness Details"` (a dictionary where each key is the expected correct answer, and the value is a boolean indicating whether the response contains the correct answer), and `"Excessive Answers"` (a list of strings indicating the excessive answers). + +Make sure you return a valid JSON string. Pay special attention to quotes, commas and special characters in the JSON string. Make sure to escape all special characters and quotes in the JSON string. +""".strip() # noqa: E501 + +GRADER_RATING_OUTPUT_EXAMPLE = r""" + +**Example (Partial):** + +"```json +{{ + "Answer Correctness": {{ + "Explanation": "The response correctly identified Belgium and France but also includes an excessive answer, Italy.", + "Correctness Details": {{ + "Belgium": true, + "France": true, + }}, + "Excessive Answers": [ "Italy" ] + }} +}} +```" + +**Now, proceed with the evaluation using the provided User Prompt, AI Response, and Correct Answer.** + +User Prompt (Wrapped in and ): + +{prompt} + +-------------------- +** Correct Answer (Wrapped in and ): +Prompt Type: {prompt_type} + +{answer} + +-------------------- +AI assistant response (Wrapped in and ): + +{response} + + +-------------------- +Rating:""" # noqa: E501 + + +def build_grader_prompt(question: str, reference: str, answer_type: str, response: str) -> str: + return DEEPSEARCH_QA_PROMPT + GRADER_RATING_OUTPUT_EXAMPLE.format( + prompt=question, + answer=reference, + prompt_type=answer_type, + response=response, + ) + + +def rule_fallback_score(prediction: str, reference: str, answer_type: str) -> Tuple[Dict[str, float], Dict[str, Any]]: + normalized_prediction = _normalize_text(prediction) + if answer_type == 'Single Answer': + reference_parts = [reference.strip()] if reference.strip() else [] + correct = int(any(part and _normalize_text(part) in normalized_prediction for part in reference_parts)) + expected = 1 if reference_parts else 0 + excessive = 0 + else: + reference_parts = _split_reference(reference) + correct = sum(1 for part in reference_parts if part and _normalize_text(part) in normalized_prediction) + expected = len(reference_parts) + excessive = 0 + return _score_from_counts(correct=correct, expected=expected, excessive=excessive), { + 'source': 'rule_fallback', + 'correct': correct, + 'expected': expected, + 'excessive': excessive, + } + + +def parse_judge_response(response: str) -> Tuple[Dict[str, float], Dict[str, Any]]: + if not response: + return {}, { + 'empty_auto_rater_response': True, + 'error_message': 'Auto-rater response was empty.', + 'judge_raw': response, + } + + parsed = _extract_json_object(response) + if parsed is None: + return _invalid_score('Invalid JSON response from auto-rater.', response) + + answer_correctness = parsed.get('Answer Correctness') + if not isinstance(answer_correctness, dict): + return _invalid_score("Missing or malformed 'Answer Correctness' node.", response) + + explanation = answer_correctness.get('Explanation') + if not isinstance(explanation, str): + return _invalid_score("Missing or malformed 'Explanation' in Answer Correctness.", response) + + details = answer_correctness.get('Correctness Details') + if not isinstance(details, dict) or not all(isinstance(k, str) for k in details.keys()) \ + or not all(isinstance(v, bool) for v in details.values()): + return _invalid_score("Invalid 'Correctness Details' in Answer Correctness.", response) + + excessive_answers = answer_correctness.get('Excessive Answers', []) + if not isinstance(excessive_answers, list) or not all(isinstance(item, str) for item in excessive_answers): + return _invalid_score("Invalid 'Excessive Answers' in Answer Correctness.", response) + + expected = len(details) + correct = sum(1 for value in details.values() if value) + excessive = len([answer for answer in excessive_answers if answer.strip()]) + + metadata = { + 'answer_correctness_explanation': explanation, + 'correct': correct, + 'expected': expected, + 'excessive': excessive, + 'correctness_details': details, + 'excessive_answers': excessive_answers, + } + return _score_from_counts(correct=correct, expected=expected, excessive=excessive), metadata + + +def aggregate_official_scores(sample_scores: List[SampleScore]) -> List[AggScore]: + total = len(sample_scores) + if not total: + return [] + + valid_scores = [] + empty_model = 0 + empty_auto_rater = 0 + invalid_auto_rater = 0 + + for sample_score in sample_scores: + metadata = sample_score.score.metadata or {} + if metadata.get('empty_model_response'): + empty_model += 1 + continue + if metadata.get('empty_auto_rater_response'): + empty_auto_rater += 1 + continue + if metadata.get('invalid_auto_rater_response'): + invalid_auto_rater += 1 + continue + valid_scores.append(sample_score) + + metric_values: Dict[str, List[float]] = { + 'precision': [], + 'recall': [], + 'f1_score': [], + } + metric_ids: Dict[str, List[Any]] = {metric_name: [] for metric_name in metric_values} + + for sample_score in valid_scores: + for metric_name in metric_values: + if metric_name in sample_score.score.value: + metric_values[metric_name].append(float(sample_score.score.value[metric_name])) + metric_ids[metric_name].append(sample_score.sample_id) + + agg_scores = [] + for metric_name, values in metric_values.items(): + if values: + agg_scores.append( + AggScore( + score=sum(values) / len(values), + metric_name=metric_name, + aggregation_name='mean', + num=len(values), + ids=metric_ids[metric_name], + ) + ) + + for metric_name, count in { + 'empty_model_response': empty_model, + 'empty_auto_rater_response': empty_auto_rater, + 'invalid_auto_rater_response': invalid_auto_rater, + }.items(): + agg_scores.append(AggScore( + score=count / total, + metric_name=metric_name, + aggregation_name='rate', + num=total, + )) + + return agg_scores + + +def _normalize_text(text: str) -> str: + return ' '.join(re.sub(r'[^\w\s]', ' ', text.lower()).split()) + + +def _split_reference(answer: str) -> List[str]: + if not answer: + return [] + answer_text = answer.strip() + if not answer_text: + return [] + return [part.strip() for part in re.split(r',|;|\n', answer_text) if part.strip()] + + +def _extract_json_object(text: str) -> Optional[Dict[str, Any]]: + if not text: + return None + + json_str = text.strip() + start_marker = '```json' + start_idx = json_str.find(start_marker) + if start_idx != -1: + json_str = json_str[start_idx + len(start_marker):].strip() + end_idx = json_str.rfind('```') + if end_idx != -1: + json_str = json_str[:end_idx].strip() + + try: + parsed = json.loads(json_str) + except json.JSONDecodeError: + return None + + return parsed if isinstance(parsed, dict) else None + + +def _calculate_metric(true_positives: int, false_positives: int, false_negatives: int) -> Dict[str, float]: + precision = true_positives / (true_positives + false_positives) if true_positives + false_positives else 0.0 + recall = true_positives / (true_positives + false_negatives) if true_positives + false_negatives else 0.0 + f1_score = 2 * precision * recall / (precision + recall) if precision + recall else 0.0 + return { + 'precision': precision, + 'recall': recall, + 'f1_score': f1_score, + } + + +def _score_from_counts(correct: int, expected: int, excessive: int) -> Dict[str, float]: + return _calculate_metric( + true_positives=correct, + false_positives=excessive, + false_negatives=expected - correct, + ) + + +def _invalid_score(error_message: str, response: str) -> Tuple[Dict[str, float], Dict[str, Any]]: + return {}, { + 'invalid_auto_rater_response': True, + 'error_message': error_message, + 'judge_raw': response, + } diff --git a/tests/benchmark/test_agent.py b/tests/benchmark/test_agent.py index 26aff7139..a849abbf2 100644 --- a/tests/benchmark/test_agent.py +++ b/tests/benchmark/test_agent.py @@ -120,6 +120,25 @@ def test_browsecomp(self): review = json.loads(review_files[0].read_text(encoding='utf-8').strip()) self.assertNotIn('canary', review['sample_score']['sample_metadata']) + def test_deepsearchqa(self): + """Test DeepSearchQA benchmark end-to-end.""" + config_overrides = { + 'collect_perf': False, + 'debug': False, + 'eval_batch_size': 1, + 'judge_strategy': JudgeStrategy.RULE, + 'limit': 1, + 'no_timestamp': True, + 'work_dir': 'outputs/test_agent_deepsearchqa', + } + + self._run_dataset_test('deepsearchqa', use_mock=True, **config_overrides) + + review_files = list(Path('outputs/test_agent_deepsearchqa').glob('reviews/*/deepsearchqa_default.jsonl')) + self.assertEqual(len(review_files), 1) + review = json.loads(review_files[0].read_text(encoding='utf-8').strip()) + self.assertIn('answer_type', review['sample_score']['sample_metadata']) + def test_swe_bench_verified_agentic(self): """Test SWE-bench-verified agentic dataset using docker environment.""" dataset_args = { diff --git a/tests/benchmark/test_deepsearchqa.py b/tests/benchmark/test_deepsearchqa.py new file mode 100644 index 000000000..93dbf3388 --- /dev/null +++ b/tests/benchmark/test_deepsearchqa.py @@ -0,0 +1,105 @@ +import json +import unittest + +from evalscope.api.metric import SampleScore, Score +from evalscope.benchmarks.deepsearchqa.utils import aggregate_official_scores, parse_judge_response, rule_fallback_score + + +class TestDeepSearchQAUtils(unittest.TestCase): + def test_rule_fallback_handles_single_answer_substring(self): + value, metadata = rule_fallback_score('The answer is Aotearoa.', 'Aotearoa', 'Single Answer') + + self.assertEqual(metadata['correct'], 1) + self.assertEqual(metadata['expected'], 1) + self.assertEqual(value['f1_score'], 1.0) + + def test_rule_fallback_does_not_match_empty_reference_part(self): + value, metadata = rule_fallback_score('', '', 'Single Answer') + + self.assertEqual(metadata['correct'], 0) + self.assertEqual(metadata['expected'], 0) + self.assertEqual(value['f1_score'], 0.0) + + def test_rule_fallback_accepts_reordered_set_answers(self): + value, metadata = rule_fallback_score('France; Belgium', 'Belgium, France', 'Set Answer') + + self.assertEqual(metadata['correct'], 2) + self.assertEqual(metadata['excessive'], 0) + self.assertEqual(value['f1_score'], 1.0) + + def test_rule_fallback_does_not_count_missing_set_answers_as_excessive(self): + value, metadata = rule_fallback_score('Belgium', 'Belgium, France', 'Set Answer') + + self.assertEqual(metadata['correct'], 1) + self.assertEqual(metadata['expected'], 2) + self.assertEqual(metadata['excessive'], 0) + self.assertEqual(value['precision'], 1.0) + self.assertEqual(value['recall'], 0.5) + + def test_parse_judge_response_handles_official_json_fence(self): + judge_response = """ + ```json + { + "Answer Correctness": { + "Explanation": "Both answers are present.", + "Correctness Details": {"Belgium": true, "France": true}, + "Excessive Answers": [] + } + } + ``` + """ + + value, metadata = parse_judge_response(judge_response) + + self.assertEqual(value['f1_score'], 1.0) + self.assertEqual(metadata['correctness_details'], {'Belgium': True, 'France': True}) + + def test_parse_judge_response_rejects_surrounding_text_to_match_official_parser(self): + payload = { + 'Answer Correctness': { + 'Explanation': 'Only one expected answer is present.', + 'Correctness Details': {'Belgium': True, 'France': False}, + 'Excessive Answers': ['Italy'], + } + } + + value, metadata = parse_judge_response(f'Rating follows:\n{json.dumps(payload)}\nDone.') + + self.assertEqual(value, {}) + self.assertTrue(metadata['invalid_auto_rater_response']) + + def test_parse_judge_response_rejects_unknown_boolean_strings(self): + payload = { + 'Answer Correctness': { + 'Explanation': 'Malformed flag.', + 'Correctness Details': {'Belgium': 'maybe'}, + 'Excessive Answers': [], + } + } + + value, metadata = parse_judge_response(json.dumps(payload)) + + self.assertEqual(value, {}) + self.assertTrue(metadata['invalid_auto_rater_response']) + + def test_aggregate_scores_excludes_empty_and_invalid_responses_from_means(self): + sample_scores = [ + SampleScore( + sample_id=0, + score=Score(value={'precision': 1.0, 'recall': 0.5, 'f1_score': 2 / 3}, metadata={}), + ), + SampleScore(sample_id=1, score=Score(value={}, metadata={'empty_model_response': True})), + SampleScore(sample_id=2, score=Score(value={}, metadata={'invalid_auto_rater_response': True})), + ] + + scores = { + f'{score.aggregation_name}_{score.metric_name}': score for score in aggregate_official_scores(sample_scores) + } + + self.assertEqual(scores['mean_precision'].num, 1) + self.assertEqual(scores['rate_empty_model_response'].score, 1 / 3) + self.assertEqual(scores['rate_invalid_auto_rater_response'].score, 1 / 3) + + +if __name__ == '__main__': + unittest.main()