Yunnglin · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026 · Jul 21, 2026
diff --git a/docs/en/benchmarks/deepsearchqa.md b/docs/en/benchmarks/deepsearchqa.md
@@ -0,0 +1,101 @@
+# DeepSearchQA
+
+## Overview
+
+DeepSearchQA is a Google DeepMind benchmark for evaluating deep research agents on difficult multi-step information-seeking tasks across the open web. It contains 900 prompts spanning 17 domains and is designed to measure exhaustive answer-set generation rather than single-answer retrieval alone.
+
+## Task Description
+
+- **Task Type**: Search-agent factual question answering
+- **Input**: A natural-language research question
+- **Output**: A single answer or complete answer set, depending on the question
+- **Grading**: LLM-as-judge semantic matching against the gold answer and answer type
+
+## Key Features
+
+- Tests systematic collation of fragmented information from multiple sources
+- Requires entity resolution and de-duplication for set-answer tasks
+- Penalizes both under-retrieval and excessive/hallucinated answers
+- Uses `problem_category` for analysis metadata; `answer_type` is withheld from the model during inference
+- Compatible with EvalScope agent configurations for native or external web-capable agents
+
+## Agent Tool Configuration
+
+DeepSearchQA does not hard-code a search provider. By default it runs through EvalScope native AgentLoop without external search tools. To evaluate a web-capable agent, set `TaskConfig.agent_config` and attach the search/fetch tools that should be available to the model. If `NativeAgentConfig.max_steps` is omitted, DeepSearchQA uses its benchmark-level AgentLoop default of 30 steps.
+
+See the [DeepSearchQA usage guide](https://evalscope.readthedocs.io/en/latest/third_party/deepsearchqa.html) for
+runtime examples, MCP search/fetch configuration, and evaluation notes.
+
+## Evaluation Notes
+
+- EvalScope loads the ModelScope dataset `google/deepsearchqa` from the `eval` split.
+- LLM judge is enabled by default. Official starter code uses Gemini 2.5 Flash with the DeepSearchQA judge prompt, but EvalScope can use any configured judge model for local runs.
+- The primary metric is `f1_score`; `precision`, `recall`, and empty/invalid response rates are also reported.
+- `JudgeStrategy.RULE` provides a conservative exact/substring fallback for smoke tests and is not equivalent to official LLM judging.
+
+## Properties
+
+| Property | Value |
+|----------|-------|
+| **Benchmark Name** | `deepsearchqa` |
+| **Dataset ID** | [google/deepsearchqa](https://modelscope.cn/datasets/google/deepsearchqa/summary) |
+| **Paper** | [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) |
+| **Tags** | `Agent`, `Knowledge`, `QA`, `Retrieval` |
+| **Metrics** | `f1_score`, `precision`, `recall` |
+| **Default Shots** | 0-shot |
+| **Evaluation Split** | `eval` |
+
+
+## Data Statistics
+
+| Metric | Value |
+|--------|-------|
+| Total Samples | 900 |
+| Prompt Length (Mean) | 295.54 chars |
+| Prompt Length (Min/Max) | 49 / 1007 chars |
+
+## Sample Example
+
+*Sample example not available.*
+
+## Prompt Template
+
+**Prompt Template:**
+```text
+{question}
+```
+
+## Usage
+
+### Using CLI
+
+```bash
+evalscope eval \
+    --model YOUR_MODEL \
+    --api-url OPENAI_API_COMPAT_URL \
+    --api-key EMPTY_TOKEN \
+    --datasets deepsearchqa \
+    --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30}' \
+    --limit 10  # Remove this line for formal evaluation
+```
+
+### Using Python
+
+```python
+from evalscope import TaskConfig, run_task
+from evalscope.api.agent import NativeAgentConfig
+
+task_cfg = TaskConfig(
+    model='YOUR_MODEL',
+    api_url='OPENAI_API_COMPAT_URL',
+    api_key='EMPTY_TOKEN',
+    datasets=['deepsearchqa'],
+    agent_config=NativeAgentConfig(
+        strategy='function_calling',
+        max_steps=30,
+    ),
+    limit=10,  # Remove this line for formal evaluation
+)
+
+run_task(task_cfg=task_cfg)
+```
diff --git a/docs/en/get_started/supported_dataset/agent.md b/docs/en/get_started/supported_dataset/agent.md
@@ -9,6 +9,7 @@ Below is the list of supported AGENT benchmarks. Click on a benchmark name for d
 | `browsecomp` | [BrowseComp](../../benchmarks/browsecomp.md) | `Agent`, `Knowledge`, `QA` |
 | `claw_eval` | [Claw-Eval](../../benchmarks/claw_eval.md) | `Agent`, `MultiModal`, `MultiTurn` |
 | `deep_swe` | [DeepSWE](../../benchmarks/deep_swe.md) | `Agent`, `Coding`, `MultiTurn` |
+| `deepsearchqa` | [DeepSearchQA](../../benchmarks/deepsearchqa.md) | `Agent`, `Knowledge`, `QA`, `Retrieval` |
 | `gaia` | [GAIA](../../benchmarks/gaia.md) | `Agent`, `MultiTurn`, `Reasoning` |
 | `gdpval` | [GDPval](../../benchmarks/gdpval.md) | `Agent`, `Knowledge`, `MultiTurn` |
 | `general_fc` | [General-FunctionCalling](../../benchmarks/general_fc.md) | `Agent`, `Custom`, `FunctionCalling` |
@@ -41,6 +42,7 @@ Below is the list of supported AGENT benchmarks. Click on a benchmark name for d
 ../../benchmarks/browsecomp.md
 ../../benchmarks/claw_eval.md
 ../../benchmarks/deep_swe.md
+../../benchmarks/deepsearchqa.md
 ../../benchmarks/gaia.md
 ../../benchmarks/gdpval.md
 ../../benchmarks/general_fc.md

diff --git a/docs/en/third_party/deepsearchqa.md b/docs/en/third_party/deepsearchqa.md
@@ -0,0 +1,133 @@
+# DeepSearchQA
+
+## Introduction
+
+[DeepSearchQA](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) evaluates
+deep research agents on difficult web information-seeking questions. The ModelScope dataset
+[`google/deepsearchqa`](https://modelscope.cn/datasets/google/deepsearchqa/summary) contains 900 examples in the
+`eval` split. Each row contains a question, category, gold answer, and answer type (`Single Answer` or `Set Answer`).
+
+EvalScope implements the ModelScope dataset adapter and the public Kaggle starter-code judging protocol. The official
+starter uses Gemini 2.5 Flash with Google Search for answer generation and a Gemini judge. EvalScope does not hard-code
+either provider: you can run any OpenAI-compatible model and attach search/fetch tools through AgentLoop MCP servers.
+
+## Installation
+
+Install MCP support if the run uses MCP servers:
+
+```bash
+pip install 'evalscope[mcp]'
+```
+
+`evalscope[mcp]` includes the official `mcp` Python SDK and `mcp-server-fetch`. Fetch is useful for reading known URLs,
+but it is not a search engine. A formal DeepSearchQA agent run should also provide a real search MCP server.
+
+## Search MCP Requirements
+
+DeepSearchQA questions usually require discovering multiple web pages, resolving aliases, and checking several sources.
+The agent should have at least two capabilities:
+
+- `search` or `web_search`: accepts a query string and returns ranked results with title, snippet, and URL.
+- `fetch` or `fetch_url`: accepts a URL and returns readable page text.
+
+`mcp_server_fetch` only provides the second capability. If it is the only MCP server configured, the model can open URLs
+that it already knows, but it cannot reliably discover new sources.
+
+## Native AgentLoop Example
+
+The example below uses Fetch MCP plus a remote search MCP endpoint. Replace the search endpoint with your own
+ModelScope MCP, Brave/Tavily-backed MCP, or self-hosted MCP server.
+
+```python
+import os
+import sys
+
+from evalscope import TaskConfig, run_task
+from evalscope.api.agent import NativeAgentConfig
+from evalscope.api.agent.mcp import MCPServerConfigHTTP, MCPServerConfigStdio
+
+task_config = TaskConfig(
+    model='YOUR_AGENT_MODEL',
+    api_url='OPENAI_COMPATIBLE_URL',
+    api_key=os.getenv('MODEL_API_KEY'),
+    eval_type='openai_api',
+    datasets=['deepsearchqa'],
+    eval_batch_size=1,
+    limit=10,
+    judge_strategy='llm',
+    judge_model_args={
+        'model_id': 'YOUR_JUDGE_MODEL',
+        'api_url': 'OPENAI_COMPATIBLE_JUDGE_URL',
+        'api_key': os.getenv('JUDGE_API_KEY'),
+        'generation_config': {'temperature': 0.0},
+    },
+    agent_config=NativeAgentConfig(
+        strategy='function_calling',
+        max_steps=30,
+        mcp_servers=[
+            MCPServerConfigStdio(
+                name='fetch',
+                command=sys.executable,
+                args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'],
+            ),
+            MCPServerConfigHTTP(
+                name='search',
+                url='https://mcp.api-inference.modelscope.net/<search-server-id>/mcp',
+                headers={'Authorization': 'Bearer <MCP_TOKEN>'},
+            ),
+        ],
+        kwargs={
+            'system_prompt': (
+                'Use the available search and fetch tools to research the question. '
+                'Cross-check important facts before answering. When finished, call submit(answer=...) with only the '
+                'final answer or complete answer set.'
+            ),
+        },
+    ),
+)
+
+run_task(task_config)
+```
+
+## CLI Example
+
+When configuring MCP servers through CLI JSON, include each server's `type` field.
+
+```bash
+evalscope eval \
+  --model YOUR_AGENT_MODEL \
+  --api-url OPENAI_COMPATIBLE_URL \
+  --api-key "$MODEL_API_KEY" \
+  --datasets deepsearchqa \
+  --limit 10 \
+  --eval-batch-size 1 \
+  --judge-strategy llm \
+  --judge-model-args '{"model_id":"YOUR_JUDGE_MODEL","api_url":"OPENAI_COMPATIBLE_JUDGE_URL","api_key":"'"$JUDGE_API_KEY"'","generation_config":{"temperature":0.0}}' \
+  --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30,"mcp_servers":[{"type":"stdio","name":"fetch","command":"python","args":["-m","mcp_server_fetch","--ignore-robots-txt"]},{"type":"http","name":"search","url":"https://mcp.api-inference.modelscope.net/<search-server-id>/mcp","headers":{"Authorization":"Bearer <MCP_TOKEN>"}}],"kwargs":{"system_prompt":"Use search and fetch tools to research the question. Cross-check important facts before answering. When finished, call submit(answer=...) with only the final answer."}}'
+```
+
+## Scoring
+
+The primary metric is `f1_score`; `precision` and `recall` are also reported. EvalScope additionally emits pipeline
+health rates from the official starter logic:
+
+- `empty_model_response`
+- `empty_auto_rater_response`
+- `invalid_auto_rater_response`
+
+Empty model responses and empty/invalid judge responses are excluded from the valid-sample mean precision/recall/F1
+denominator and reported separately as rates. Per-sample reviews keep the judge details, including which expected
+answers were found and which excessive answers were detected.
+
+## Notes
+
+- `function_calling` expects the model to call `submit(answer=...)` when it is done.
+- DeepSearchQA adds a max-step finalization turn: if the loop reaches `max_steps` with an empty completion, the model is
+  asked to answer from the information already gathered.
+- For comparable runs, keep the judge temperature at `0.0` and use the same judge model across all systems.
+
+## Resources
+
+- [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf)
+- [ModelScope dataset](https://modelscope.cn/datasets/google/deepsearchqa/summary)
+- [Agent MCP guide](../user_guides/agent/native.md#mcp-server-tools)
diff --git a/docs/en/third_party/index.md b/docs/en/third_party/index.md
@@ -12,6 +12,7 @@ skillsbench.md
 toolathlon.md
 gaia.md
 wide_search.md
+deepsearchqa.md
 swe_bench.md
 swe_bench_pro.md
 tau_bench.md

diff --git a/docs/en/user_guides/agent/native.md b/docs/en/user_guides/agent/native.md
@@ -99,6 +99,9 @@ pip install evalscope[mcp]
 
 `evalscope[mcp]` bundles the official `mcp` Python SDK plus `mcp-server-fetch` (universal HTTP fetching, no API key). Other MCP servers can be installed on demand or run via `uvx` / `npx`.
 
+For search-agent benchmark examples that combine search MCP servers with fetch/browser-style tools, see the
+[DeepSearchQA](../../third_party/deepsearchqa.md) and [WideSearch](../../third_party/wide_search.md) guides.
+
 ### Quick start
 
 Attach HTTP fetching and a remote MCP tool to a GAIA evaluation:

diff --git a/docs/zh/benchmarks/deepsearchqa.md b/docs/zh/benchmarks/deepsearchqa.md
@@ -0,0 +1,100 @@
+# DeepSearchQA
+
+## 概述
+
+DeepSearchQA 是 Google DeepMind 推出的一项基准测试，用于评估深度研究智能体在开放网络上执行复杂多步骤信息检索任务的能力。该基准包含 900 个提示，涵盖 17 个领域，旨在衡量模型生成完整答案集合的能力，而不仅仅是单个答案的检索。
+
+## 任务描述
+
+- **任务类型**：搜索智能体事实型问答
+- **输入**：自然语言形式的研究问题
+- **输出**：单个答案或完整的答案集合（取决于问题）
+- **评分方式**：使用大语言模型（LLM）作为评判器，通过语义匹配对比标准答案及答案类型
+
+## 核心特性
+
+- 测试从多个来源系统性整合碎片化信息的能力
+- 对于答案为集合的问题，要求进行实体消歧和去重
+- 同时惩罚检索不足和过度/幻觉生成的答案
+- 使用 `problem_category` 提供分析元数据；推理过程中对模型隐藏 `answer_type`
+- 兼容 EvalScope 智能体配置，支持原生或具备外部网络能力的智能体
+
+## 智能体工具配置
+
+DeepSearchQA 不硬编码指定搜索引擎。默认情况下，它通过 EvalScope 原生 AgentLoop 运行，不使用外部搜索工具。若要评估具备网络能力的智能体，请设置 `TaskConfig.agent_config` 并附加模型可用的搜索/获取工具。如果未指定 `NativeAgentConfig.max_steps`，DeepSearchQA 将使用其基准级 AgentLoop 默认值 30 步。
+
+有关运行示例、MCP 搜索/获取配置和评估说明，请参阅 [DeepSearchQA 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/third_party/deepsearchqa.html)。
+
+## 评估说明
+
+- EvalScope 从 `eval` 切分中加载 ModelScope 数据集 `google/deepsearchqa`。
+- 默认启用 LLM 评判器。官方起始代码使用 Gemini 2.5 Flash 模型配合 DeepSearchQA 评判提示，但 EvalScope 在本地运行时可使用任意已配置的评判模型。
+- 主要指标为 `f1_score`；同时报告 `precision`、`recall` 以及空/无效响应率。
+- `JudgeStrategy.RULE` 为冒烟测试提供保守的精确匹配/子串匹配回退机制，但不等同于官方的 LLM 评判。
+
+## 属性
+
+| 属性 | 值 |
+|----------|-------|
+| **基准测试名称** | `deepsearchqa` |
+| **数据集ID** | [google/deepsearchqa](https://modelscope.cn/datasets/google/deepsearchqa/summary) |
+| **论文** | [Paper](https://storage.googleapis.com/deepmind-media/DeepSearchQA/DeepSearchQA_benchmark_paper.pdf) |
+| **标签** | `Agent`, `Knowledge`, `QA`, `Retrieval` |
+| **指标** | `f1_score`, `precision`, `recall` |
+| **默认示例数** | 0-shot |
+| **评估切分** | `eval` |
+
+
+## 数据统计
+
+| 指标 | 值 |
+|--------|-------|
+| 总样本数 | 900 |
+| 提示词长度（平均） | 295.54 字符 |
+| 提示词长度（最小/最大） | 49 / 1007 字符 |
+
+## 样例示例
+
+*样例示例不可用。*
+
+## 提示模板
+
+**提示模板：**
+```text
+{question}
+```
+
+## 使用方法
+
+### 使用 CLI
+
+```bash
+evalscope eval \
+    --model YOUR_MODEL \
+    --api-url OPENAI_API_COMPAT_URL \
+    --api-key EMPTY_TOKEN \
+    --datasets deepsearchqa \
+    --agent-config '{"mode":"native","strategy":"function_calling","max_steps":30}' \
+    --limit 10  # 正式评估时请删除此行
+```
+
+### 使用 Python
+
+```python
+from evalscope import TaskConfig, run_task
+from evalscope.api.agent import NativeAgentConfig
+
+task_cfg = TaskConfig(
+    model='YOUR_MODEL',
+    api_url='OPENAI_API_COMPAT_URL',
+    api_key='EMPTY_TOKEN',
+    datasets=['deepsearchqa'],
+    agent_config=NativeAgentConfig(
+        strategy='function_calling',
+        max_steps=30,
+    ),
+    limit=10,  # 正式评估时请删除此行
+)
+
+run_task(task_cfg=task_cfg)
+```
diff --git a/docs/zh/get_started/supported_dataset/agent.md b/docs/zh/get_started/supported_dataset/agent.md
@@ -9,6 +9,7 @@
 | `browsecomp` | [BrowseComp](../../benchmarks/browsecomp.md) | `Agent`, `Knowledge`, `QA` |
 | `claw_eval` | [Claw-Eval](../../benchmarks/claw_eval.md) | `Agent`, `MultiModal`, `MultiTurn` |
 | `deep_swe` | [DeepSWE](../../benchmarks/deep_swe.md) | `Agent`, `Coding`, `MultiTurn` |
+| `deepsearchqa` | [DeepSearchQA](../../benchmarks/deepsearchqa.md) | `Agent`, `Knowledge`, `QA`, `Retrieval` |
 | `gaia` | [GAIA](../../benchmarks/gaia.md) | `Agent`, `MultiTurn`, `Reasoning` |
 | `gdpval` | [GDPval](../../benchmarks/gdpval.md) | `Agent`, `Knowledge`, `MultiTurn` |
 | `general_fc` | [General-FunctionCalling](../../benchmarks/general_fc.md) | `Agent`, `Custom`, `FunctionCalling` |
@@ -41,6 +42,7 @@
 ../../benchmarks/browsecomp.md
 ../../benchmarks/claw_eval.md
 ../../benchmarks/deep_swe.md
+../../benchmarks/deepsearchqa.md
 ../../benchmarks/gaia.md
 ../../benchmarks/gdpval.md
 ../../benchmarks/general_fc.md