feat: search.open page reading via Jina Reader + top-10 search default

README.md

@@ -6,7 +6,7 @@ A minimal CLI coding agent with a persistent IPython execution environment and o
 
 The model gets a single built-in tool, `ipython`: a persistent IPython kernel for Python, shell commands via `!command`, and multi-line shell scripts via `%%bash`. The tool set is not configurable. File edits, shell work, and orchestration all go through it.
 
-For convenience, rlm ships built-in *skills* that can be enabled per run via `RLM_SKILLS` (comma-separated, off by default): `edit` (single-occurrence string replacement) and `search` (web search via Serper, needs `SERPER_API_KEY`). Enabled skills are pre-imported into the IPython kernel like any other skill (see [Skills](#skills)), so the agent calls `await edit(path=..., old_str=..., new_str=...)` or `await search(query=...)`.
+For convenience, rlm ships built-in *skills* that can be enabled per run via `RLM_SKILLS` (comma-separated, off by default): `edit` (single-occurrence string replacement), `search` (web search via Serper, needs `SERPER_API_KEY`), and `open_webpage` (fetch a URL as text; uses the Jina Reader API when `JINA_API_KEY` is set, direct fetch otherwise). Enabled skills are pre-imported into the IPython kernel like any other skill (see [Skills](#skills)), so the agent calls `await edit(path=..., old_str=..., new_str=...)`, `await search(query=...)`, or `await open_webpage(url=...)`.
 
 Context is reclaimed automatically: when a turn's prompt token count crosses `RLM_SUMMARIZE_AT_TOKENS`, the engine compacts the conversation into a summary and continues on a fresh branch. The IPython kernel keeps running across the compaction, so REPL state survives (see [Compaction](#compaction)).
 
@@ -57,9 +57,10 @@ All configuration is via environment variables:
 | `RLM_MODEL` | `openai/gpt-5-mini` | Model name (PI Inference slug). Override with `--model` or `RLM_MODEL` for OpenAI/Anthropic direct (e.g. `gpt-4o`, `claude-sonnet-4-5`) |
 | `RLM_API_KEY` / `RLM_BASE_URL` | — / SDK default (`https://api.openai.com/v1`) | Explicit override (highest priority). Independent: setting `RLM_API_KEY` alone targets the SDK default endpoint; set `RLM_BASE_URL` too for a custom endpoint. For PI, use `PRIME_API_KEY` (below) which owns the full pair. |
 | `SERPER_API_KEY` | — | API key for the built-in `search` skill (Serper backend). Required when `search` is enabled. |
+| `JINA_API_KEY` | — | API key for the built-in `open_webpage` skill. Optional: when set, pages are fetched via the Jina Reader API (markdown, handles JS-rendered pages), with direct-fetch fallback. |
 | `PRIME_API_KEY` | — | PI Inference pair: targets `https://api.pinference.ai/api/v1` and forwards `PRIME_TEAM_ID` as `X-Prime-Team-ID` when set. |
 | `OPENAI_API_KEY` / `OPENAI_BASE_URL` | resolved by SDK | OpenAI pair — when `OPENAI_API_KEY` is set, AsyncOpenAI's native env handling is used (covers OpenAI direct and verifiers' rollout tunnel both). Provider precedence: explicit → PI → OpenAI. Keys are scoped to their own base URL so an `OPENAI_API_KEY` lying around can't leak to PI Inference. |
-| `RLM_SKILLS` | — | Comma-separated built-in skills to enable (`edit`, `search`); pre-imported into the kernel. Unknown names raise. See [Skills](#skills). |
+| `RLM_SKILLS` | — | Comma-separated built-in skills to enable (`edit`, `search`, `open_webpage`); pre-imported into the kernel. Unknown names raise. See [Skills](#skills). |
 | `RLM_MCP_CONFIG` | — | Standard `mcpServers` URL map; each server's tools become pre-imported IPython skills (`<server>_<tool>`). See [MCP tools as skills](#mcp-tools-as-skills). |
 | `RLM_MAX_DEPTH` | `0` | Max recursion depth (`0` means no sub-agents) |
 | `RLM_EXEC_TIMEOUT` | `300` | Seconds per IPython execution |
@@ -119,7 +120,7 @@ These artifacts are consumable for debugging, visualization, or training-data ex
 
 ## Skills
 
-`rlm` ships a small set of built-in skills enabled per run via `RLM_SKILLS` (`edit`, `search`; see [MCP tools as skills](#mcp-tools-as-skills) for the related MCP path). `search` does web search through Serper and needs `SERPER_API_KEY`; it returns title/URL/snippet for a single query (`await search(query="...")`). Additional skills are supplied by the host environment: before `install.sh` runs, the environment places skill packages under `/task/rlm-skills/<name>/`, and `install.sh` installs them alongside `rlm` so they're both importable and on `$PATH`.
+`rlm` ships a small set of built-in skills enabled per run via `RLM_SKILLS` (`edit`, `search`, `open_webpage`; see [MCP tools as skills](#mcp-tools-as-skills) for the related MCP path). `search` does web search through Serper and needs `SERPER_API_KEY`; it returns title/URL/snippet for a single query (`await search(query="...")`) or a list of queries batched into one API call (`await search(query=["...", "..."])`), top 10 results per query by default. `open_webpage` fetches a URL and returns its text (`await open_webpage(url="...")`), via the Jina Reader API when `JINA_API_KEY` is set and a direct fetch (HTML/PDF parsed locally) otherwise. Additional skills are supplied by the host environment: before `install.sh` runs, the environment places skill packages under `/task/rlm-skills/<name>/`, and `install.sh` installs them alongside `rlm` so they're both importable and on `$PATH`.
 
 From IPython, import a skill and call its async `run(...)` entrypoint:
 

pyproject.toml

@@ -24,6 +24,7 @@ dependencies = [
     "scipy",
     "beautifulsoup4",
     "lxml",
+    "pdfminer.six",
     "pydantic",
 ]
 

src/rlm/prompt.py

@@ -76,7 +76,14 @@
     "For web search, use the pre-imported async `search` skill from IPython: "
     '`await search(query="...")`. Results come back as title, URL, and snippet; '
     "assign the result to a variable so you can revisit it. To cover "
-    "several angles at once, fan out with `asyncio.gather(search(...), search(...))`."
+    "several angles at once, pass a list of queries — they are batched into "
+    'one API call: `await search(query=["...", "..."])`.'
+)
+OPEN_WEBPAGE_SKILL_PROMPT = (
+    "To read a page in full, use the pre-imported async `open_webpage` skill "
+    'from IPython: `await open_webpage(url="...")`. It returns the page text '
+    "(HTML and PDF are parsed); assign the result to a variable so you can "
+    "slice and search it instead of re-fetching."
 )
 
 
@@ -127,6 +134,8 @@ def build_system_prompt(
             skill_lines.append(EDIT_SKILL_PROMPT)
         if "search" in installed_skills:
             skill_lines.append(SEARCH_SKILL_PROMPT)
+        if "open_webpage" in installed_skills:
+            skill_lines.append(OPEN_WEBPAGE_SKILL_PROMPT)
     if skill_lines:
         parts.extend(["", *skill_lines])
 

src/rlm/skills/__init__.py

@@ -13,6 +13,7 @@
 # Built-in skill name -> module its ``run`` is re-exported from.
 _BUILTIN_SKILLS: dict[str, str] = {
     "edit": "rlm.skills.edit",
+    "open_webpage": "rlm.skills.open_webpage",
     "search": "rlm.skills.search",
 }
 

src/rlm/skills/open_webpage.py

@@ -0,0 +1,92 @@
+"""Built-in ``open_webpage`` skill — fetch a URL and return its text content.
+
+Enabled via ``RLM_SKILLS``; pre-imported into the IPython kernel so the agent calls
+``await open_webpage(url="...")``. When ``JINA_API_KEY`` is set, pages are fetched
+through the Jina Reader API (markdown output, handles JS-rendered pages and PDFs),
+falling back to a direct fetch parsed locally on failure.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import io
+import logging
+import os
+import re
+
+import httpx
+
+JINA_READER_URL = "https://r.jina.ai"
+
+
+def _pdf_to_text(pdf_bytes: bytes) -> str:
+    from pdfminer.high_level import extract_text
+
+    logging.getLogger("pdfminer").setLevel(logging.ERROR)
+    return extract_text(io.BytesIO(pdf_bytes)) or ""
+
+
+def _html_to_text(html_text: str) -> str:
+    from bs4 import BeautifulSoup
+
+    soup = BeautifulSoup(html_text, "lxml")
+    for tag in soup(["script", "style", "noscript", "svg"]):
+        tag.decompose()
+    text = soup.get_text("\n")
+    text = re.sub(r"[ \t]{2,}", " ", text)
+    text = re.sub(r"\n{3,}", "\n\n", text)
+    return text
+
+
+def _fetch_jina(url: str, api_key: str, timeout: float) -> str:
+    response = httpx.get(
+        f"{JINA_READER_URL}/{url}",
+        timeout=timeout,
+        headers={"Authorization": f"Bearer {api_key}", "X-Timeout": str(int(timeout))},
+    )
+    response.raise_for_status()
+    return response.text.strip()
+
+
+def _fetch_direct(url: str, timeout: float) -> str:
+    response = httpx.get(
+        url,
+        timeout=timeout,
+        follow_redirects=True,
+        headers={"User-Agent": "Mozilla/5.0"},
+    )
+    response.raise_for_status()
+    content_type = (response.headers.get("content-type") or "").lower()
+    body = response.content
+    if body.startswith(b"%PDF-") or "application/pdf" in content_type:
+        return _pdf_to_text(body).strip()
+    if "text/html" in content_type or "<html" in response.text[:2048].lower():
+        return _html_to_text(response.text).strip()
+    return response.text.strip()
+
+
+def open_webpage(url: str, timeout: float = 30) -> str:
+    """Fetch a URL and return its text content, via Jina Reader when available."""
+    api_key = os.environ.get("JINA_API_KEY", "")
+    if api_key:
+        try:
+            return _fetch_jina(url, api_key, timeout)
+        except httpx.HTTPError:
+            pass  # fall back to direct fetch
+    return _fetch_direct(url, timeout)
+
+
+async def run(url: str, *, timeout: float = 30) -> str:
+    """Fetch a URL and return its text content. Handles HTML and PDF.
+
+    Uses the Jina Reader API (markdown; handles JS-rendered pages) when
+    ``JINA_API_KEY`` is set, falling back to a direct fetch parsed locally.
+
+    Args:
+        url: The URL to fetch.
+        timeout: Request timeout in seconds.
+
+    Returns:
+        The page text.
+    """
+    return await asyncio.to_thread(open_webpage, url, timeout)

src/rlm/skills/search.py

@@ -1,8 +1,9 @@
 """Built-in ``search`` skill — web search via Serper.
 
 Enabled via ``RLM_SKILLS``; pre-imported into the IPython kernel so the agent calls
-``await search(query="...")``. Needs ``SERPER_API_KEY``. Ported from the Serper ``websearch``
-skill in research-environments/rlm_browsecomp.
+``await search(query="...")``, or ``await search(query=["...", "..."])`` to batch
+several queries into one API call. Needs ``SERPER_API_KEY``. Ported from the Serper
+``websearch`` skill in research-environments/rlm_browsecomp.
 """
 
 from __future__ import annotations
@@ -32,30 +33,44 @@ def format_results(results, query: str) -> str:
     return "\n\n---\n\n".join(sections)
 
 
-def search(query: str, num_results: int = 5) -> str:
-    """Run a synchronous Serper web search and return formatted results."""
+def search(queries: list[str], num_results: int = 10) -> str:
+    """Run one Serper API call for one or more queries and return formatted results."""
     api_key = os.environ.get("SERPER_API_KEY", "")
     if not api_key:
         return "Error: SERPER_API_KEY environment variable is not set"
+    payload = [{"q": query, "num": num_results} for query in queries]
     response = httpx.post(
         SERPER_URL,
-        json={"q": query},
+        json=payload[0] if len(payload) == 1 else payload,
         headers={"X-API-KEY": api_key, "Content-Type": "application/json"},
         timeout=45,
     )
     response.raise_for_status()
-    organic = response.json().get("organic") or []
-    return format_results(organic[:num_results], query)
+    data = response.json()
+    responses = data if isinstance(data, list) else [data]
+    sections = [
+        format_results((r.get("organic") or [])[:num_results], query)
+        for query, r in zip(queries, responses, strict=True)
+    ]
+    if len(sections) == 1:
+        return sections[0]
+    return "\n\n==========\n\n".join(
+        f'Results for query "{query}":\n\n{section}'
+        for query, section in zip(queries, sections)
+    )
 
 
-async def run(query: str, *, num_results: int = 5) -> str:
-    """Run a web search via Serper and return formatted results.
+async def run(query: str | list[str], *, num_results: int = 10) -> str:
+    """Run web search(es) via Serper and return formatted results.
 
     Args:
-        query: Web search query.
-        num_results: Number of results to return.
+        query: A search query, or a list of queries batched into one API call.
+        num_results: Number of results to return per query.
 
     Returns:
-        Formatted results (title, URL, snippet).
+        Formatted results (title, URL, snippet); one section per query when batched.
     """
-    return await asyncio.to_thread(search, query, num_results)
+    queries = [query] if isinstance(query, str) else list(query)
+    if not queries:
+        return "Error: query must be a non-empty string or list of strings"
+    return await asyncio.to_thread(search, queries, num_results)

tests/test_builtin_skills.py

@@ -1,4 +1,4 @@
-"""Tests for built-in skills (``rlm.skills``): the ``edit``/``search`` skills + enable mechanism."""
+"""Tests for built-in skills (``rlm.skills``): the ``edit``/``search``/``open_webpage`` skills + enable mechanism."""
 
 from __future__ import annotations
 
@@ -48,12 +48,56 @@ def test_search_enable_writes_stub(tmp_path):
     assert (tmp_path / "search.py").read_text() == "from rlm.skills.search import run\n"
 
 
+def test_open_webpage_enable_writes_stub(tmp_path):
+    assert "open_webpage" in available_builtin_skills()
+    assert enable_builtin_skills(["open_webpage"], tmp_path) == ["open_webpage"]
+    assert (
+        tmp_path / "open_webpage.py"
+    ).read_text() == "from rlm.skills.open_webpage import run\n"
+
+
 async def test_search_missing_api_key_returns_error(monkeypatch):
     monkeypatch.delenv("SERPER_API_KEY", raising=False)
     result = await run_search(query="anything")
     assert "SERPER_API_KEY" in result
 
 
+async def test_search_batches_queries_into_one_call(monkeypatch):
+    import rlm.skills.search as search_skill
+
+    captured = {}
+    organic = [{"title": "t", "link": "https://x", "snippet": "s"}]
+
+    class FakeResponse:
+        def __init__(self, data):
+            self._data = data
+
+        def raise_for_status(self):
+            pass
+
+        def json(self):
+            return self._data
+
+    def fake_post(url, json=None, headers=None, timeout=None):
+        captured["json"] = json
+        if isinstance(json, list):
+            return FakeResponse([{"organic": organic} for _ in json])
+        return FakeResponse({"organic": organic})
+
+    monkeypatch.setenv("SERPER_API_KEY", "test-key")
+    monkeypatch.setattr(search_skill.httpx, "post", fake_post)
+
+    single = await run_search(query="one", num_results=3)
+    assert captured["json"] == {"q": "one", "num": 3}
+    assert single.startswith("Result 1: t")
+
+    batched = await run_search(query=["one", "two"])
+    assert captured["json"] == [{"q": "one", "num": 10}, {"q": "two", "num": 10}]
+    assert 'Results for query "one":' in batched
+    assert 'Results for query "two":' in batched
+    assert "\n\n==========\n\n" in batched
+
+
 def test_search_format_results():
     results = [
         {"title": "First", "link": "https://a", "snippet": "snippet one"},