feat: search.open page reading via Jina Reader + top-10 search default

README.md

@@ -6,7 +6,7 @@ A minimal CLI coding agent with a persistent IPython execution environment and o
 
 The model gets a single built-in tool, `ipython`: a persistent IPython kernel for Python, shell commands via `!command`, and multi-line shell scripts via `%%bash`. The tool set is not configurable. File edits, shell work, and orchestration all go through it.
 
-For convenience, rlm ships built-in *skills* that can be enabled per run via `RLM_SKILLS` (comma-separated, off by default): `edit` (single-occurrence string replacement) and `search` (web search via Serper, needs `SERPER_API_KEY`). Enabled skills are pre-imported into the IPython kernel like any other skill (see [Skills](#skills)), so the agent calls `await edit(path=..., old_str=..., new_str=...)` or `await search(query=...)`.
+For convenience, rlm ships built-in *skills* that can be enabled per run via `RLM_SKILLS` (comma-separated, off by default): `edit` (single-occurrence string replacement) and `search` (web search via Serper, needs `SERPER_API_KEY`; also exposes `search.open` to fetch a URL as text, via the Jina Reader API when `JINA_API_KEY` is set and a direct fetch otherwise). Enabled skills are pre-imported into the IPython kernel like any other skill (see [Skills](#skills)), so the agent calls `await edit(path=..., old_str=..., new_str=...)`, `await search(query=...)`, or `await search.open(url=...)`.
 
 Context is reclaimed automatically: when a turn's prompt token count crosses `RLM_SUMMARIZE_AT_TOKENS`, the engine compacts the conversation into a summary and continues on a fresh branch. The IPython kernel keeps running across the compaction, so REPL state survives (see [Compaction](#compaction)).
 
@@ -57,6 +57,7 @@ All configuration is via environment variables:
 | `RLM_MODEL` | `openai/gpt-5-mini` | Model name (PI Inference slug). Override with `--model` or `RLM_MODEL` for OpenAI/Anthropic direct (e.g. `gpt-4o`, `claude-sonnet-4-5`) |
 | `RLM_API_KEY` / `RLM_BASE_URL` | — / SDK default (`https://api.openai.com/v1`) | Explicit override (highest priority). Independent: setting `RLM_API_KEY` alone targets the SDK default endpoint; set `RLM_BASE_URL` too for a custom endpoint. For PI, use `PRIME_API_KEY` (below) which owns the full pair. |
 | `SERPER_API_KEY` | — | API key for the built-in `search` skill (Serper backend). Required when `search` is enabled. |
+| `JINA_API_KEY` | — | API key for `search.open` page reading. Optional: when set, pages are fetched via the Jina Reader API (markdown, handles JS-rendered pages), with direct-fetch fallback. |
 | `PRIME_API_KEY` | — | PI Inference pair: targets `https://api.pinference.ai/api/v1` and forwards `PRIME_TEAM_ID` as `X-Prime-Team-ID` when set. |
 | `OPENAI_API_KEY` / `OPENAI_BASE_URL` | resolved by SDK | OpenAI pair — when `OPENAI_API_KEY` is set, AsyncOpenAI's native env handling is used (covers OpenAI direct and verifiers' rollout tunnel both). Provider precedence: explicit → PI → OpenAI. Keys are scoped to their own base URL so an `OPENAI_API_KEY` lying around can't leak to PI Inference. |
 | `RLM_SKILLS` | — | Comma-separated built-in skills to enable (`edit`, `search`); pre-imported into the kernel. Unknown names raise. See [Skills](#skills). |
@@ -119,7 +120,7 @@ These artifacts are consumable for debugging, visualization, or training-data ex
 
 ## Skills
 
-`rlm` ships a small set of built-in skills enabled per run via `RLM_SKILLS` (`edit`, `search`; see [MCP tools as skills](#mcp-tools-as-skills) for the related MCP path). `search` does web search through Serper and needs `SERPER_API_KEY`; it returns title/URL/snippet for a single query (`await search(query="...")`). Additional skills are supplied by the host environment: before `install.sh` runs, the environment places skill packages under `/task/rlm-skills/<name>/`, and `install.sh` installs them alongside `rlm` so they're both importable and on `$PATH`.
+`rlm` ships a small set of built-in skills enabled per run via `RLM_SKILLS` (`edit`, `search`; see [MCP tools as skills](#mcp-tools-as-skills) for the related MCP path). `search` does web search through Serper and needs `SERPER_API_KEY`; it returns title/URL/snippet for a single query (`await search(query="...")`), top 10 results by default. It also exposes `search.open` to fetch a URL and return its text (`await search.open(url="...")`), via the Jina Reader API when `JINA_API_KEY` is set and a direct fetch (HTML/PDF parsed locally) otherwise. Additional skills are supplied by the host environment: before `install.sh` runs, the environment places skill packages under `/task/rlm-skills/<name>/`, and `install.sh` installs them alongside `rlm` so they're both importable and on `$PATH`.
 
 From IPython, import a skill and call its async `run(...)` entrypoint:
 

pyproject.toml

@@ -24,6 +24,7 @@ dependencies = [
     "scipy",
     "beautifulsoup4",
     "lxml",
+    "pdfminer.six",
     "pydantic",
 ]
 

src/rlm/prompt.py

@@ -76,7 +76,10 @@
     "For web search, use the pre-imported async `search` skill from IPython: "
     '`await search(query="...")`. Results come back as title, URL, and snippet; '
     "assign the result to a variable so you can revisit it. To cover "
-    "several angles at once, fan out with `asyncio.gather(search(...), search(...))`."
+    "several angles at once, fan out with `asyncio.gather(search(...), search(...))`. "
+    'To read a promising result in full, use `await search.open(url="...")` — it '
+    "returns the page text (HTML and PDF are parsed); assign it to a variable so "
+    "you can slice and search it instead of re-fetching."
 )
 
 

src/rlm/skills/__init__.py

@@ -1,7 +1,8 @@
 """Built-in skills shipped with rlm, enabled per run via ``RLM_SKILLS``.
 
 Each built-in skill is a module here exposing an async ``run(...)`` (the same contract as an
-uploaded skill). When enabled, a thin re-export module is written into the session directory
+uploaded skill), plus any extra public async functions listed in its ``__all__`` (e.g.
+``search.open``). When enabled, a thin re-export module is written into the session directory
 (on the kernel's ``sys.path``) so the kernel pre-imports it by name — the same path MCP-tool
 skills take (see ``rlm.mcp``). The agent then calls it from IPython, e.g. ``await edit(...)``.
 """
@@ -10,7 +11,7 @@
 
 from pathlib import Path
 
-# Built-in skill name -> module its ``run`` is re-exported from.
+# Built-in skill name -> module whose ``__all__`` is re-exported.
 _BUILTIN_SKILLS: dict[str, str] = {
     "edit": "rlm.skills.edit",
     "search": "rlm.skills.search",
@@ -35,7 +36,5 @@ def enable_builtin_skills(names: list[str], dest_dir: Path) -> list[str]:
         )
     dest_dir.mkdir(parents=True, exist_ok=True)
     for name in names:
-        (dest_dir / f"{name}.py").write_text(
-            f"from {_BUILTIN_SKILLS[name]} import run\n"
-        )
+        (dest_dir / f"{name}.py").write_text(f"from {_BUILTIN_SKILLS[name]} import *\n")
     return names

src/rlm/skills/edit.py

@@ -9,6 +9,8 @@
 
 from pathlib import Path
 
+__all__ = ["run"]
+
 
 async def run(path: str, old_str: str, new_str: str) -> str:
     """Replace a unique string in a file.

src/rlm/skills/search.py

@@ -1,18 +1,26 @@
-"""Built-in ``search`` skill — web search via Serper.
+"""Built-in ``search`` skill — web search via Serper, plus ``search.open`` page reading.
 
 Enabled via ``RLM_SKILLS``; pre-imported into the IPython kernel so the agent calls
-``await search(query="...")``. Needs ``SERPER_API_KEY``. Ported from the Serper ``websearch``
-skill in research-environments/rlm_browsecomp.
+``await search(query="...")`` and ``await search.open(url="...")``. Search needs
+``SERPER_API_KEY``. ``open`` fetches pages through the Jina Reader API when
+``JINA_API_KEY`` is set, falling back to a direct fetch parsed locally. Ported from
+the Serper ``websearch`` skill in research-environments/rlm_browsecomp.
 """
 
 from __future__ import annotations
 
 import asyncio
+import io
+import logging
 import os
+import re
 
 import httpx
 
+__all__ = ["run", "open"]
+
 SERPER_URL = "https://google.serper.dev/search"
+JINA_READER_URL = "https://r.jina.ai"
 
 
 def format_results(results, query: str) -> str:
@@ -32,14 +40,14 @@ def format_results(results, query: str) -> str:
     return "\n\n---\n\n".join(sections)
 
 
-def search(query: str, num_results: int = 5) -> str:
+def search(query: str, num_results: int = 10) -> str:
     """Run a synchronous Serper web search and return formatted results."""
     api_key = os.environ.get("SERPER_API_KEY", "")
     if not api_key:
         return "Error: SERPER_API_KEY environment variable is not set"
     response = httpx.post(
         SERPER_URL,
-        json={"q": query},
+        json={"q": query, "num": num_results},
         headers={"X-API-KEY": api_key, "Content-Type": "application/json"},
         timeout=45,
     )
@@ -48,9 +56,11 @@ def search(query: str, num_results: int = 5) -> str:
     return format_results(organic[:num_results], query)
 
 
-async def run(query: str, *, num_results: int = 5) -> str:
+async def run(query: str, *, num_results: int = 10) -> str:
     """Run a web search via Serper and return formatted results.
 
+    Use ``await search.open(url=...)`` to read a result page in full.
+
     Args:
         query: Web search query.
         num_results: Number of results to return.
@@ -59,3 +69,77 @@ async def run(query: str, *, num_results: int = 5) -> str:
         Formatted results (title, URL, snippet).
     """
     return await asyncio.to_thread(search, query, num_results)
+
+
+def _pdf_to_text(pdf_bytes: bytes) -> str:
+    from pdfminer.high_level import extract_text
+
+    logging.getLogger("pdfminer").setLevel(logging.ERROR)
+    return extract_text(io.BytesIO(pdf_bytes)) or ""
+
+
+def _html_to_text(html_text: str) -> str:
+    from bs4 import BeautifulSoup
+
+    soup = BeautifulSoup(html_text, "lxml")
+    for tag in soup(["script", "style", "noscript", "svg"]):
+        tag.decompose()
+    text = soup.get_text("\n")
+    text = re.sub(r"[ \t]{2,}", " ", text)
+    text = re.sub(r"\n{3,}", "\n\n", text)
+    return text
+
+
+def _fetch_jina(url: str, api_key: str, timeout: float) -> str:
+    response = httpx.get(
+        f"{JINA_READER_URL}/{url}",
+        timeout=timeout,
+        headers={"Authorization": f"Bearer {api_key}", "X-Timeout": str(int(timeout))},
+    )
+    response.raise_for_status()
+    return response.text.strip()
+
+
+def _fetch_direct(url: str, timeout: float) -> str:
+    response = httpx.get(
+        url,
+        timeout=timeout,
+        follow_redirects=True,
+        headers={"User-Agent": "Mozilla/5.0"},
+    )
+    response.raise_for_status()
+    content_type = (response.headers.get("content-type") or "").lower()
+    body = response.content
+    if body.startswith(b"%PDF-") or "application/pdf" in content_type:
+        return _pdf_to_text(body).strip()
+    if "text/html" in content_type or "<html" in response.text[:2048].lower():
+        return _html_to_text(response.text).strip()
+    return response.text.strip()
+
+
+def open_page(url: str, timeout: float = 30) -> str:
+    """Fetch a URL and return its text content, via Jina Reader when available."""
+    api_key = os.environ.get("JINA_API_KEY", "")
+    if api_key:
+        try:
+            return _fetch_jina(url, api_key, timeout)
+        except httpx.HTTPError:
+            pass  # fall back to direct fetch
+    return _fetch_direct(url, timeout)
+
+
+# Shadows the builtin within this module on purpose: the agent-facing API is `search.open`.
+async def open(url: str, *, timeout: float = 30) -> str:
+    """Fetch a URL and return its text content. Handles HTML and PDF.
+
+    Uses the Jina Reader API (markdown; handles JS-rendered pages) when
+    ``JINA_API_KEY`` is set, falling back to a direct fetch parsed locally.
+
+    Args:
+        url: The URL to fetch.
+        timeout: Request timeout in seconds.
+
+    Returns:
+        The page text.
+    """
+    return await asyncio.to_thread(open_page, url, timeout)

src/rlm/tools/ipython.py

@@ -203,12 +203,16 @@ def _wrap_callable(mod, log_source):
     wrapped = _CallableModule(mod.__name__)
     wrapped.__dict__.update(mod.__dict__)
     if log_source is not None:
-        _original_run = wrapped.run
-        @functools.wraps(_original_run)
-        async def _logged_run(*args, **kwargs):
-            _log_programmatic_call(mod.__name__, log_source)
-            return await _original_run(*args, **kwargs)
-        wrapped.run = _logged_run
+        def _logged(fn):
+            @functools.wraps(fn)
+            async def _call(*args, **kwargs):
+                _log_programmatic_call(mod.__name__, log_source)
+                return await fn(*args, **kwargs)
+            return _call
+        # Log every public async function the skill exposes (`run`, `search.open`, ...).
+        for _attr, _fn in list(wrapped.__dict__.items()):
+            if not _attr.startswith('_') and inspect.iscoroutinefunction(_fn):
+                setattr(wrapped, _attr, _logged(_fn))
     # Mirror run's signature and docstring onto the module so
     # `inspect.signature(<skill>)` and `help(<skill>)` expose the real API
     # surface instead of `_CallableModule.__call__`'s `(*args, **kwargs)`

tests/test_builtin_skills.py

@@ -34,7 +34,7 @@ async def test_edit_missing_file_raises(tmp_path):
 def test_enable_builtin_skills_writes_stub(tmp_path):
     assert "edit" in available_builtin_skills()
     assert enable_builtin_skills(["edit"], tmp_path) == ["edit"]
-    assert (tmp_path / "edit.py").read_text() == "from rlm.skills.edit import run\n"
+    assert (tmp_path / "edit.py").read_text() == "from rlm.skills.edit import *\n"
 
 
 def test_enable_unknown_skill_raises(tmp_path):
@@ -45,7 +45,14 @@ def test_enable_unknown_skill_raises(tmp_path):
 def test_search_enable_writes_stub(tmp_path):
     assert "search" in available_builtin_skills()
     assert enable_builtin_skills(["search"], tmp_path) == ["search"]
-    assert (tmp_path / "search.py").read_text() == "from rlm.skills.search import run\n"
+    assert (tmp_path / "search.py").read_text() == "from rlm.skills.search import *\n"
+
+
+def test_search_exports_run_and_open():
+    import rlm.skills.search as search_skill
+
+    assert search_skill.__all__ == ["run", "open"]
+    assert callable(search_skill.open)
 
 
 async def test_search_missing_api_key_returns_error(monkeypatch):
@@ -54,6 +61,30 @@ async def test_search_missing_api_key_returns_error(monkeypatch):
     assert "SERPER_API_KEY" in result
 
 
+async def test_search_requests_num_results(monkeypatch):
+    import rlm.skills.search as search_skill
+
+    captured = {}
+
+    class FakeResponse:
+        def raise_for_status(self):
+            pass
+
+        def json(self):
+            return {"organic": [{"title": "t", "link": "https://x", "snippet": "s"}]}
+
+    def fake_post(url, json=None, headers=None, timeout=None):
+        captured["json"] = json
+        return FakeResponse()
+
+    monkeypatch.setenv("SERPER_API_KEY", "test-key")
+    monkeypatch.setattr(search_skill.httpx, "post", fake_post)
+
+    out = await run_search(query="one")
+    assert captured["json"] == {"q": "one", "num": 10}
+    assert out.startswith("Result 1: t")
+
+
 def test_search_format_results():
     results = [
         {"title": "First", "link": "https://a", "snippet": "snippet one"},