Skip to content

feat: site-affordance layer (site_describe / site_intents / site_open) - #510

Open
Robin Sayar (r-sayar) wants to merge 2 commits into
browseros-ai:mainfrom
r-sayar:feat/site-affordances
Open

Robin Sayar (r-sayar) wants to merge 2 commits into
browseros-ai:mainfrom
r-sayar:feat/site-affordances

Conversation

@r-sayar

Copy link
Copy Markdown

Why

The base BrowserOS tool surface (take_snapshot, click, fill, navigate_page, evaluate_script, …) is low-level. An agent using only those has to (1) discover what's possible on each site, and (2) re-derive the click sequence every time. Both are slow and flaky.

This PR adds a discovery layer on top: tell the agent "you're on Gmail — call gmail_compose, not take_snapshot+click+fill" so it lands on a cached path instead of re-deriving the click sequence every turn.

What

Three new tools and a small registry — all under apps/server/src/affordances/ and apps/server/src/tools/affordances.ts:

Tool What it does
site_describe Given a page id, returns the matching recipe — intents, URL-aware contextual hints, method classification (mcp/api/script). Falls back gracefully for unknown sites.
site_intents Returns the full registry — useful for planning multi-site tasks.
site_open Open a known site by id (e.g. wikipedia). Returns page id + intents so the agent can act immediately.

Recipes are pure metadata — they don't execute anything. The actual deterministic work happens in whatever site-MCP / API / script each recipe points at via its intents.

method classification

Each recipe declares how it's implemented so reviewers (and the agent) see the cost / reliability / auth profile up front:

  • mcp — built on top of an existing first/third-party MCP server. Cheapest.
  • api — calls the site's official HTTP API. Needs an API key / OAuth.
  • script — drives a live authenticated browser tab via BrowserOS primitives. Most flexible, most brittle.

Surfaces in every tool's output.

Built-in starter recipes

Two universal, no-auth examples ship in affordances/recipes.ts:

  • wikipedia — search + article extraction via get_page_content
  • pubmed — search + abstract extraction; notes the NCBI E-utilities API path if method: 'api' is preferred

Both point at existing BrowserOS primitives so they work without any third-party MCP server.

Non-goals

  • Not an execution engine. Recipes don't run JavaScript or open tabs themselves.
  • Not a router. The agent still picks which tool to call.
  • Not auth-aware. Recipes assume the calling agent has handled login.

Reference implementation

A larger Python implementation covering 15 sites (Gmail, Outlook, Canvas, Claude.ai, Notion, LinkedIn, YouTube, …) lives at r-sayar/cloud-browser-mcp — each site has its own stdio MCP server driving a logged-in BrowserOS tab. This PR ports the abstraction layer to TypeScript and ships two universal starter recipes; if you'd rather not bake recipes into core, happy to move them to an examples/ dir.

Verification

  • bun run typecheck on @browseros/server exits clean.
  • bun run lint:fix (Biome) clean (one import-order fix it auto-applied).
  • 547-line diff is entirely additive — no behaviour changes to existing tools.

The full-monorepo typecheck OOMs on @browseros/agent at default Node heap — unrelated to these changes; bumping NODE_OPTIONS=--max-old-space-size=8192 resolves it. Happy to land that as a separate PR if useful.

Open questions for reviewers

  1. Should the registry singleton live in apps/server/src/tools/affordances.ts or somewhere more configurable (env var, config file)?
  2. Want built-in starter recipes in core, or should they move to examples/ so the abstraction stays neutral?
  3. Should this be opt-in via env / config flag rather than always-on?

Happy to iterate.

🤖 Generated with Claude Code

Adds a discovery surface for site-aware cached recipes on top of BrowserOS's
primitives. Instead of "snapshot → find compose button → click → wait →
snapshot → find To field → fill → ..." every time, the agent calls
`site_describe(page)`, sees that Gmail's compose URL params skip the click
chain, and uses `navigate_page` with the right URL.

## What's added

- `apps/server/src/affordances/types.ts` — Recipe + Intent + Method types,
  zod schemas, and a URL matcher (host substrings; path-prefix entries
  match against host+path to avoid auth-redirect false positives).
- `apps/server/src/affordances/registry.ts` — `RecipeRegistry` class with
  register / findByUrl / findById / all.
- `apps/server/src/affordances/recipes.ts` — two built-in starter recipes
  (wikipedia, pubmed) that point at existing BrowserOS primitives, so
  they work out of the box with no third-party MCP needed.
- `apps/server/src/tools/affordances.ts` — three new tools:
  - `site_describe(page)` — match URL → return recipe metadata
  - `site_intents()` — list the full registry
  - `site_open(site_id)` — open a known site by id
- `apps/server/src/tools/registry.ts` — registers the three tools.
- `docs/affordances.md` — design doc.

## Method classification

Each recipe declares `method: "mcp" | "api" | "script"` so the agent sees
the cost / reliability / auth profile up front:

- `mcp`    — built on top of an existing MCP server. Cheapest.
- `api`    — calls the site's official HTTP API. Reliable.
- `script` — DOM/JS via BrowserOS primitives. Most flexible, most brittle.

The classification surfaces in every tool output.

## Non-goals

- Not an execution engine. Recipes don't run JS or open tabs themselves —
  that's `navigate_page` / `evaluate_script` / your site-MCP.
- Not a router. The agent still picks which tool to call.
- Not auth-aware. Recipes assume login has been handled.

## Reference implementation

A larger Python implementation covering 15 sites (Gmail, Outlook, Canvas,
Claude.ai, Notion, LinkedIn, YouTube, …) lives at
https://github.com/r-sayar/cloud-browser-mcp — each site has its own
stdio MCP server driving a logged-in BrowserOS tab. This PR ports the
abstraction layer to TypeScript and ships two universal starter recipes.

## Verification

- `bun run typecheck` on `@browseros/server` exits clean.
- `bun run lint:fix` (Biome) clean.
- The full-monorepo typecheck OOMs on `@browseros/agent` (the UI extension
  package) at default Node heap — unrelated to these changes; bumping
  `NODE_OPTIONS=--max-old-space-size=8192` lets it pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented May 11, 2026

Copy link
Copy Markdown
Contributor

PR author is not in the allowed authors list.

@github-actions

github-actions Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA. Thank you!
Posted by the CLA Assistant Lite bot.

@r-sayar

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

Optimisations (all validated on Hetzner BrowserOS — 61-79% speed gain):

1. KV-cache prompt ordering: static sections before dynamic ones so the
   cacheable prefix is maximised across sessions.
2. script-over-click prompt section: agent prefers evaluate_script for
   DOM-accessible ops (form.submit, .click(), reading values) over the
   slower snapshot → identify → click chain.
3. take_screenshot demotion: description now explicitly directs agent to
   take_snapshot for interaction and get_page_content for extraction.
4. evaluate_script promotion: description updated to make it a first-choice
   tool for DOM-accessible operations.
5. fill/clear no longer auto-include snapshots: saves one snapshot per field
   in multi-field forms (email compose = 3 fewer round-trips).
6. press_key adds auto-snapshot: Enter to submit was previously blind.
7. get_page_content viewportOnly defaults to true: extraction stays lean.
8. action-cache prompt strengthened with the 3-5x speedup framing.

Benchmarks (apps/server/tests/benchmarks/):
- framework.ts: BenchmarkMetrics with char-based token estimates
  (chars/4 for text, +1000 per screenshot image)
- scenarios.ts: Gmail compose, Doctolib slot search, Discord type
- v1-baseline.test.ts / v2-optimized.test.ts: bun:test suites
- run-on-remote.ts: standalone runner for live servers
  (ANTHROPIC_API_KEY / OPENAI_API_KEY bypasses free-tier quota)

Hetzner results (pre-quota-exhaustion):
  Gmail compose:  442s → 172s  (-61%)  37 → 21 steps  (-43%)
  Doctolib:       336s → 70s   (-79%)  33 → 5 steps   (-85%)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant