Repository navigation
feat: site-affordance layer (site_describe / site_intents / site_open) - #510
Open
Robin Sayar (r-sayar) wants to merge 2 commits into
Open
Robin Sayar (r-sayar) wants to merge 2 commits into
Robin Sayar (r-sayar) wants to merge 2 commits into
Conversation
Adds a discovery surface for site-aware cached recipes on top of BrowserOS's primitives. Instead of "snapshot → find compose button → click → wait → snapshot → find To field → fill → ..." every time, the agent calls `site_describe(page)`, sees that Gmail's compose URL params skip the click chain, and uses `navigate_page` with the right URL. ## What's added - `apps/server/src/affordances/types.ts` — Recipe + Intent + Method types, zod schemas, and a URL matcher (host substrings; path-prefix entries match against host+path to avoid auth-redirect false positives). - `apps/server/src/affordances/registry.ts` — `RecipeRegistry` class with register / findByUrl / findById / all. - `apps/server/src/affordances/recipes.ts` — two built-in starter recipes (wikipedia, pubmed) that point at existing BrowserOS primitives, so they work out of the box with no third-party MCP needed. - `apps/server/src/tools/affordances.ts` — three new tools: - `site_describe(page)` — match URL → return recipe metadata - `site_intents()` — list the full registry - `site_open(site_id)` — open a known site by id - `apps/server/src/tools/registry.ts` — registers the three tools. - `docs/affordances.md` — design doc. ## Method classification Each recipe declares `method: "mcp" | "api" | "script"` so the agent sees the cost / reliability / auth profile up front: - `mcp` — built on top of an existing MCP server. Cheapest. - `api` — calls the site's official HTTP API. Reliable. - `script` — DOM/JS via BrowserOS primitives. Most flexible, most brittle. The classification surfaces in every tool output. ## Non-goals - Not an execution engine. Recipes don't run JS or open tabs themselves — that's `navigate_page` / `evaluate_script` / your site-MCP. - Not a router. The agent still picks which tool to call. - Not auth-aware. Recipes assume login has been handled. ## Reference implementation A larger Python implementation covering 15 sites (Gmail, Outlook, Canvas, Claude.ai, Notion, LinkedIn, YouTube, …) lives at https://github.com/r-sayar/cloud-browser-mcp — each site has its own stdio MCP server driving a logged-in BrowserOS tab. This PR ports the abstraction layer to TypeScript and ships two universal starter recipes. ## Verification - `bun run typecheck` on `@browseros/server` exits clean. - `bun run lint:fix` (Biome) clean. - The full-monorepo typecheck OOMs on `@browseros/agent` (the UI extension package) at default Node heap — unrelated to these changes; bumping `NODE_OPTIONS=--max-old-space-size=8192` lets it pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
PR author is not in the allowed authors list. |
Contributor
|
All contributors have signed the CLA. Thank you! |
Author
|
I have read the CLA Document and I hereby sign the CLA |
Optimisations (all validated on Hetzner BrowserOS — 61-79% speed gain): 1. KV-cache prompt ordering: static sections before dynamic ones so the cacheable prefix is maximised across sessions. 2. script-over-click prompt section: agent prefers evaluate_script for DOM-accessible ops (form.submit, .click(), reading values) over the slower snapshot → identify → click chain. 3. take_screenshot demotion: description now explicitly directs agent to take_snapshot for interaction and get_page_content for extraction. 4. evaluate_script promotion: description updated to make it a first-choice tool for DOM-accessible operations. 5. fill/clear no longer auto-include snapshots: saves one snapshot per field in multi-field forms (email compose = 3 fewer round-trips). 6. press_key adds auto-snapshot: Enter to submit was previously blind. 7. get_page_content viewportOnly defaults to true: extraction stays lean. 8. action-cache prompt strengthened with the 3-5x speedup framing. Benchmarks (apps/server/tests/benchmarks/): - framework.ts: BenchmarkMetrics with char-based token estimates (chars/4 for text, +1000 per screenshot image) - scenarios.ts: Gmail compose, Doctolib slot search, Discord type - v1-baseline.test.ts / v2-optimized.test.ts: bun:test suites - run-on-remote.ts: standalone runner for live servers (ANTHROPIC_API_KEY / OPENAI_API_KEY bypasses free-tier quota) Hetzner results (pre-quota-exhaustion): Gmail compose: 442s → 172s (-61%) 37 → 21 steps (-43%) Doctolib: 336s → 70s (-79%) 33 → 5 steps (-85%) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The base BrowserOS tool surface (
take_snapshot,click,fill,navigate_page,evaluate_script, …) is low-level. An agent using only those has to (1) discover what's possible on each site, and (2) re-derive the click sequence every time. Both are slow and flaky.This PR adds a discovery layer on top: tell the agent "you're on Gmail — call
gmail_compose, nottake_snapshot+click+fill" so it lands on a cached path instead of re-deriving the click sequence every turn.What
Three new tools and a small registry — all under
apps/server/src/affordances/andapps/server/src/tools/affordances.ts:site_describesite_intentssite_openwikipedia). Returns page id + intents so the agent can act immediately.Recipes are pure metadata — they don't execute anything. The actual deterministic work happens in whatever site-MCP / API / script each recipe points at via its
intents.methodclassificationEach recipe declares how it's implemented so reviewers (and the agent) see the cost / reliability / auth profile up front:
mcp— built on top of an existing first/third-party MCP server. Cheapest.api— calls the site's official HTTP API. Needs an API key / OAuth.script— drives a live authenticated browser tab via BrowserOS primitives. Most flexible, most brittle.Surfaces in every tool's output.
Built-in starter recipes
Two universal, no-auth examples ship in
affordances/recipes.ts:wikipedia— search + article extraction viaget_page_contentpubmed— search + abstract extraction; notes the NCBI E-utilities API path ifmethod: 'api'is preferredBoth point at existing BrowserOS primitives so they work without any third-party MCP server.
Non-goals
Reference implementation
A larger Python implementation covering 15 sites (Gmail, Outlook, Canvas, Claude.ai, Notion, LinkedIn, YouTube, …) lives at r-sayar/cloud-browser-mcp — each site has its own stdio MCP server driving a logged-in BrowserOS tab. This PR ports the abstraction layer to TypeScript and ships two universal starter recipes; if you'd rather not bake recipes into core, happy to move them to an
examples/dir.Verification
bun run typecheckon@browseros/serverexits clean.bun run lint:fix(Biome) clean (one import-order fix it auto-applied).The full-monorepo typecheck OOMs on
@browseros/agentat default Node heap — unrelated to these changes; bumpingNODE_OPTIONS=--max-old-space-size=8192resolves it. Happy to land that as a separate PR if useful.Open questions for reviewers
apps/server/src/tools/affordances.tsor somewhere more configurable (env var, config file)?examples/so the abstraction stays neutral?Happy to iterate.
🤖 Generated with Claude Code