Skip to content

Commit e4942ee

Browse files
committed
Add a live camera view on station start; close the gripper when rehoming
Two independent changes. Each can be reverted on its own by undoing the files listed under it. 1. Live camera view on `rua serve` Why: whoever watches the robot (a developer debugging an app, or a person following an agent's changes) needs to see what the wrist camera sees in real time. - hebbian_rua/live_view.py (new): waits up to 120 s for the media service (default http://127.0.0.1:8781) to list an available camera via GET /cameras, writes a self-contained HTML page to $TMPDIR/rua-live-view.html with the media URL and camera names baked in (macOS drops query strings from file URLs), and opens it with webbrowser.open. The page shows one <img> per camera that re-requests GET /cameras/<name>?t=<now> about every 33 ms, shows a frame rate, and retries every 0.5 s while the station is down. No LiveKit, no new dependencies, no long-running process. It logs "rua live view: opened <url>" (or why not) to stderr. Run it manually with `python -m hebbian_rua.live_view [media-url]`. - hebbian_rua/cli.py: for `rua serve`, starts that module in a detached subprocess before exec'ing the Rust server, using `--media-address` (0.0.0.0 / [::] are mapped to 127.0.0.1). New flag `--no-view` skips it; the launcher removes the flag before exec, so the Rust CLI never sees it. - tests/test_live_view.py (new): media URL parsing from serve arguments. - docs/cameras.md: new "Live view" section. Revert: delete live_view.py and the test, restore cli.py to a plain os.execv of sys.argv[1:], and remove the docs section. 2. Rehome closes the gripper by default Why: so the arm always comes back to the same resting state (jaws closed), which also keeps the wrist camera's view consistent between runs. - hebbian_robot_use_client/.../station_client.py: LeasedArm.rehome() gains `close_gripper: bool = True`. When true it first sends set_gripper(0.0) with the caller's expected_revision; if that does not complete, it returns that result without moving. It then moves the joints to the rest pose as before. The close happens first, while the jaws are clear of the table; closing stops on contact, so a held object stays held. `rehome(close_gripper=False)` restores the old behaviour. `rest()` (alias) inherits the new default. - The model-facing `rehome` tool description changed from "...This moves the arm without changing the gripper..." to "Close the gripper, then return this arm to the joint pose recorded when it was enabled. Closing stops on contact, so a held object stays held; ensure the path and workspace are clear." The same string is edited in contracts/src/robot_tools.rs, schemas/robot-tools.json and hebbian_robot_use_client/.../llm/generated_robot_tools.py (contracts/tests/robot_catalog.rs checks they match). - Affects every rehome: the example apps (inspect-blocks, image-qa) and LLM agents calling the rehome tool. Revert: remove the close_gripper parameter and its block, and restore the old description string in all three files. Verified: ruff, ty, pytest (5 passed), cargo test -p hebbian-contracts --test robot_catalog. On a physical SO-101 on macOS: the live view opened on `rua serve` and showed an Image Q&A run, and rehome() from a half-open gripper (0.49) closed it to 0.0, then returned to rest.
1 parent c40797d commit e4942ee

8 files changed

Lines changed: 185 additions & 6 deletions

File tree

‎contracts/src/robot_tools.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,7 @@ pub fn robot_tool_catalog() -> Vec<RobotToolDefinition> {
174174
RobotToolDefinition {
175175
name: "rehome",
176176
operation: RobotToolOperation::Rehome,
177-
description: "Return this arm to the joint pose recorded when it was enabled. This moves the arm without changing the gripper; ensure the path and workspace are clear.".into(),
177+
description: "Close the gripper, then return this arm to the joint pose recorded when it was enabled. Closing stops on contact, so a held object stays held; ensure the path and workspace are clear.".into(),
178178
parameters: model_arguments::<StopArguments>(),
179179
},
180180
RobotToolDefinition {

‎docs/cameras.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,15 @@ from your uv project with a USB camera:
1212
uv run rua serve --profile ./robot.toml --camera table=0
1313
```
1414

15+
## Live view
16+
17+
`rua serve` opens a browser window with a live feed from every camera once
18+
the media service has one, so whoever watches the robot sees what its cameras
19+
see. It needs no LiveKit: the page polls each camera's newest frame from the
20+
media service, shows the frame rate, and keeps retrying while the station
21+
restarts. Pass `--no-view` to skip it, for example on a headless machine. To open
22+
it again, run `uv run python -m hebbian_rua.live_view`.
23+
1524
## Configure LiveKit
1625

1726
Create your own [LiveKit project](https://cloud.livekit.io) and copy its URL,

‎hebbian_robot_use_client/hebbian_robot_use_client/llm/generated_robot_tools.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -591,7 +591,7 @@
591591
}
592592
},
593593
{
594-
"description": "Return this arm to the joint pose recorded when it was enabled. This moves the arm without changing the gripper; ensure the path and workspace are clear.",
594+
"description": "Close the gripper, then return this arm to the joint pose recorded when it was enabled. Closing stops on contact, so a held object stays held; ensure the path and workspace are clear.",
595595
"name": "rehome",
596596
"operation": "rehome",
597597
"parameters": {

‎hebbian_robot_use_client/hebbian_robot_use_client/station_client.py‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,8 +193,15 @@ async def move_to_pose(
193193
expected_revision=expected_revision,
194194
)
195195

196-
async def rehome(self, *, expected_revision: int | None = None) -> CommandResult:
197-
"""Return to the pose the arm was enabled at."""
196+
async def rehome(
197+
self, *, close_gripper: bool = True, expected_revision: int | None = None
198+
) -> CommandResult:
199+
"""Return to the pose the arm was enabled at, by default closing the gripper first.
200+
201+
The jaws close before the move, while they are clear of the table; closing
202+
stops on contact, so a held object stays held. Pass `close_gripper=False`
203+
to leave the gripper as it is.
204+
"""
198205
rest_pose = (await self._client.arm(self.name).state()).rest_pose
199206
if rest_pose is None:
200207
return CommandResult.of(
@@ -203,6 +210,10 @@ async def rehome(self, *, expected_revision: int | None = None) -> CommandResult
203210
detail=f"{self.name} has no rest pose; enable it first",
204211
)
205212
)
213+
if close_gripper:
214+
closed = await self.set_gripper(0.0, expected_revision=expected_revision)
215+
if not closed.completed:
216+
return closed
206217
return await self.move_to_joints(
207218
list(rest_pose), expected_revision=expected_revision
208219
)

‎hebbian_rua/cli.py‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,13 @@
22

33
import os
44
import shutil
5+
import subprocess
56
import sys
67
import sysconfig
78
from pathlib import Path
89

10+
from hebbian_rua.live_view import media_url_from_serve_arguments
11+
912

1013
def main() -> None:
1114
executable_name = "hebbian-robot-use-server"
@@ -28,4 +31,19 @@ def main() -> None:
2831
raise SystemExit(
2932
"RUA server binary is missing. Install a hebbian-rua wheel for your platform, or build the server from source."
3033
)
31-
os.execv(str(executable), ["rua", *sys.argv[1:]])
34+
arguments = sys.argv[1:]
35+
if arguments[:1] == ["serve"]:
36+
if "--no-view" in arguments:
37+
arguments.remove("--no-view")
38+
else:
39+
start_live_view(media_url_from_serve_arguments(arguments))
40+
os.execv(str(executable), ["rua", *arguments])
41+
42+
43+
def start_live_view(media_url: str) -> None:
44+
"""Open the camera window from a separate process, since exec replaces this one."""
45+
subprocess.Popen(
46+
[sys.executable, "-m", "hebbian_rua.live_view", media_url],
47+
stdin=subprocess.DEVNULL,
48+
start_new_session=True, # Ctrl-C on the station leaves the page opener alone
49+
)

‎hebbian_rua/live_view.py‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
"""Open a live camera window in the browser once the station's media service is up.
2+
3+
`rua serve` starts this in the background so whoever is watching the robot can see
4+
what its cameras see. Run it yourself with `python -m hebbian_rua.live_view
5+
[media-url]`. The browser fetches each camera's newest JPEG from the media service
6+
directly, so nothing here keeps running once the page is open.
7+
"""
8+
9+
import json
10+
import sys
11+
import tempfile
12+
import time
13+
import urllib.request
14+
import webbrowser
15+
from pathlib import Path
16+
17+
DEFAULT_MEDIA_URL = "http://127.0.0.1:8781"
18+
# A simulator may connect its cameras well after the station starts.
19+
CAMERA_WAIT_SECONDS = 120.0
20+
21+
PAGE = """<!doctype html>
22+
<meta charset="utf-8">
23+
<title>RUA live view</title>
24+
<style>
25+
body { margin: 0; background: #111; color: #ddd; font: 14px system-ui, sans-serif; }
26+
main { display: flex; flex-wrap: wrap; gap: 8px; padding: 8px; }
27+
figure { margin: 0; flex: 1 1 480px; }
28+
img { width: 100%; background: #000; display: block; }
29+
figcaption { padding: 4px 0; }
30+
</style>
31+
<main></main>
32+
<script>
33+
const MEDIA = __MEDIA__;
34+
const CAMERAS = __CAMERAS__;
35+
const FRAME_INTERVAL_MS = 33; // ~30 fps; the media service always serves its newest frame
36+
for (const name of CAMERAS) {
37+
const figure = document.createElement("figure");
38+
const image = document.createElement("img");
39+
const caption = document.createElement("figcaption");
40+
figure.append(image, caption);
41+
document.querySelector("main").append(figure);
42+
let frames = 0;
43+
const next = () => {
44+
image.src = `${MEDIA}/cameras/${encodeURIComponent(name)}?t=${Date.now()}`;
45+
};
46+
image.onload = () => { frames += 1; setTimeout(next, FRAME_INTERVAL_MS); };
47+
// The station may be restarting; keep the window and retry.
48+
image.onerror = () => { caption.textContent = `${name}: waiting for the station`; setTimeout(next, 500); };
49+
setInterval(() => {
50+
if (frames > 0) caption.textContent = `${name}: ${frames} fps`;
51+
frames = 0;
52+
}, 1000);
53+
next();
54+
}
55+
</script>
56+
"""
57+
58+
59+
def available_cameras(media_url: str) -> list[str]:
60+
"""The cameras the media service can serve now; empty while it is starting."""
61+
try:
62+
with urllib.request.urlopen(f"{media_url}/cameras", timeout=2) as response:
63+
listing = json.load(response)
64+
except (OSError, ValueError):
65+
return []
66+
return [camera["name"] for camera in listing if camera.get("available")]
67+
68+
69+
def open_live_view(
70+
media_url: str = DEFAULT_MEDIA_URL, wait_seconds: float = CAMERA_WAIT_SECONDS
71+
) -> bool:
72+
"""Wait for at least one camera, then open the live page; False if none appeared."""
73+
deadline = time.monotonic() + wait_seconds
74+
while not (cameras := available_cameras(media_url)):
75+
if time.monotonic() > deadline:
76+
return False
77+
time.sleep(0.5)
78+
page = Path(tempfile.gettempdir()) / "rua-live-view.html"
79+
# Values go into the page itself: macOS drops query strings from file URLs.
80+
page.write_text(
81+
PAGE.replace("__MEDIA__", json.dumps(media_url)).replace(
82+
"__CAMERAS__", json.dumps(cameras)
83+
)
84+
)
85+
return webbrowser.open(page.as_uri())
86+
87+
88+
def media_url_from_serve_arguments(arguments: list[str]) -> str:
89+
"""The media URL for `rua serve`'s `--media-address`, reachable from this machine."""
90+
address = "127.0.0.1:8781"
91+
for index, argument in enumerate(arguments):
92+
if argument == "--media-address" and index + 1 < len(arguments):
93+
address = arguments[index + 1]
94+
elif argument.startswith("--media-address="):
95+
address = argument.split("=", 1)[1]
96+
host, _, port = address.rpartition(":")
97+
if host in ("0.0.0.0", "[::]", ""): # bound to every interface
98+
host = "127.0.0.1"
99+
return f"http://{host}:{port}"
100+
101+
102+
def main() -> None:
103+
media_url = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_MEDIA_URL
104+
page = Path(tempfile.gettempdir()) / "rua-live-view.html"
105+
if open_live_view(media_url):
106+
print(f"rua live view: opened {page.as_uri()}", file=sys.stderr)
107+
else:
108+
print(
109+
f"rua live view: no camera at {media_url} or no browser; open {page.as_uri()}",
110+
file=sys.stderr,
111+
)
112+
113+
114+
if __name__ == "__main__":
115+
main()

‎schemas/robot-tools.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -586,7 +586,7 @@
586586
}
587587
},
588588
{
589-
"description": "Return this arm to the joint pose recorded when it was enabled. This moves the arm without changing the gripper; ensure the path and workspace are clear.",
589+
"description": "Close the gripper, then return this arm to the joint pose recorded when it was enabled. Closing stops on contact, so a held object stays held; ensure the path and workspace are clear.",
590590
"name": "rehome",
591591
"operation": "rehome",
592592
"parameters": {

‎tests/test_live_view.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
from hebbian_rua.live_view import media_url_from_serve_arguments
2+
3+
4+
def test_media_url_defaults_to_the_local_media_service() -> None:
5+
assert (
6+
media_url_from_serve_arguments(["serve", "--camera", "table=0"])
7+
== "http://127.0.0.1:8781"
8+
)
9+
10+
11+
def test_media_url_follows_media_address_in_either_form() -> None:
12+
assert (
13+
media_url_from_serve_arguments(["serve", "--media-address", "10.0.0.5:9000"])
14+
== "http://10.0.0.5:9000"
15+
)
16+
assert (
17+
media_url_from_serve_arguments(["serve", "--media-address=10.0.0.5:9000"])
18+
== "http://10.0.0.5:9000"
19+
)
20+
21+
22+
def test_media_url_bound_to_every_interface_is_opened_locally() -> None:
23+
assert (
24+
media_url_from_serve_arguments(["serve", "--media-address", "0.0.0.0:8781"])
25+
== "http://127.0.0.1:8781"
26+
)

0 commit comments

Comments
 (0)