V2 media galleries, a media viewer and a richer drawer Media section (0.261.262)

[Paul Lizer (paullizer)]

Summary

Agents that pull evidence from remote services answer with many images and clips: a dozen stills and a couple of clips in one reply, each under a one-line caption, or a timeline where every entry links a clip and shows a still. Until now each image rendered as a full-width card and each clip as a full-width player, so one such reply filled several screens. The Documents drawer listed clips and recordings as plain text rows.

This PR lays reply media out as compact tiles, adds one viewer for all of it, and rebuilds the drawer's Media section.

Galleries in replies

• Two or more images or clips in a row show as a gallery of tiles, three to a row, each with its caption underneath. A short line of text directly above an image or clip, in the same paragraph, becomes its caption, and a colon-ended label over a single image captions it without the colon.
• In a list, the clip and still at the end of each item sit side by side under the item's text, so a timeline reads as text plus a compact row per entry.
• A clip tile shows its first frame, its length and a play mark, and plays in place. View larger continues it in the viewer from the same point.
• A single image keeps its full-width card. Recordings keep their player bar, next to the text about them.
• Grouping is a rehype plugin (lib/mediaGallery.ts) over the parsed tree, so paragraphs, line breaks and list items come from the parser rather than a text scanner.

Media viewer

• Opening a tile shows the image or clip large. Previous and Next buttons and the arrow keys step through the gallery. The header shows the position, such as "3 of 15".
• The viewer also offers View at actual size, Download and Open in a new tab. From the drawer it adds Show in conversation, which closes the viewer and scrolls to the message.
• It is built on the shared Modal, so Escape, Tab trapping and focus returning to the tile behave like every other dialog.

Drawer Media section

• Media is grouped as Images, Videos and Audio, with counts. Clips are tiles with their first frame instead of text rows.
• Images and clips open one viewer that steps through all of them.
• Recordings play in the drawer: play, stop, seek, volume, speed, Download and Show in conversation.

Downloads

• Recordings, clips and images save under the file name in their link.
• When a host does not let the page read the file (no CORS headers), the file opens in a new tab instead, where the browser can save it, and a toast explains why. If the browser blocks that tab, the toast offers Open.
• An expired link reports that it has expired instead of opening a tab that would show the same error.

Bug fix

• Pressing Escape in a dialog opened from the conversation drawer (a document preview, the media viewer) also closed the drawer. Escape now closes only the dialog.

Security

• No raw HTML. The plugin only moves the img and a elements markdown already produced into div > figure > figcaption wrappers. react-markdown's URL transform still runs on every image and link afterwards; a Node test proves this with a custom transform.
• Every media element and link in the new components reads its address through safeMediaUrl or resolveImageSource, which test_v2_media_gallery_wiring.py asserts. New-tab links use rel="noopener noreferrer", and window.open severs the opener.
• Downloads send the session only to the app. App image endpoints use the client's credentials mode, and every other host gets credentials: 'omit'.
• Clip tiles request their first frame with a #t=0.1 media fragment, which never reaches the server, so signed links are untouched. Tiles load only once on screen.
• No new npm, CDN or Python dependency, and the CSP is unchanged.

Validation

• node functional_tests/test_v2_media_gallery_logic.mjs: 16 checks pass. They run the real plugin inside the real react-markdown pipeline over:
  • caption pairs, alternating caption and media lines, image paragraphs, and colon labels
  • evidence lists, tight lists, and mid-sentence clips
  • unsafe and data: sources, block quotes, tables, and placeholders in captions
  • the URL transform, file naming, and the still-frame address
• pytest passes 104 functional tests: the new test_v2_media_gallery_wiring.py, the updated test_v2_inline_media_players.py and test_v2_rich_rendering.py, and the renderer, citation, masking, lightbox and image editor tests that read the same files.
• ui_tests/test_v2_media_galleries.py passes 5 tests in Chromium with production CSS:
  • the three-column layout and captions
  • evidence-list rows
  • viewer navigation by buttons and keys, with focus returning to the tile
  • a clip playing in its tile and opening larger
  • the drawer groups, with a recording playing and downloading, Show in conversation, and Escape closing only the viewer
  • the new-tab fallback, and an expired link
• The tests play a WebM recorded by the browser under test and a WAV built by the test, so no media files are committed. ui_tests/test_v2_collaboration_ux.py (4) and the image upload and generated image UI tests also pass.
• Docs: app surface coverage passes 7/7 and site quality 6/6. The inventory is unchanged.
• tsc -b --noEmit is clean.
• Checked locally against the replies of a real long conversation, without committing any of it. Fifteen-item evidence replies form one gallery, timeline lists form a clip-and-still row per entry, and three captioned document images form one gallery.

Pre-existing failures, identical on the base before this change:

• functional_tests/test_v2_inline_image_proposals.py::test_no_third_party_browser_assets_were_added reports @xyflow/react as an added dependency. It was added by 31e7d16; this PR does not touch package.json.
• Seven tests in ui_tests/test_v2_orchestration_drawer_views.py and ui_tests/test_v2_orchestration_elicitation.py fail with 'NoneType' object has no attribute 'evaluate'. They fail the same way on c269964.

Docs

• New: docs/explanation/features/V2_MEDIA_GALLERIES_AND_VIEWER.md
• Updated: docs/reference/chat-controls.md (new "Media in replies" section, drawer Media row), V2_INLINE_MEDIA_AND_AGENT_MESSAGES.md, V2_COLLABORATION_UX.md, the features index, and the release notes

Version

0.261.262. React V2 is at 0.261.261 (#1663). Renumber on merge if the order changes.

Not in this PR

Ideas for follow-ups:

• a filmstrip of thumbnails in the viewer
• moving between galleries from different replies in one viewer session
• filter chips in the drawer
• refreshing an expired signed link from the action that produced it, which needs server support