Repository navigation
Improve Ink conversations and add touch-friendly Note scrolling - #291
Merged
Merged
Conversation
…back Treat Ink as mode-independent input, prefer permitted Canvas results, and preserve readable interpretation records across live and historical messages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a selection-to-reading tap cycle and persistent reading scrollbar for overflowing fixed-height Notes, preserving normal mouse behavior and cancelling pointer ownership on Canvas teardown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Liangwei Wang (LiangweiOIO)
October 10, 2026 09:12
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The review found unresolved first-pen routing, Ink presentation ordering, and contradictory Operate prompt guidance.
3 open findings
What changed in this PR
This PR improves Ink interpretation history and adds touch/pen scrolling for overflowing fixed-height Notes.
Changes:
- Adds validated, durable Ink interpretation records across live streams, history, search, and UI.
- Preserves Ask/Operate/external conversation modes for Ink submissions.
- Adds a three-state touch/pen Note reading cycle with gesture arbitration and scroll feedback.
| File | Description |
|---|---|
packages/shared/src/utils/ink-interpretation.ts |
Adds interpretation parsing and settlement. |
packages/shared/src/utils/ink-interpretation.test.ts |
Tests interpretation compatibility and validation. |
packages/shared/src/types/api/agent.ts |
Defines Ink report and delivery schemas. |
packages/shared/src/types/agent/chat.ts |
Adds interpretations to chat history. |
packages/shared/package.json |
Exports interpretation utilities. |
docs/proposals/touch-note-reading.md |
Documents the Note-reading trial. |
docs/proposals/ink-lasso-question-flow.md |
Updates Ink reporting and mode semantics. |
docs/backlog/touch-and-pen-usage-pain-points.md |
Records touch/pen usability issues. |
docs/architecture/sketch-node.md |
Documents mode-independent Ink behavior. |
docs/architecture/question-node.md |
Updates interpretation-based title behavior. |
docs/architecture/note-node.md |
Documents touch Note reading. |
docs/architecture/canvas-input-interactions.md |
Defines Note gesture arbitration. |
docs/architecture/agent-reachback.md |
Updates the external Ink report contract. |
docs/architecture/agent-architecture.md |
Documents Ink runtime semantics. |
apps/web/src/store/gesturePreviewStore.ts |
Stores transient Note-reading state. |
apps/web/src/store/chatTypes.ts |
Replaces inferred intent with interpretations. |
apps/web/src/i18n/resources/zh-CN/common.json |
Adds Chinese interaction and interpretation strings. |
apps/web/src/i18n/resources/en/common.json |
Adds English interaction and interpretation strings. |
apps/web/src/hooks/useChatHistory.ts |
Reconciles interpretations during history reconnects. |
apps/web/src/hooks/useChatHistory.test.tsx |
Tests reconnect interpretation settlement. |
apps/web/src/hooks/useCanvasPointerRouter.ts |
Integrates Note pointer ownership and teardown. |
apps/web/src/hooks/agentTurnController.ts |
Tracks live interpretation lifecycle. |
apps/web/src/hooks/agentTurnController.test.ts |
Tests reporting ownership and settlement. |
apps/web/src/handler/pointerRouter.ts |
Cancels active pointers during teardown. |
apps/web/src/handler/pointerRouter.test.ts |
Tests teardown cancellation. |
apps/web/src/handler/canvasPointerRecognizers/noteContent.ts |
Implements Note reading gestures. |
apps/web/src/handler/canvasPointerRecognizers/noteContent.test.ts |
Tests Note gesture behavior. |
apps/web/src/handler/canvasPointerRecognizers/nodeDrag.ts |
Makes node dragging reusable by Note gestures. |
apps/web/src/handler/canvasGestureSession.ts |
Adds the Note-scroll gesture kind. |
apps/web/src/components/Panels/ChatPanel/agentMenu.test.tsx |
Verifies Ask and Operate menu options. |
apps/web/src/components/Panels/Canvas/FloatingToolbars/StrokeSelectionToolbar.tsx |
Preserves modes and improves submission feedback. |
apps/web/src/components/Panels/Canvas/FloatingToolbars/StrokeSelectionToolbar.test.tsx |
Tests mode preservation and progress states. |
apps/web/src/components/Panels/Canvas/FloatingToolbars/InkAgentDestinationPicker.tsx |
Distinguishes busy and unavailable destinations. |
apps/web/src/components/Panels/Canvas/FloatingToolbars/InkAgentDestinationPicker.test.tsx |
Tests destination presentation and selection. |
apps/web/src/components/Nodes/note/noteScrollbar.css |
Separates native and touch scrollbar visibility. |
apps/web/src/components/Nodes/note/noteScroll.ts |
Calculates the touch scrollbar thumb. |
apps/web/src/components/Nodes/note/noteScroll.test.ts |
Tests scrollbar geometry. |
apps/web/src/components/Nodes/note/NoteNode.tsx |
Renders and preserves touch-reading state. |
apps/web/src/components/Nodes/note/NoteContentViewport.tsx |
Exposes Note scrolling state to routing and CSS. |
apps/web/src/components/Messages/UserMessage.tsx |
Renders inline Ink interpretations. |
apps/web/src/components/Messages/UserMessage.test.tsx |
Tests interpretation presentation and localization. |
apps/web/src/components/Messages/MessageList.tsx |
Passes interpretations into user messages. |
apps/web/src/components/Messages/MessageList.test.tsx |
Tests interpretation forwarding. |
apps/web/src/components/Common/Spinner.tsx |
Adds reduced-motion support. |
apps/web/e2e/note-touch-scroll.spec.ts |
Covers touch/pen Note reading end to end. |
apps/web/e2e/ink-user-message.spec.ts |
Covers Ink history presentation. |
apps/web/e2e/ink-toolbar.spec.ts |
Covers Ink modes and submission spinner. |
apps/web/e2e/fixtures/ink-toolbar.tsx |
Extends Ink E2E fixtures. |
apps/server/src/prompt/agents/operate/AGENT.md |
Updates Operate guidance for Ink. |
apps/server/src/prompt/agents/ask/AGENT.md |
Adds reporting-only Ink support to Ask. |
apps/server/src/modules/remote_fs/ink-intent.rfs.test.ts |
Tests the new external report contract. |
apps/server/src/modules/canvas/canvas-search.ts |
Indexes interpretation text and explanations. |
apps/server/src/modules/canvas/canvas-search.test.ts |
Tests interpretation search behavior. |
apps/server/src/modules/agent/tools/handlers/report-ink-intent.ts |
Renames pending Questions from interpretation text. |
apps/server/src/modules/agent/tools/handlers/report-ink-intent.test.ts |
Tests normalized reports and title protection. |
apps/server/src/modules/agent/tools/definitions.ts |
Updates the interpretation tool definition. |
apps/server/src/modules/agent/conversation/transcript/history.ts |
Projects interpretation states into history. |
apps/server/src/modules/agent/conversation/transcript/history.test.ts |
Tests history projection and compatibility. |
apps/server/src/modules/agent/conversation/prompt/ink-intent.ts |
Adds mode-aware Ink steering. |
apps/server/src/modules/agent/conversation/prompt/ink-intent.test.ts |
Tests Ink prompt contracts. |
apps/server/src/modules/agent/agent.service.test.ts |
Tests temporary Ask reporting tools. |
apps/server/src/modules/agent/agent.route.ts |
Marks active history interpretations as pending. |
apps/server/src/modules/agent/agent.route.test.ts |
Tests Ask-mode Ink admission. |
apps/server/src/modules/agent/agent.history-page.route.test.ts |
Tests active versus terminal history states. |
apps/server/src/modules/agent/agent-thread.service.ts |
Carries resolved conversation mode into invocation. |
apps/server/src/modules/agent/agent-thread.service.test.ts |
Tests mode and binding preservation. |
apps/server/src/modules/agent/acp/service.test.ts |
Tests external interpretation host events. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Publish final Ink answers through the host when no substantive Canvas result exists, preserving Agent permissions and conversation linkage. Reuse shared placement and validate persistence, completion ordering, failures, and deduplication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep interpretation summaries adjacent to the pen icon, resolve live input policy before pointer gating and recognition, and leave discussion presentation to the host. Add regression coverage for first-pen Auto routing and preserve explicit input preferences. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Liangwei Wang (LiangweiOIO)
October 10, 2026 10:19
View session
Use canonical node-type dimensions when persisted anchor geometry is missing or zero. Cover realistic auto-height Questions, zero dimensions, and positive sizes with exact placement assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR covers Ink conversation/history improvements and touch/pen reading for overflowing Notes, with a follow-up for direct-summary voice and host-owned Ink answer delivery.
Ink input, interpretation history, and submission feedback
{ text, explanation? }interpretation record. Preserve legacy read compatibility, first successful reporting, title protection, and consistent live/reconnect/history/search projection.Touch and pen Note reading
Validation
pnpm typecheck,pnpm format,pnpm check:i18n, andgit diff --checkpassed.pnpm lint:fixreports five existing errors in the ignored localtmp/dev-ipad.mjsdebugging script. Excluding exactly that file produces zero errors and 350 existing warnings. Both commit hooks passed staged ESLint/Prettier checks.Scope and remaining validation