sessions: read Reasonix conversations and retained native usage

[h4rk8s]

What changes

Add Reasonix local conversations and retained native usage to Magpie's existing session subsystem. Reasonix remains an Agent; no subscription/provider plugin or request-path changes are introduced.

The earlier revision was too narrow: it read legacy transcripts and reported canonical stores as unsupported. This revision actually reads the canonical stores instead of substituting an empty-state warning.

• Legacy message JSONL and 1.x turn ledgers remain supported.
• Published 2.29.0 transcript/wire captures cover human raw input, assistant messages without timestamps, retained usage, per-process telemetry and fresh-process resume sequence epochs.
• Canonical events/v3, linear/v3, linear/v3.1 and linear/v4 are decoded read-only. Global/project sessions-v* and Studio desktop-sessions-v*/by-id are discovered by manifest codec, not directory version. The v5 directory uses the v4 codec.
• Complete JSONL commits and checksummed RX4F/Zstandard transactions alone become visible. Torn/in-progress tails are ignored until committed; malformed boundaries, digest failures and unknown mandatory events are reported.
• Content-addressed payloads verify length/SHA-256 and path confinement. Canonical complete/upsert/retract/history replacement is projected; provider context changes do not erase UI history.
• Explicit imported legacy sources keep their retained usage under the canonical session identity, without a second conversation or duplicated cold-cache totals.
• Native sessions reach List, management, stats aggregation and transcript HTTP handlers. Large transcripts resolve final identity/order first and materialize only the shared display window. Summary caches retain aggregates, not full transcript bodies.
• Diagnostics now describe actual unknown/unreadable data, rather than rejecting every v4/v5 directory. Obsolete/Studio catalogs that the CLI cannot select are viewable read-only.

Usage boundary

Canonical Reasonix currently uses an in-memory turn ledger and daily provider stats without a session identity. Those global records cannot be joined exactly to individual conversations by timestamp alone. Explicit manual heuristic recovery is described below; it is separate from the normal reader. These conversations are visible with partial native usage history; explicit retained session receipts are counted, and new Magpie gateway requests retain their independent gateway accounting. Missing attribution is never presented as known zero or invented per-session spend.

Producer evidence

• testdata/reasonix-2.29.0: files emitted by the published 2.29.0 native host, including a fresh resumed process against a loopback fake provider.
• testdata/reasonix-stores: all four canonical codecs emitted by Reasonix's source producer at 4a050542, with synthetic public input and a real external payload object. This is codec producer coverage, not a claim that every frontend was driven interactively.
• Authorized read-only local history check: 19 store directories decoded without an error; 16 nonempty native sessions, including 8 active within seven days, are visible through the actual session summary. No private transcript is included in the PR.

Verification at this PR head

• go test -tags nogui ./internal/sessions ./internal/gui passes at 442c7130.
• Canonical regression tests cover all codecs, Studio layout, a future-number directory with a known codec, external payloads, committed/torn resume append, upsert/retract, context versus history replacement, unknown required events, corrupted objects, linked imported usage and cold replay.
• Real HTTP handlers verify native list/manage/transcript responses.
• Focused canonical race tests pass in the integrated release lane. Chromium language/layout checks pass; changed diagnostic expectations were corrected and rechecked. The standalone WebKit automation baseline remains unverified and is not counted as passing.
• Read-only isolated local summary measurement: cold about 2 s, warm 10 ms; native transcript about 2.4 s for the shared cut window. This is not a full production page benchmark.

The 26 review perspectives were reapplied as an author self-review, including producer compatibility, identity/attribution, partial tails, private data isolation, actual user path, scale, integration and installation/runtime separation. This does not claim exhaustive proof or independent review.

Owning contract: Reasonix sessions.

Local BYOK release tooling and removable patch archives are excluded from this PR. The integrated BYOK release v0.1.1108-22-gcb8966fd (Build 2190) has passed its full release gates and is installed on the local Mac with controlled configuration hashes unchanged. Native UI-controller startup currently fails; the running App restart and production-page observation are explicitly still pending, so no live UI completion is claimed.

---

🎬 界面预览

_{CI 用这个 PR 的代码构建并真实运行 magpie（沙盒环境，配置了真实的 DeepSeek key），按改动自动操作、截图和录屏 · 747b479 · 运行记录}

改动（按代码）：这个 PR 给会话子系统接入了 Reasonix：用量 › 会话页在存在无法读取的 Reasonix 存储时，把统计行（#sessNote）和空列表文案换成兼容性提示；用量历史不完整的会话会在会话行和会话详情里显示「用量历史不完整」及一行说明，并补上中/日/德翻译。

Note

沙盒里看不到：沙盒里没有安装 Reasonix，也没有它的会话文件或存储目录，所以 unsupported_reasonix 与 usage_incomplete 都不会为真：新增的「有 N 个 Reasonix 会话存储存在无法读取或未知的数据」提示和会话行/详情里的「用量历史不完整」都不会出现，Agent 筛选条里也没有 Reasonix 这一项。场景只能展示这些文案会落到的位置（会话统计行、会话行、会话详情），内容仍是沙盒里 Claude Code、Codex 等既有会话。

_{▶️ 点图打开录屏（可暂停、拖动）· 直接下载 mp4 · 红线是鼠标轨迹，红色圆环是点击}

用量 › 会话页的提示位置

用量 › 会话：会话统计行与列表

会话统计行：本次改动改的就是这一行的文案

会话行与会话详情里的用量标记

会话列表中的一行：标题、模型、用量与费用

会话详情面板顶部：用量说明加在这一段的前面

Follow-up verification at 6765231

Merged current upstream main (6a80954) without dropping the upstream session pagination/count note or translations. Draft remains intentional while CI runs.

Fixed a native-history regression: sessions with retained authored messages but no token receipts/model metadata now participate in session/message totals. Legacy host session-context envelopes no longer become user titles; explicit user origin takes precedence. Activity is accumulated before applying the latest store-event timestamp. Native summary revision is bumped so existing caches reparse.

Regression covers an unmetered native history with a legacy host envelope, a real user/assistant pair, a 10-second activity interval, exact session/message totals, unknown usage, and cold reload. Local checks: go test -tags nogui ./internal/sessions ./internal/gui passed; the selected Chromium/session/i18n suites passed 15/15. No token values were inferred from text, and no user history was included in fixtures.

Title follow-up at af5be43

The native page exposed two more preview cases: legacy hook-context and compaction-summary text. Preview derivation now strips the producer's ten leading transient-context block kinds before truncation, skips legacy compaction summaries, and preserves raw user input plus explicitly user-authored compaction literals. Original transcripts are unchanged. Native preview cache revision is bumped.

Full sessions/gui Go suites pass; regression covers all ten transient tags, context-only records, wrapped real text, raw-input precedence, and explicit user provenance. A private read-only audit of native stores found no remaining session-context/hook-context/compaction-summary preview titles; user content is not included here.

All 26 review perspectives were revisited as an author self-review, not independent review or 26 automatic tests. Remaining gaps: absent canonical per-session usage receipts, top-level zero presentation when usage is unknown, full standalone WebKit interaction coverage, and CI/native release validation of this latest follow-up. Draft is retained. Prior head 6765231 passed all eight remote CI checks; those results are not attributed to af5be43.

Historical recovery closure at 1f7990a

The supported-format contract and manual recovery boundaries are now documented
in docs/subsystems/reasonix-sessions.md, with a standard-library-only offline
preview tool in tools/reasonix-history/ (Python 3.11+).

Supported by this reader: legacy JSONL/metadata, retained 1.x turn ledgers,
published 2.29.0 wire/telemetry, and canonical events/v3, linear/v3,
linear/v3.1 and linear/v4. Known codecs work independently of directory version.
Unknown mandatory codecs/events and corrupt committed records remain diagnosed;
conversation availability and retained usage availability are distinct.

Manual tools in this PR: exact receipt/daily-stat reconciliation is the
default. --allow-estimates --activity-index ... explicitly permits a reviewed
model/activity/lifetime heuristic. It generates per-record source hashes,
alternatives, confidence grades, per-session totals and a candidate overlay.
A new output directory is required. No network, credential refresh, source
history rewriting, installation, migration or process control occurs. Malformed
JSONL rows are skipped, so this is not a repair/completeness guarantee. The
canonical activity-index exporter is not included; users must supply a reviewed
index. This is an opt-in offline recovery preview, not an automatic startup path.

Local BYOK treatment, excluded from this PR: a removable overlay reader and
estimated UI badge replace covered-date native receipts rather than adding the
same consumption twice. Removing the overlay restores native-only accounting.
The local historical import is installed on Max, not copied as shared config.
The upstream reader in this PR does not consume that candidate overlay file;
there is no claimed upstream import command. A dated snapshot must be regenerated
for later usage on covered dates. Durable future exact attribution needs the
producer to persist session/request identities alongside usage.

Evidence: 13 Python tests pass, including duplicate/conflicting receipt
identity, ambiguous joins, integer-millisecond timestamps, ownership mismatch,
source immutability, model/nearest-time/overlap/lifetime policies, explicit consent,
output confinement, and a CLI preview with exact token conservation. A private
read-only real-history rerun assigned 11,937 daily records to 32 sessions and
conserved all token totals; every ownership decision matched the installed local
snapshot. Latest evidence grades: 519 measured, 5,683 high, 1,902 medium, 3,833
low. The 54 promotions from high to measured are exact integer-millisecond
receipt reconciliation, not additional usage. Grades are not calibrated
probabilities or a claim of correct individual ownership. No private history,
activity index, tokens/keys or original output artifacts are committed.

This follow-up changes documentation/offline tools only; it does not change the
App/gateway reader or require a new App release. Draft remains intentional. The
13 tool checks are not presented as all 26 review perspectives or as full native
UI/remote CI completion. Earlier review limitations remain explicit above.

Upstream race fix synchronization at 64b4bfb

The previous Ubuntu/macOS CI failures were reproduced on the exact upstream
base 52ad2d9, without the Reasonix changes: all six affected Claude gateway
tests report races in claudeCredentials.marshal. CI had tested merge 249e6d1
(PR 1f7990a + base 52ad2d9), not the PR branch alone.

Upstream has already fixed this in 15afd98: clone both the credential blob and
its nested OAuth map before serialization. This PR now merges upstream 7d8cda1
and includes that existing fix and its upstream concurrency regression test;
no duplicate provider fix or separate PR is added. After synchronization, the
credential regression and all six previously failing gateway tests pass with
-race -count=10, sessions and gui pass -race, and all 13 offline recovery tests pass.
New-head full remote CI is still pending; Draft is retained. No App installation
or process restart is part of this follow-up.

Latest upstream 7d8cda1 also adds the nested ChatGPT compaction-error fallback. Only gateway codex_backend.go and protection_test.go changed since dc08da8; the latest-head focused race checks include those protection/compaction tests.

[yetone]

Thanks @h4rk8s, this is careful work. I merged 9d6b36a onto main (38a0f47) and ran it there:

• go test -race ./internal/sessions ./internal/agentenv passes, and so does -race -count=20 -run Reasonix. go vet is clean.
• sessions-talk and usage-sessions-agents-fit pass 16/16 in Chromium and WebKit (en/zh/ja/de). gui-ja, gui-de and i18n-keys pass.
• I checked the file shapes against Reasonix's source (esengine/DeepSeek-Reasonix @ 9d4a2bd) and the published binaries. .jsonl.meta (id, topic_title, custom_title, workspace_root), the message fields (createdAt, local_only, reasoning_content, tool_calls[].name/arguments) and the sessions/ and projects/*/sessions/ layout all match.

One thing has to change before this can merge.

On the current release, every session shows 0 tokens and "Partial usage history". The usage reader only reads the .turns.jsonl ledger, and only the 1.x line writes that file. The npm latest tag is reasonix@2.29.0. Its own source says so in internal/state/store/session.go: ".turns.jsonl", // 1.x writes it into the session directory both lines share. The 2.29.0 darwin-arm64 binary has no turnevent or compactedThroughSeq at all, while 1.39.7 (next/canary) has both. 2.x keeps usage only in the daily stats/*.jsonl, which has no session id. So for a user on the default install:

• every Reasonix session lists 0 in / 0 out with "Partial usage history";
• the Models column is empty too, though each 2.x assistant message carries modelRef (provider.Message.ModelRef, "names the model that wrote this assistant turn"). The adapter never reads it.

Please:

1. Read modelRef from the assistant messages. A 2.x session then lists its models even when it has no token counts.
2. Add a fixture in the shape 2.29.0 actually writes: transcript plus .meta, no ledger, assistant lines with modelRef. Assert what the page shows for it.
3. Say in the PR description and in reasonix-sessions.md which release line has per-session usage (1.x) and that 2.x shows partial history by design. The current text reads as if usage works everywhere. Also say which version the "real native JSONL files" you checked came from.

Smaller things, not blocking:

• HostAuthored user messages (lines the host composed, not typed by the user) are counted as prompts. On a session with no meta they can also become its title. Skip host_authored: true the way you skip local_only.
• Dirs() now calls reasonixFiles(), which reads every .meta and the head of every metadata-free transcript, on each /api/sessions call. All Dirs() needs is to know whether ReasonixDir()/sessions (or a project's) exists. A stat is enough.

[h4rk8s]

Thanks for catching the release-line gap. Addressed all three required changes and both smaller points in 5792901:

1. Assistant modelRef now supplies model identities in both Sessions and Usage → Sessions, even without a ledger. It adds no token usage. Zero-token identities do not count as unpriced spend. Older cached transcript summaries are rebuilt once, so existing sessions also gain the models.
2. Added the 2.29.0-shaped transcript + .meta fixture under internal/sessions/testdata/reasonix-2.29.0/, without a turn ledger. Backend assertions cover model presence, zero recorded tokens/partial history, cold reload and old-cache upgrades. Both GUI suites read that fixture and assert Flash/Pro names, unknown token totals (—) and the localized partial-history label.
3. Updated the PR description and subsystem reference: per-session ledger usage is a 1.x feature; 2.x partial history is intentional because its daily stats have no session IDs. npm tags checked today are latest=2.29.0 and next/canary=1.39.7. Source fields were checked against Reasonix 9d4a2bd. The earlier real-file observation came from my local 1.x installation v1.39.4-74-g4a0505420 (4a050542060a). Historical files have no writer-version stamp, so I cannot claim an exact producer version for each old file. That observation did not validate the published 2.29.0 executable; the new 2.x checks are source-contract/fixture validation.
4. host_authored: true is excluded from prompt counts and fallback titles, like local_only. Native transcript content remains readable.
5. Dirs() now checks the native session directories, without reading any .meta or transcript heads. Empty global/project session directories are covered.

Verification:

• go test -tags nogui -race ./internal/sessions ./internal/agentenv: pass.
• go test -tags nogui -race -count=20 -run Reasonix ./internal/sessions: pass.
• go vet ./..., macOS build, Linux nogui build, Windows build: pass. Cross-builds are not platform runtime validation.
• Full go test -tags nogui -p 4 ./... under a temporary HOME with pinned Go caches: pass.
• Both affected Playwright suites: 16/16, Chromium/WebKit × en/zh/ja/de; narrow strip checks retained.
• Four mutations fail real assertions: missing 2.x models; host lines counted as an extra prompt; empty session directory omitted; existing cache hiding model identities. Restoring each fix passes.

Updated-head GitHub tests and UI preview are still running at the time of this reply; the two CLI build checks already pass. No live agent configuration was changed for these checks.

[yetone]

Thanks @h4rk8s. The 1.x side, the host-authored filter and the stat-only Dirs() are all fixed. I reviewed 5792901, merged onto main (b1cba65):

• go test -race ./internal/sessions ./internal/agentenv ./internal/gui passes. go vet, GOOS=linux go build -tags nogui and GOOS=windows go build pass too.
• With reasonixFiles() taken out of allFiles(), all six Reasonix tests fail ("want one Reasonix conversation, got 0"). With the modelRef block disabled, TestReasonix229ModelsWithoutLedger fails. So the tests do guard the code.
• sessions-talk and usage-sessions-agents-fit pass 8/8 in Chromium and in WebKit. gui-ja, gui-de and i18n-keys pass 7/7.

Then I ran the real 2.29.0 binary, because the fixture was written from the source and not from what 2.x puts on disk. I used @reasonix/cli-darwin-arm64@2.29.0 with reasonix serve (the same serve.New that cmd/reasonix-studio-host uses), in a sandbox REASONIX_HOME, against a fake OpenAI-compatible upstream. The upstream returned usage: {prompt_tokens:1234, completion_tokens:56, cached_tokens:1000}. I sent two prompts through POST /submit and pointed this PR's reader at the result. The bytes don't match the fixture, and 2.x does keep per-session usage:

1. 2.x keeps per-session usage in two places. The PR and docs/subsystems/reasonix-sessions.md:24-25 say it doesn't.

   • <id>.jsonl.telemetry.json holds the session's own totals. The host writes it (internal/state/usagereport, "the per-session token record the host writes beside a session"). Real bytes: {"version":1,"usage":{"promptTokens":2468,"completionTokens":112,"reasoningTokens":0,"cacheHitTokens":2000,"cacheMissTokens":468,"requestCount":2,"sources":{"executor":{"requestCount":2}},...}}.
   • <id>.wire.jsonl has a usage frame for each request (internal/frontend/serve/wirelog.go, wireLogKinds includes "usage"). The turn_started frame before it carries the modelRef: {"kind":"turn_started",...,"modelRef":"fake/fake-model","seq":1} … {"kind":"usage","usage":{"promptTokens":1234,"completionTokens":56,"cacheHitTokens":1000,"cacheMissTokens":234,"source":"executor","attemptId":"sa-1-1",...}}. The log is capped at 8 MB. When the cap drops frames, <id>.wire.meta.json says {"truncated":true}.

   reasonixSidecar (internal/sessions/reasonix.go:135) skips .wire, so the PR shows 0 tokens and "Partial usage history" for a session whose totals are on disk. Please read the telemetry totals, or the wire usage frames (with turn_started.modelRef for the model and .wire.meta.json for incompleteness), and correct the description and the reference doc.

2. The title and the transcript show the injected <workspace> block, not what the user typed. A real 2.x user line has "content":"<workspace>\nCurrent workspace: ...</workspace>\n\n<available-skills>...\n\nhello there\n\n<execution-policy ...>","raw_content":"hello there". reasonix.go:223 (title(m.Content)) and reasonix.go:302 (Text: m.Content) read content. The real session listed as title="<workspace> Current workspace: \"/private/tmp/... and the transcript's user parts started with the same block. Use raw_content when it is present. The meta also has "preview":"hello there".

3. Real 2.x assistant lines have no createdAt. Real line: {"role":"assistant","content":"hello back","workDurationMs":36,"modelRef":"fake/fake-model"}. Only user lines carry it. reasonix.go:236 returns before d.Replies++ and the per-day models when At <= 0. So on the real session, Usage → Sessions showed prompts=2 replies=0 models=[], while your fixture gives Replies 2, models 2. The fixture testdata/reasonix-2.29.0/session.jsonl:3-4 invents createdAt on the assistant lines.

4. The .meta fixture isn't what 2.29.0 wrote. The real one: {"id":"20261006-160431.448745000-fake-model","created_at":"...","updated_at":"...","model":"fake/fake-model","revision":4,"content_digest":"...","writer_id":"...","schema_version":2,"turns":2,"preview":"hello there"}. It has no workspace_root (it is omitempty in sessionstore/branch.go), so the real session listed cwd="". Your fixture (session.jsonl.meta) always has it. Please also test the case where it's absent. The <workspace> block or the project directory could give the folder. Also, reasonix_test.go:33 writes a .wire.jsonl containing {"role":"user",...}. A real wire log is the frames shown above, so that fixture tests the wrong thing.

Please rebuild the 2.x fixtures from bytes a real 2.29.0 run writes (system line, user with raw_content, assistant without createdAt, the real .meta, .telemetry.json and .wire.jsonl), and assert what the page shows for them. Not blocking: the [storage] table in config.toml can move the state root (configuredRootDir). ReasonixDir() doesn't follow it. A note in the reference doc is enough.

[h4rk8s]

Thanks for checking the published producer. My previous 2.x conclusion and source-derived fixture were wrong. Revision 640e136 replaces them with native files from the integrity-verified official 2.29.0 executable and addresses all four findings:

1. Read retained .wire.jsonl usage by actual model/date; telemetry verifies coverage rather than being added again. Truncation/missing anchors/invalid or estimated usage stays partial.
2. Human titles and transcript prefer raw_content (including empty presence), ahead of injected workspace/skills/policy.
3. Timestamp-free native assistant replies/models belong to the preceding native user turn's date. The real two-turn fixture gives 2 prompts / 2 replies, not 2 / 0.
4. The real meta has no workspace_root; the explicit host workspace block supplies the fallback. Native metadata preview/fields are retained, and the custom [storage] discovery limitation is documented.

I also ran a new published serve --resume process, rather than simulating resume inside one process. That exposed an additional boundary: wire seq restarts at 1, while telemetry resets to only the latest process's 1 request. The wire still has 3 usages. The resumed native fixture now yields 3/3 prompts/replies, 702 uncached input, 168 output, 3000 cache read without double counting or dropping earlier turns. Fixtures and normalization provenance are in this head.

Verification I completed before this push:

• Exact head merged without conflict onto upstream 290c60a; final local integration 40dec791. Sandboxed HOME, pinned caches, proxy variables cleared for the final full suite.
• sessions/agentenv/gui package and race runs, Reasonix -race -shuffle=on -count=20, vet and three platform builds pass. Full integrated nogui suite passes every package.
• The actual sessions/manage/stats/transcript HTTP handlers read the published native fixture; both browser pages plus ja/de/i18n pass 23/23 on PR and main integration.
• Seven deliberate regressions each fail behavior assertions (raw_content, assistant date, resumed telemetry, truncation marker, sidecar cache invalidation, usage buckets, global seq dedup). No compile failure was used as evidence.
• 20-iteration M1 Max microbenchmarks: 1.x text-heavy append ~0.28 ms; 2.x wire beside a 16 MB irrelevant transcript tail ~0.73 ms. A scratch ~8.25 MB wire stress case retains the same usage at ~16.9 ms/rebuild. Timestamp lookup stops at the last referenced user index.

Earlier failures are retained in the verification record and PR body: standalone old-base fixed-date usage test (already corrected in current main), temp-HOME WebKit path, and inherited loopback proxy affecting TestOffline. No unrelated production changes or live app/config changes were made. The PR description now explicitly retracts the old 2.x statement and separates published-host proof, handler integration, browser mocks, real-vendor/OS-runtime gaps and new-head CI.

[h4rk8s]

Follow-up summary for 1f7990a: I updated the PR description and supported-format/recovery contract, and added an offline historical recovery preview tool.

• The reader covers legacy JSONL/metadata, retained 1.x turn ledgers, published 2.29.0 wire/telemetry, and canonical events/v3, linear/v3, linear/v3.1 and linear/v4. Directory version alone does not select the decoder; unknown mandatory codecs/events and corrupt committed records remain diagnosed. Readable conversations do not imply complete retained usage.
• The tool defaults to exact receipt/daily-stat reconciliation. Heuristic ownership requires explicit --allow-estimates and a reviewed activity index. It produces per-record evidence/alternatives, evidence grades, per-session totals and a candidate overlay without rewriting history, accessing providers, installing anything or restarting processes. Canonical activity-index export is still a gap.
• The removable overlay reader and estimated UI labels are local BYOK patches, excluded from this PR. Upstream does not yet import the candidate overlay. The local recovery replaces covered-date receipts rather than adding duplicate consumption; removal restores native accounting. It is a historical snapshot, not ongoing attribution. Future exact attribution requires the producer to persist session/request identities with usage.
• Verification: 13 offline-tool tests pass. A private read-only rerun assigned 11,937 daily records to 32 sessions, conserved token totals and matched every ownership decision in the installed local snapshot. Evidence grades are not probabilities or proof of exact individual ownership. Private histories and recovery outputs are not committed. These checks do not stand in for full native UI validation or new-head remote CI.

I am keeping this PR Draft while the remaining integration/review gaps are resolved; this comment is a status clarification, not another request for you to review an unfinished revision. I am sorry the earlier submissions cost you repeated review and repair time. I should have validated the published producer and real historical-data path before asking for review. The description now separates implemented behavior, the local workaround and remaining gaps explicitly.