fix(collab): normalize canonical markdown so tables/structural docs don't wedge projections#65
Open
JVL1 wants to merge 2 commits into
Open
fix(collab): normalize canonical markdown so tables/structural docs don't wedge projections#65JVL1 wants to merge 2 commits into
JVL1 wants to merge 2 commits into
Conversation
…don't wedge projections Documents whose stored canonical markdown differs structurally from what the collab Yjs fragment serializes back to were permanently projection-stale: readSource=yjs_fallback, mutationReady=false. Agent mutations returned 409 PROJECTION_STALE and browser edits were dropped (0 document_y_updates). The Milkdown GFM serializer reformats structural markdown on the fragment round trip — tables get column-padded and gain :--- alignment markers; a list immediately followed by a heading gains a canonical blank line — so any doc created with such markdown never converged. Startup reconcile is disabled, so wedged docs never self-healed. Confirmed hypothesis: the projection freshness/snapshot path requires the fragment serialization to match stored canonical (modulo trailing whitespace); structural divergence wedges it. Not table-specific — a plain list-then-heading doc wedges too. Fix: - deriveCanonicalMarkdownForStorage() normalizes markdown through the exact seed->derive pipeline (a serializer fixed point; idempotent). - POST /documents stores the normalized canonical, so new docs cannot wedge. - healCanonicalMarkdownForCollabFragment() converges an existing doc, gated on structural (trimEnd) divergence so trailing-newline-only diffs are skipped. - scripts/heal-table-projection-wedge.ts: one-time idempotent heal (--dry-run). Tests: collab-table-projection-freshness-regression and collab-table-projection-heal-regression. Updated agent-edit-live-viewer-regression to assert a blocked edit leaves the post-create canonical unchanged rather than hard-coding raw markdown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rite paths, guard heal
Multi-reviewer (Gemini + Claude) found the create-only fix was incomplete and the
heal could lose data. Verified both empirically and fixed:
- Edit/mutation write paths re-wedged (CRITICAL, both reviewers): mutateCanonicalDocument
stored a plain parse->serialize canonical, not the fragment fixed point, so editing a
table via edit/v2 / PUT / rewrite flipped the doc back to readSource=yjs_fallback
(reproduced: healthy after create, wedged after a table edit, permanent). Now the
non-rich branch stores deriveCanonicalMarkdownFromProseMirrorDoc(parsedNext.doc) — the
same fragment the transaction persists — so canonical matches by construction. Verified
the edited table doc stays projection/mutationReady. handleShareMarkdown (POST
/share/markdown) also normalizes now, like POST /documents.
- Heal could drop unflushed edits (CRITICAL, both reviewers): healCanonicalMarkdownForCollabFragment
cleared persisted Yjs state via invalidateCollabDocument. Now guarded to seed-only docs
(getYUpdatesAfter(slug, 0) empty), where re-seeding from the healed canonical is lossless;
docs with persisted updates are skipped ('has_persisted_edits') and kept consistent by
write-path normalization instead.
- Y.Doc leak: deriveCanonicalMarkdownForStorage / the new ProseMirror helper now destroy()
their scratch Y.Doc in a finally.
- Heal script --dry-run now strips ephemeral collab spans to mirror the real gate.
- New collab-table-projection-route-regression test asserts POST /documents itself
normalizes (the freshness unit test bypassed the route).
- Freshness test now pins the corrected value (fragment `:--` table alignment markers),
not just self-consistency.
Full suite: 99 passed; the 35 failures are pre-existing/environmental (missing scripts/lib,
no live server) and identical to baseline — the edit-v2/canonical/mutation tests that pass
on baseline still pass. Codex reviewer unavailable (binary missing); quorum met via
Gemini + Claude.
Note: canonical markdown is now stored in the editor's serialization at every write path,
so a GET no longer echoes the exact POSTed bytes for structural markdown (tables padded,
canonical blank lines). This matches what the editor produces on first open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents containing structural markdown (GFM tables, or a list immediately followed by a heading) were permanently wedged:
readSource=yjs_fallback,mutationReady=false. Agent mutations returned409 PROJECTION_STALEand browser edits were silently dropped (0document_y_updates). Startup reconcile is disabled, so wedged docs never self-healed.Root cause: stored canonical markdown (
documents.markdown) could differ from what the collab Yjs fragment serializes back to. The Milkdown GFM serializer reformats structural markdown on the fragment round trip — tables get column-padded and gain:---alignment markers; a list-then-heading gains a canonical blank line. The projection freshness / snapshot path requires the fragment serialization to match stored canonical (modulo trailing whitespace), so any doc whose canonical wasn't already the fragment fixed point wedged.What changed
deriveCanonicalMarkdownForStorage()/deriveCanonicalMarkdownFromProseMirrorDoc()(server/collab.ts) — normalize markdown into the fragment's serialization (the fixed point; verified idempotent). ScratchY.Docs are destroyed.POST /documentsandPOST /share/markdown(server/routes.ts)mutateCanonicalDocument— edit/v2, PUT, rewrite (server/canonical-document.ts); the non-rich branch now stores the fragment form. Previously anedit/v2that touched a table re-wedged the doc (reproduced and fixed).healCanonicalMarkdownForCollabFragment()+scripts/heal-table-projection-wedge.ts— one-time, idempotent heal for existing wedged docs (--dry-runsupported). Guarded to seed-only docs (no persisted Yjs updates) so it can never drop unflushed edits.Review
Reviewed by multi-reviewer (Gemini + Claude). Two CRITICALs — incomplete write-path coverage and heal data-loss risk — plus four minors were found, verified, and fixed in the second commit. Codex was unavailable (missing binary locally).
Test Plan
npm test— 74 passed, 0 failed (server + share lifecycle, incl./share/markdown)collab-table-projection-freshness,collab-table-projection-heal,collab-table-projection-route(pin the fragment:--form; assert no wedge; heal idempotent)scripts/lib, no live server)projection/mutationReady;/opscomment → HTTP 200; edit table viaedit/v2→ stays fresh; heal un-wedges a real 20 KB table doc<br>/data-proof) intentionally keeps its non-fragment formNote
Canonical markdown is now stored in the editor's serialization at every write path, so a
GETno longer echoes the exact POSTed bytes for structural markdown (tables padded, canonical blank lines). This matches what the editor produces on first open.🤖 Generated with Claude Code