ralph: defer per-Note Pandoc AST for #66 (-57% RSS)

docs/dev/ralph/memory-66-deferred/README.md

@@ -0,0 +1,158 @@
+---
+slug: ralph-memory-66-deferred
+---
+
+# Defer the per-Note Pandoc AST (#66)
+
+A second pass at [issue #66](https://github.com/srid/emanote/issues/66),
+implemented from scratch as a fresh take on
+[PR #678](https://github.com/srid/emanote/pull/678)'s approach. PR
+[#740](https://github.com/srid/emanote/pull/740) shipped the
+behaviour-preserving local fixes (`deepseq` + `-F1.5` + `_relCtx`
+removal, **−29%**); this branch attacks the structural ceiling those
+cycles deliberately left in place.
+
+## What changes structurally
+
+`Note._noteDoc :: Pandoc` is no longer a stored value. It is now a
+**lazy field bound to a re-parse closure** over `_noteSourceText`.
+First access on the render path forces the closure (pure markdown
+re-parse, no Lua); Haskell's normal memoisation then writes the result
+back into the field so subsequent accesses within the same `Note`
+reference are free.
+
+Three categories of notes still bind `_noteDoc` to the
+already-evaluated Pandoc because a pure re-parse cannot reproduce
+their post-filter shape:
+
+| Retention reason     | Source                                              |
+| -------------------- | --------------------------------------------------- |
+| Lua filter declared  | `pfdParseFilters` / `pfdRenderHtmlFilters` non-empty |
+| Synthetic note       | `mkEmptyNoteWith` (folder placeholders, banners)    |
+| Parse errors         | `_noteErrors` non-empty (banner Block prepended)    |
+
+Two new pre-extracted summary fields land at parse time so neither
+`noteRels` (link index) nor `noteTasks` (task index) needs to walk the
+full Pandoc:
+
+```haskell
+data Note = Note
+  { _noteSourceText  :: !Text
+  , _noteLinks       :: ![NoteLink]
+  , _noteTaskList    :: ![NoteTask]
+  , _noteDoc         :: Pandoc      -- lazy on purpose
+  , …
+  }
+```
+
+`parseAndInsert` deepseq-forces only `_noteLinks` and `_noteTaskList`
+at insert time — never `_noteDoc`. Forcing `_noteDoc` would defeat the
+optimisation by materialising the re-parse closure into a Pandoc
+immediately. The two summary fields would otherwise hold thunks
+closing over the post-filter Pandoc that `parseNote` produced, keeping
+it alive after the local `note` reference is dropped.
+
+`mkNoteWith` also branches **eagerly** on whether the note must retain
+its Pandoc — a single `if mustRetain then doc else deferredDoc`
+expression as the field value would keep both branches alive in a
+thunk closure (`doc` *and* `deferredDoc`). Two separate
+record-construction branches commit each Note's `_noteDoc` to exactly
+one of them.
+
+`Eq` / `Ord` for `Note` are now manual and compare by `_noteRoute`
+alone. The derived versions would force the lazy `_noteDoc` on every
+IxSet insertion-time comparison, re-parsing the AST for every other
+note already in the set — quadratic, and exactly the work we're
+trying to avoid.
+
+`modelLookupBacklinks` re-extracts per-link context from the source
+note's Pandoc on demand via a new `noteRelCtxToTarget`, replacing the
+contexts that PR #740's cycle 3 already stopped storing inside `Rel`.
+
+## Final result (5-run median on 4.5k synthetic corpus, `+RTS -N`)
+
+| Metric        | Baseline @6950760d | This PR  | Δ |
+| ------------- | -----------------: | -------: | --- |
+| `READY` (s)   |                 51 |       54 | +6 % |
+| `LOAD_HWM`    |           5165 MiB | 1660 MiB | **−67.9 %** |
+| `AFTER_HWM`   |           5181 MiB | 2219 MiB | **−57.2 %** |
+
+For the smaller 1k corpus the absolute numbers drop to ~543 MiB load
+peak / ~610 MiB after-hits (~−53 % vs the same 1300 MiB baseline).
+
+## Methodology
+
+Identical to PR #740 — synthetic 4501-file / 72.4 MiB corpus produced
+by `docs/dev/ralph/memory-66-deferred/gen_corpus.py` (fixed seed),
+measured on `srid1` (32 cores, 125 GiB RAM, NixOS, no swap) via
+`measure.sh`. The driver starts `emanote run`, polls `curl /` until
+ready, samples `/proc/<pid>/status` for `VmRSS` / `VmHWM`, then pings
+six pages and re-samples.
+
+Tests (`cabal test emanote`): 125 examples, **0 failures**. The
+`RelSpec` source-order test was rephrased to assert on `_relSrcPos`
+directly (since `_relCtx` no longer carries context — same change as
+PR #740 cycle 3).
+
+## Known limitation: dense-embed render latency on this synthetic corpus
+
+The synthetic corpus is intentionally pathological — every note
+contains an average of one `![[embed]]` and a handful of `[[wiki]]`
+links to other random notes. With deferred parsing, rendering an
+embed-heavy page re-parses every transitively embedded note on first
+access; pages like `/topic00/n00032` (19 direct embeds, transitive
+fan-out into hundreds of notes) exceed the 30s curl timeout on a cold
+process.
+
+This is a real limitation of the "memoise into the lazy field"
+approach: the first render of a hub note is bounded by the size of the
+embed graph reachable from it. Subsequent renders of the same hub are
+fast because each unique embedded note has been forced once.
+
+Mitigations not implemented in this PR:
+
+* a stripped-down "summary Pandoc" for embed bodies (PR #678's
+  approach), trading retention for a fixed-cost embed render;
+* request-scoped Pandoc cache plumbed through `RenderCtx`;
+* an `unsafePerformIO` `IORef` per Note for explicit cache eviction.
+
+On the real `docs/` notebook the slowness does not surface in practice
+because the embed graph is shallow (RSS settles at ~255 MiB).
+
+## Files changed
+
+* `emanote/src/Emanote/Model/Note.hs` — new fields, lazy `_noteDoc`,
+  manual `Eq`/`Ord`/`Show` skipping the AST, `reparseMd` / `reparseOrg`
+  pure re-parse helpers.
+* `emanote/src/Emanote/Model/Link/Rel.hs` — `noteRels` reads
+  `_noteLinks` (no Pandoc walk), `noteRelCtxToTarget` recomputes on
+  demand.
+* `emanote/src/Emanote/Model/Task.hs` — `noteTasks` reads
+  `_noteTaskList` (no Pandoc walk).
+* `emanote/src/Emanote/Model/Graph.hs` — `modelLookupBacklinks` uses
+  `noteRelCtxToTarget`.
+* `emanote/src/Emanote/Pandoc/Renderer/Embed.hs` — binds
+  `noteDoc` once per embed so body + TOC splices share the forced
+  thunk.
+* `emanote/src/Emanote/Source/Patch.hs` — deepseq summary fields at
+  insert time (but not `_noteDoc`).
+* `emanote/test/Emanote/Model/Link/RelSpec.hs` — assert source order
+  via `_relSrcPos`.
+
+## Relationship to PR #740 and PR #678
+
+PR #740 (cycles 1-3) and this PR are **additive** — applying both is
+the recommended path:
+
+| Layer         | Change                                          | Δ AFTER_HWM |
+| ------------- | ----------------------------------------------- | ----------- |
+| baseline      | origin/master @6950760d                         | 5181 MiB    |
+| PR #740 ⊕     | `deepseq` + `-F1.5` + `_relCtx` drop            | 3672 MiB    |
+| this PR alone | deferred `_noteDoc`                             | 2219 MiB    |
+| (both)        | not yet measured                                | TBD         |
+
+This PR's design is parallel to PR #678's
+`NoteBodyParsed`/`NoteBodyDeferred` sum type but lands the same
+optimisation through a single record + a lazy field, avoiding the
+extra type and the embedded "summary Pandoc" — at the cost of the
+dense-embed render-latency limitation above.

docs/dev/ralph/memory-66-deferred/gen_corpus.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+"""Generate a synthetic emanote notebook of ~4500 markdown files, ~70MB total.
+
+Each file has:
+- A title heading
+- A YAML frontmatter sometimes
+- Several paragraphs of lorem-like text with wikilinks
+- A few headings
+- Some inline code, occasional list, occasional code block
+
+Wikilinks form a random graph so the link index actually has work to do.
+"""
+import os, random, sys, hashlib, string
+
+random.seed(42)
+
+OUT = sys.argv[1] if len(sys.argv) > 1 else "/home/toor/corpus"
+N = int(sys.argv[2]) if len(sys.argv) > 2 else 4500
+TARGET_BYTES = int(sys.argv[3]) if len(sys.argv) > 3 else 70 * 1024 * 1024
+AVG_BYTES = TARGET_BYTES // N
+
+WORDS = ("the quick brown fox jumps over the lazy dog functor monad applicative haskell "
+         "pandoc emanote ema lvar parser source eval render template heist note ix set "
+         "memory leak profile retainer cost centre static unboxed strict thunk graph link "
+         "wikilink folgezettel sequel zettel obsidian roam neuron foam dendron logseq "
+         "atomic note structure architecture optimisation cycle measurement baseline "
+         "decision dependency volatility encapsulation closure capture share unsharing").split()
+
+TAGS_POOL = ["haskell", "design", "perf", "note", "todo", "idea", "ref", "math", "wip", "draft",
+             "review", "meta", "tool", "lit", "code", "infra", "ux", "ops", "test", "spec"]
+
+def folder_for(i):
+    # 32 top-level folders, optional nested
+    top = f"topic{i % 32:02d}"
+    if i % 7 == 0:
+        return os.path.join(top, f"sub{(i // 32) % 11}")
+    return top
+
+def slug(i):
+    return f"n{i:05d}"
+
+def title(i):
+    return " ".join(random.sample(WORDS, k=random.randint(2, 5))).title()
+
+def paragraph(words=120):
+    out = []
+    while sum(len(w) for w in out) + len(out) < words:
+        out.append(random.choice(WORDS))
+    s = " ".join(out)
+    return s[0].upper() + s[1:] + "."
+
+def wikilink(target_i, alias=None):
+    t = slug(target_i)
+    if alias:
+        return f"[[{t}|{alias}]]"
+    return f"[[{t}]]"
+
+def write_file(i, n, path):
+    has_fm = (i % 3 != 0)
+    lines = []
+    if has_fm:
+        tags = random.sample(TAGS_POOL, k=random.randint(0, 4))
+        lines.append("---")
+        lines.append(f"title: {title(i)}")
+        if tags:
+            lines.append("tags:")
+            for t in tags:
+                lines.append(f"  - {t}")
+        if i % 13 == 0:
+            lines.append(f"order: {i % 50}")
+        lines.append("---")
+        lines.append("")
+    lines.append(f"# {title(i)}")
+    lines.append("")
+    # body — keep generating paragraphs until size ~ AVG_BYTES
+    target = max(2000, int(random.gauss(AVG_BYTES, AVG_BYTES / 4)))
+    while sum(len(x) for x in lines) < target:
+        kind = random.random()
+        if kind < 0.55:
+            p = paragraph(random.randint(40, 180))
+            # sprinkle wikilinks
+            tokens = p.split()
+            for _ in range(random.randint(1, 4)):
+                j = random.randrange(len(tokens))
+                target_i = random.randrange(n)
+                alias = tokens[j] if random.random() < 0.5 else None
+                tokens[j] = wikilink(target_i, alias)
+            lines.append(" ".join(tokens))
+            lines.append("")
+        elif kind < 0.7:
+            lines.append(f"## {title(i)}")
+            lines.append("")
+        elif kind < 0.82:
+            # list
+            for _ in range(random.randint(3, 8)):
+                lines.append(f"- {paragraph(random.randint(8, 25))}")
+            lines.append("")
+        elif kind < 0.92:
+            # code block
+            lines.append("```haskell")
+            lines.append(f"foo{i} :: Int -> Int")
+            lines.append(f"foo{i} x = x + {i}")
+            lines.append("```")
+            lines.append("")
+        else:
+            # embedded note (becomes processed)
+            lines.append(f"![[{slug(random.randrange(n))}]]")
+            lines.append("")
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    with open(path, "w") as f:
+        f.write("\n".join(lines))
+
+def main():
+    os.makedirs(OUT, exist_ok=True)
+    # an index.md at root
+    with open(os.path.join(OUT, "index.md"), "w") as f:
+        f.write("# Synthetic Corpus\n\nGenerated by gen_corpus.py for emanote #66 reproduction.\n")
+    for i in range(N):
+        rel = os.path.join(folder_for(i), slug(i) + ".md")
+        write_file(i, N, os.path.join(OUT, rel))
+    # report
+    total = 0
+    count = 0
+    for root, _, files in os.walk(OUT):
+        for f in files:
+            if f.endswith(".md"):
+                total += os.path.getsize(os.path.join(root, f))
+                count += 1
+    print(f"Wrote {count} files, {total/1024/1024:.1f} MB total", flush=True)
+
+if __name__ == "__main__":
+    main()

docs/dev/ralph/memory-66-deferred/measure.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Measure RSS of `emanote run` at: (a) ready, (b) after page hits, plus VmHWM.
+set -eo pipefail
+EMANOTE=${EMANOTE:-/home/toor/code/emanote/dist-newstyle/build/x86_64-linux/ghc-9.8.4/emanote-2.0.0.0/x/emanote/build/emanote/emanote}
+export emanote_datadir=${emanote_datadir:-/home/toor/code/emanote/emanote/default}
+CORPUS=${1:?corpus path}
+RTS=${2:-}
+PORT=${PORT:-$(( RANDOM % 10000 + 9000 ))}
+TIMEOUT=${TIMEOUT:-600}
+LOG=$(mktemp)
+cd "$CORPUS"
+RTS_ARGS=""
+[ -n "$RTS" ] && RTS_ARGS="+RTS $RTS -RTS"
+$EMANOTE -L "$CORPUS" run --port "$PORT" $RTS_ARGS > "$LOG" 2>&1 &
+PID=$!
+READY=0
+for i in $(seq 1 "$TIMEOUT"); do
+  if ! kill -0 $PID 2>/dev/null; then echo "DIED" >&2; tail -40 "$LOG" >&2; exit 1; fi
+  if curl -s -o /dev/null --max-time 1 "http://localhost:$PORT/"; then READY=$i; break; fi
+  sleep 1
+done
+[ "$READY" = 0 ] && { echo "TIMEOUT" >&2; kill $PID; exit 1; }
+LOAD_RSS=$(awk '/VmRSS/{print $2}' /proc/$PID/status)
+LOAD_HWM=$(awk '/VmHWM/{print $2}' /proc/$PID/status)
+for p in / /topic00/n00000 /topic05/n00160 /topic15/n00480 /topic25/n00800 / ; do
+  curl -s -o /dev/null --max-time 5 "http://localhost:$PORT$p" || true
+done
+sleep 3
+HIT_RSS=$(awk '/VmRSS/{print $2}' /proc/$PID/status 2>/dev/null || echo 0)
+HIT_HWM=$(awk '/VmHWM/{print $2}' /proc/$PID/status 2>/dev/null || echo 0)
+printf "READY=%d LOAD_RSS=%d LOAD_HWM=%d AFTER_RSS=%d AFTER_HWM=%d\n" "$READY" "$(($LOAD_RSS/1024))" "$(($LOAD_HWM/1024))" "$(($HIT_RSS/1024))" "$(($HIT_HWM/1024))"
+kill -INT $PID 2>/dev/null || true
+sleep 1
+kill $PID 2>/dev/null || true
+wait $PID 2>/dev/null || true

emanote/emanote.cabal

@@ -113,6 +113,7 @@ common library-common
     , commonmark-wikilink    >=0.2
     , containers
     , data-default
+    , deepseq
     , deriving-aeson
     , directory
     , ema                    >=0.10.1