docs(site): generate the llms.txt corpus from the built site by bloxster · Pull Request #23336 · erigontech/erigon

bloxster · 2026-08-17T09:11:30Z

Takes each page's text from the HTML the Docusaurus site builds, instead of re-parsing the MDX source. Components arrive expanded and links resolved, so a card-grid landing page keeps its prose and renders each grid as a complete link list: Why using Erigon carries all 11 cards with their own descriptions, where the MDX parser emitted 8 and mispaired two. Unresolved version tokens, escaped underscores in RPC-method headings and the FAQ page's inline schema array no longer reach the corpus, and internal links are absolute rather than relative.

Mermaid diagrams are spliced back from the source, since Docusaurus renders them to SVG. Each returns at the top of the section it was written under — the right one when a page repeats that heading text — and a run of them keeps its source order. Nothing finer is reconstructed: recovering the exact paragraph a diagram followed, and the list item or blockquote it sat in, was more machinery than this corpus can justify, holding as it does a single top-level diagram.

The discoverability half of the original change went in separately as #23799.

Tests: 147 in test_generate_llms.py, plus 4 corpus assertions in llms-corpus.test.mjs that run against the generated artifacts.

Copilot

Pull request overview

Improves discovery of Erigon’s LLM documentation artifacts and fixes landing-card extraction.

Changes:

Advertises artifacts through page metadata, sitemap entries, and documentation links.
Parses landing cards within individual <Link> boundaries.
Adds regression tests and regenerates the full corpus.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`llms-full.txt`	Updates the repository corpus.
`docs/site/static/llms-full.txt`	Updates the deployed corpus.
`docs/site/scripts/test_generate_llms.py`	Adds parser regression tests.
`docs/site/scripts/generate-llms.py`	Fixes card parsing and adds validation.
`docs/site/docusaurus.config.ts`	Adds head and sitemap discovery.
`docs/site/docs/get-started/why-using-erigon.mdx`	Links to LLM artifacts.
`docs/site/docs/fundamentals/mcp.mdx`	Documents artifact usage.

Suppressed comments (1)

docs/site/docusaurus.config.ts:177

The MCP page now links both artifacts, so “Nothing else on the web links to them” is inaccurate. The relevant rationale is that static files are omitted from Docusaurus's default sitemap.

          // The llms.txt artifacts live in static/, so Docusaurus never routes
          // them and the default sitemap omits them. Nothing else on the web
          // links to them either, which leaves them unindexable and unreachable
          // by search — append them explicitly.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

yperbasis

Requesting changes for three correctness issues:

docs/site/docusaurus.config.ts: advertise /llms.txt with rel="describedby". The llms.txt v2 proposal reserves rel="alternate" type="text/markdown" for a page-specific Markdown representation and defines rel="describedby" for the llms.txt file covering a page: https://llmstxt.org/#proposal. These site-wide aggregate files are not alternate representations of every page, and v2-aware agents may specifically look for describedby. Keep llms-full.txt discoverable through llms.txt, the visible docs, and the sitemap instead of declaring it a page-wide alternate.
docs/site/scripts/generate-llms.py: the mismatch guard is bypassed when every card fails extraction. if not cards: return None runs before expected is computed, so collect_pages silently falls back to strip_mdx. Compute expected before the early return and add an all-malformed-grid regression test. This matches the existing unresolved thread: #23336 (comment).
docs/site/docs/fundamentals/mcp.mdx: llms-full.txt is advertised as containing every documentation page in full, but collect_pages replaces the complete body of every card-grid page with the synthesized card list. For example, the generated Why using Erigon entry omits its introduction, benefits prose, and MCP section. Preserve the non-card prose or describe this as a cleaned and synthesized corpus instead of claiming complete page contents.

Reviewed at eee030a9f1c37e951048dc8420c5af6d61fcce63. The 81 documentation-script tests, artifact check, diff check, and GitHub docs build are green.

@yperbasis

… claims Review feedback from @yperbasis and Copilot on #23336. Five fixes. 1. Advertise llms.txt with rel="describedby", not rel="alternate". The llmstxt.org proposal defines describedby for the llms.txt file that covers a page, and reserves alternate + text/markdown for a *per-page* Markdown representation. A site-wide index is not an alternate representation of every page, and v2-aware agents look for describedby. llms-full.txt is no longer advertised in head at all: it describes no single page. It stays discoverable through llms.txt, the sitemap, and the MCP docs page. 2. Close a hole in the card-count guard. `if not cards: return None` ran before the count was taken, so a grid where *every* card failed to parse was indistinguishable from an ordinary prose page: the caller fell back to strip_mdx and the guard never ran — silently degrading the exact case it exists to catch. Count first, parse second, and return None only when the page has no cards at all. 3. Stop claiming llms-full.txt holds "every documentation page, in full". It does not: synthesize_landing replaces the whole body of a card-grid page with its card list, so why-using-erigon loses its introduction and prose. Describe the corpus as cleaned rather than verbatim, and say what is dropped. 4. Stop grouping llms.txt with llms-full.txt as "the whole documentation as one plain-text file" on why-using-erigon. llms.txt is only an index. 5. Drop "nothing else links to them" from the config comment — this PR adds the MCP page links, which makes it false. Also refresh the stated file size, 420 KB -> 430 KB.

bloxster · 2026-08-17T12:55:32Z

@yperbasis all three addressed in 1dd741e, rebased onto your merge of main.

1. rel="describedby" — you're right, and I checked the proposal rather than taking it on trust: it defines describedby for the llms.txt file covering a page and reserves alternate + text/markdown for a per-page Markdown representation. A site-wide index is not an alternate representation of every page. Switched to describedby, and dropped the llms-full.txt head tag entirely — it describes no single page, so it has no business in a per-page relation. It stays reachable via llms.txt, the sitemap, and the MCP page.

2. Guard hole — confirmed before fixing. With every card malformed, cards was empty, return None fired before the count, and collect_pages fell back to strip_mdx: total parser failure was indistinguishable from a prose page, which is precisely what the guard was meant to catch. Now counts first, parses second, and returns None only when the page has no cards. New test verified to fail against the old ordering.

3. "Every documentation page, in full" — confirmed and materially false: why-using-erigon.mdx is 183 source lines and reaches the corpus as its 11-bullet card list alone, losing the introduction, the prose and the MCP section. The table now says "the text of every documentation page", with an explicit sentence that the corpus is cleaned rather than verbatim and that card-grid index pages are reduced to their links.

Verified after the rebase: --check green, 82 tests pass, build clean, describedby on 198/199 pages, zero alternate/text/plain tags and zero llms-full head tags remaining, sitemap still carries both files.

One thing worth flagging: #23335 already merged into release/3.6 with all three defects. I'll open a follow-up there carrying these same fixes.

bloxster · 2026-08-17T12:57:58Z

Follow-up for release/3.6 opened: #23341 — same three fixes, since #23335 merged before this review landed.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/site/docusaurus.config.ts:100

The PR description still promises two rel="alternate" type="text/plain" head tags on every page, while this now emits one describedby link and deliberately omits llms-full.txt. Please either restore the advertised tags or update the description and verification so they match the shipped discovery contract.

          rel: 'describedby',
          href: 'https://docs.erigon.tech/llms.txt',

yperbasis

Requesting changes for three remaining correctness issues:

The landing-card mismatch guard still misses cards whose title marker disappears.
The advertised llms.txt index does not directly expose llms-full.txt.
The new completeness wording ignores the published archived documentation versions.

Reviewed at 1dd741e8c1. Targeted validation is green: 82 documentation tests and the llms artifact drift check pass.

@yperbasis

…, scope Follow-up review from @yperbasis on #23336. Three findings, all confirmed by reproduction before fixing. 1. The guard counted the marker it was validating. `expected` came from `lp-card-title`, so if a card lost or renamed that marker the count shrank in step with the loss it was meant to detect: verified that a two-card grid with one renamed marker emitted one bullet and raised nothing. If every marker changed, `expected` hit zero and the page fell back to strip_mdx. Count `lp-card` containers instead — the wrapper is not consumed by the parse, so the two signals stay independent. Attributes are now matched order-independently and `to=` is read separately, which the container match no longer pins down. 2. llms.txt had no route to llms-full.txt. Neither committed index contained the string at all, so once the llms-full head tag was removed, an agent following rel="describedby" reached an index with no way to find the full corpus. The generator now emits that link, and both copies are regenerated. 3. "Every documentation page" was still wrong. SECTIONS scans only docs/ and help-center/, while the site also publishes v3.3 and v3.4 from versioned_docs/ — neither artifact contains those URLs. Say current documentation, and state the exclusion outright. Two new tests: a renamed title marker must raise rather than be absorbed (verified to fail against the previous count), and card attributes must parse in either order.

@yperbasis

…, scope Follow-up review from @yperbasis on #23336. Three findings, all confirmed by reproduction before fixing. 1. The guard counted the marker it was validating. `expected` came from `lp-card-title`, so if a card lost or renamed that marker the count shrank in step with the loss it was meant to detect: verified that a two-card grid with one renamed marker emitted one bullet and raised nothing. If every marker changed, `expected` hit zero and the page fell back to strip_mdx. Count `lp-card` containers instead — the wrapper is not consumed by the parse, so the two signals stay independent. Attributes are now matched order-independently and `to=` is read separately, which the container match no longer pins down. 2. llms.txt had no route to llms-full.txt. Neither committed index contained the string at all, so once the llms-full head tag was removed, an agent following rel="describedby" reached an index with no way to find the full corpus. The generator now emits that link, and both copies are regenerated. 3. "Every documentation page" was still wrong. SECTIONS scans only docs/ and help-center/, while the site also publishes v3.3 and v3.4 from versioned_docs/ — neither artifact contains those URLs. Say current documentation, and state the exclusion outright. Two new tests: a renamed title marker must raise rather than be absorbed (verified to fail against the previous count), and card attributes must parse in either order.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

docs/site/scripts/generate-llms.py:299

This block records the old regex's failure sequence and test-level details rather than the lasting invariant, making it easy for the explanation to become stale. Keep only the card-boundary and independent-count rationale; the regression tests preserve the specific history.

# Parsed in two stages, deliberately. A single pattern spanning the whole card
# cannot express "and never cross into the next card": with `[^<]+` for the text
# and `.*?` for the gaps, a description containing inline markup (`<strong>`,
# `<code>`) fails to match locally, and the engine then scans forward and pairs
# the title with the *next* card's description — silently swallowing the card in

docs/site/docusaurus.config.ts:100

The PR description and verification still promise two rel="alternate" type="text/plain" tags for both artifacts, but this implementation intentionally emits one rel="describedby" link for llms.txt. Please update the PR description and verification claims (including the parser guard's stated count source) to match the final implementation, or restore the advertised behavior.

          rel: 'describedby',
          href: 'https://docs.erigon.tech/llms.txt',

yperbasis

Requesting changes for three remaining correctness issues and one repository-guideline issue.

Reviewed at a97ed68440. Targeted validation is green: 84 documentation tests and the llms artifact drift check pass; the GitHub docs-site build is also green.

… marker Third review round on #23336. The `describedby` link sat in config-level `headTags`, which Docusaurus emits on every route — /v3.3/** and /v3.4/** included. generate-llms.py never walks versioned_docs, so llms.txt covers only the current docs and the help center: an agent reading an archived page was pointed at current, version-specific guidance. It moves to src/theme/Root.tsx, which drops it on archived routes. Verified against a full build: present on /, /fundamentals/**, /help-center, absent on /v3.3/** and /v3.4/**. The expected card count still came from the wrapper alone. That fixed the renamed-title case and left its mirror open: rename `lp-card` and the card vanishes from `containers` and from `expected` together, so one renamed wrapper silently omits a card and a wholesale rename returns None and falls back to strip_mdx. The count is now the largest of three independent markers — wrapper, title, desc — so no single rename can shrink both sides of the guard. Card fields were also accepted empty: `(.*?)` matches "" and `_card_text` reduces markup-only content to "", while the match object stays truthy, so a card emitted as `- [](url): ` passed the count guard. Empty flattened title or description now fails the card, which the guard reports. Adds four regression tests (renamed wrapper, wholesale wrapper rename, empty title, markup-only description) and trims the parser comment to the two invariants it exists to state, per AGENTS.md.

bloxster · 2026-08-18T12:38:14Z

All four review points are addressed on 0387ec253e; each thread carries the detail and is resolved.

Descriptor scope — moved out of config-level headTags into src/theme/Root.tsx, which drops the tag on archived routes. Verified against a full build: present on /, /fundamentals/**, /help-center; absent on /v3.3/** and /v3.4/**. I took scoping over per-version indexes deliberately — the latter is a feature, not a review fix.
Wrapper-marker dependency — you were right that the previous round only moved the blind spot. expected is now the largest of three independently counted markers (wrapper, title, desc), so no single rename can shrink both sides of the guard.
Empty flattened fields — validated before append, so an empty or markup-only title/description fails the card and the count guard reports it instead of emitting - [](url): .
Comment altitude — 12 lines of forensics reduced to the two invariants; the narrative lives in the commit message.

35 tests pass (31 existing + 4 new: renamed wrapper, wholesale rename, empty title, markup-only description) and generate-llms.py --check is clean.

Copilot

🔵 Needs a closer look

The large custom HTML-to-Markdown and Mermaid reconstruction pipeline requires final human validation despite its extensive tests.

Review details

Files not reviewed (1)

docs/site/package-lock.json: Generated file

Files reviewed: 10/13 changed files
Comments generated: 0 new
Review effort level: Balanced

yperbasis

Requesting changes at 0e3e56c3362a910624dbf65984f4974da9589899. I read the full discussion and all submitted reviews and inline comments, including the latest Copilot review.

Two additional P2 findings are described inline:

A correctly contained Mermaid diagram fails the containment test because the HTML count excludes Mermaid while the corpus count includes it.
Consecutive diagrams after an introductory paragraph are reordered from introduction → A → B to B → introduction → A.

Four existing P2 findings still reproduce at this head:

A blockquote nested inside a list item escapes the item: #23336 (comment)
A diagram in an outer list item inherits the nested preceding paragraph's depth: #23336 (comment)
A custom heading ID prevents the heading match and moves the diagram to the end of the page: #23336 (comment)
The anchor search crosses the next section boundary: #23336 (comment)

The URL: configuration-example split is fixed at this head. The fixture from #23336 (comment) now builds, renders correctly, and passes all four unchanged Node corpus tests. The related Copilot thread at #23336 (comment) is also addressed.

Validation on the unmodified PR passes: production Docusaurus build with Node 24.3.0, all 210 Python documentation-script tests, 12 Node tests, TypeScript, the four-artifact freshness check for 72 pages, and git diff --check.

All seven separate MDX fixtures also build successfully. Five focused rendering assertions pass with the merge-base generator (c5f2fb08b4e7e68ab888e3ff36f59df9cf63b830) and fail with this head. The correctly contained diagram passes the rendering assertions but fails the unchanged containment test.

Only the first diagram of a consecutive run has prose to anchor to: the line before each of the others closes the previous fence, so they reach splice_diagram with no `preceding` and fall back to the heading. Each then landed immediately after that heading, which put the run in reverse order and ahead of the paragraph introducing it — a page written as intro, first, second came out as heading, second, intro, first. Place such a diagram after the last one already spliced into the same section instead. The search for it is bounded at the next heading of the same or a shallower level, so a run can neither adopt the next section's diagram nor place its own past the section it was written in.

The check reads the number of contained fenced blocks off the built HTML and compares it with the corpus. Docusaurus draws mermaid diagrams client-side, so the built page carries no `<pre>` for them while the corpus carries the fence the generator splices back. A diagram correctly indented into its list item therefore made the two totals disagree, and the check failed on documentation that was right. Take the count of diagrams written inside a container from the sources and add it to the expectation, so both sides describe the same set of blocks, and cover a correctly contained diagram directly.

A diagram is absent from the built page, so the generator reconstructed where it sat: the prose paragraph it followed, and the list item or blockquote it was written in. Both were inferred, and both got it wrong in ways that took a prose-anchor search bounded only at its start and a container model carried as a boolean — a heading with a custom ID, a bullet marker the build rewrites, a quote nested in a list item, a diagram one list level away from the paragraph above it. None of those shapes exist here. This corpus holds one diagram, at the top level, under a heading, with no prose anchor, and the output is byte-identical without any of it. Place a diagram at the top of its section and stop there: a run keeps its source order, and the section's prose follows it. Removes _insert_after, _anchor_key, _continuation_prefix, _blank_in and _indent_block; mermaid_blocks yields (heading, occurrence, block).

…rability-main # Conflicts: # docs/site/static/llms-full.txt # llms-full.txt

bloxster · 2026-09-16T09:18:18Z

Merged main and regenerated at 96b3ca9cfa — the branch was conflicting on llms-full.txt again after #24012 landed. Only that commit's two edits moved in the corpus: the x86-64-v2 baseline note on the Docker page and the GCC 11 prerequisite. 72 pages in, 72 out.

--check, 200 Python tests and 12 Node tests pass against a production build.

@yperbasis the three findings from your review yesterday are in at 52856c9d3c and each thread is resolved.

Copilot

🟡 Changes recommended

Fence parsing can alter or lose Mermaid content, and version masking can hide real documentation drift.

Get a fresh assessment by requesting another Copilot review.

Review details

Files not reviewed (1)

docs/site/package-lock.json: Generated file

Files reviewed: 10/13 changed files
Comments generated: 3
Review effort level: Balanced

yperbasis

Requesting changes at 96b3ca9cfad2e2612de2e3a451ee7750eec970db for four P2 regressions. I read all discussion comments, submitted reviews, and inline comments, including the latest Copilot review.

[P2] Preserve blockquotes nested inside list items (docs/site/scripts/generate-llms.py:594-601). _apply_indent always puts list indentation after the quote marker. A quote nested inside step 1 is emitted as > Quote inside step. instead of > Quote inside step., so its shell block loses its list-item ancestor and the ordered list is split. The resolved finding at #23336 (comment) still reproduces. Removing the Mermaid placement code does not remove this HTML conversion path. Please preserve the order of list and quote containers and cover both nesting orders.
[P2] Match custom heading IDs to rendered headings (docs/site/scripts/generate-llms.py:1033-1040). Source ## Flow {#custom-flow} becomes ## Flow in the built HTML. The exact comparison fails and appends the diagram after ## Later section. This still violates the revised contract that a diagram stays at the start of its own section. The resolved finding at #23336 (comment) still reproduces. Please normalize heading identity on both sides, including the occurrence count.
[P2] Preserve literal greater-than signs in list-contained diagrams (docs/site/scripts/generate-llms.py:965). Peeling a same-line list marker changes opener, which incorrectly enables _peel_quote on the diagram body. For example:

- ```mermaid
  flowchart TD
  A["first line
  > second line"]
  ```

The generator changes the label to second line, dropping >. The installed MDX compiler and Docusaurus Mermaid plugin preserve it. Please enable quote peeling only when the opener is actually quoted. This confirms #23336 (comment).

[P2] Preserve the literal version prefix in drift comparisons (docs/site/scripts/generate-llms.py:1501-1502). The optional v? makes erigon:v3.6.1 and erigon:3.6.1 compare equal, even without a release bump. A focused check_outputs test reports both changed full-corpus files as version drift instead of stale. The prefix is source text outside {ERIGON_VERSION}; changing it must require artifact regeneration. Please mask only the version digits. This confirms #23336 (comment).

Validation on the unmodified PR passes: production Docusaurus build on Node 24.3.0, all 200 Python script tests, 12 Node tests, TypeScript, the four-artifact freshness check for 72 pages, and git diff --check.

Nine separate MDX fixture pages also build successfully. The three rendering assertions above pass with the base generator (d831b85a92400250abfbbb9c3651d0c1b297900e) and fail with this head. The version-prefix assertion fails with this head; the base correctly reports two stale files. Consecutive diagram order and the previous section-boundary case pass under the revised placement contract.

The fence/comment finding at #23336 (comment) also reproduces, but the same fixture loses its diagram with the base generator. I am treating that as a separate follow-up, rather than a new regression blocking this PR.

…rability-main # Conflicts: # docs/site/static/llms-full.txt # llms-full.txt

Four rendering and comparison defects, each reproduced first and covered by a test that fails on the commit before this one. A list item's indentation was always written after the blockquote marker, so a quote nested inside an item was emitted at column zero, which closed the item and split the list. Each item now records the quote depth it opened at, and its indentation goes back there: after the markers that were already around it, in front of the ones opened inside it. Both nesting orders, and the two combined, are covered. A custom heading id is the heading's id, not its text — the built page carries `Flow` for a source `## Flow {#custom-flow}` — so matching on the source text found no such heading and appended the diagram instead. Heading identity is now normalised on both sides, including the occurrence count. Peeling a same-line list marker also changed the opener, which enabled quote peeling on the diagram body and dropped a legitimate `>` from a label. The quote is now tested for directly. Masking the optional `v` made `erigon:v3.6.1` and `erigon:3.6.1` compare equal, so an edit that added or dropped the prefix passed as release drift and never forced regeneration. Only the digits are masked.

bloxster · 2026-09-16T11:38:34Z

Four findings addressed at 9d01f165b4, each reproduced first and covered by a test that fails on the commit before it.

1. Blockquote inside a list item. The indent was applied relative to the quote marker rather than to where the item opened, so only one nesting order could be right at a time. Each item now records the quote depth its marker was written under, and its indent goes back there — after the markers already around it, before the ones opened inside it. Your page comes out 1. Run this: / > Quote inside step. with the list intact. Both orders and the two combined are covered; the existing quoted-list test is unchanged.

2. Custom heading ids. ## Flow {#custom-flow} renders as Flow, so the source text matched nothing in the built page. Heading identity is normalised on both sides, and the occurrence count is taken after normalising, so ## Flow {#first} and a later ## Flow are sightings one and two of the same heading.

3. Literal > in a listed diagram. Peeling a same-line list marker also changed the opener, which is what enabled quote peeling. The quote is now tested for directly, so a > inside a label survives.

4. Version prefix. Only the digits are masked; the v is carried through. erigon:v3.6.0 and erigon:3.6.0 no longer compare equal.

The fence/comment finding is left as the follow-up you called it — it reproduces on the merge base too, and the fixture needs more than the closer test to come out whole.

Also merged main at b218e543aa and regenerated: the branch was conflicting on llms-full.txt after #23972 and #23717 landed. Only the db.read.concurrency text and the --rpc.slow exclusion list moved in the corpus; 72 pages in, 72 out.

--check, 207 Python tests, 12 Node tests, TypeScript and git diff --check pass against a production build.

Copilot

🟡 Changes recommended

The corpus test incorrectly treats unrelated URL-prefixed prose as page sentinels.

Get a fresh assessment by requesting another Copilot review.

Review details

Files not reviewed (1)

docs/site/package-lock.json: Generated file

Files reviewed: 10/13 changed files
Comments generated: 1
Review effort level: Balanced

yperbasis

Requesting changes at 9d01f165b4b41d0ec8acf7a37af3a208b19b9d20. I read all discussion comments, submitted reviews, and inline comments, including the latest Copilot review.

Three new P2 findings are described inline:

Literal Markdown in HTML text becomes active syntax: an escaped-backtick paragraph opens a code fence and absorbs the following page.
Inline formatting changes prevent heading matches, so Mermaid diagrams are appended under a later section.
The containment counter treats shorter fences inside a longer example as separate blocks and fails on correctly rendered documentation.

I also confirmed the existing P2 sentinel-count finding at #23336 (comment). A valid prose paragraph containing URL: https://rpc.example builds and generates correctly, but the unchanged corpus test fails with 2 !== 1. Filter the top-level URL paragraphs through the known sentinel set before comparing their count with the H1 count.

The latest four fixes hold: both list/blockquote nesting orders, custom heading IDs, literal greater-than signs in list-contained diagram labels, and the version's literal v prefix. The fence/comment issue also reproduces with the merge-base generator and remains a separate follow-up. These findings use the revised contract that a diagram goes at the start of its section.

Validation on the unmodified PR passes: production Docusaurus build with Node 24.3.0, all 207 Python documentation-script tests, 12 Node tests, TypeScript, the four-artifact freshness check for 72 pages, and git diff --check.

Fifteen separate MDX fixture pages also build successfully. Three focused rendering assertions pass with the merge-base generator and fail with this head. The unchanged containment and sentinel tests reproduce the two false failures above.

The sentinel count took every top-level paragraph opening "URL: https://", so a configuration example written as prose — documentation, outside any fence — counted as a page boundary and failed the corpus check on a corpus where nothing had been lost. The paragraphs are now filtered through the set of URLs llms.txt lists, which the fence check already used. A page that does lose its own sentinel still drops below the heading count and fails.

Three defects, each reproduced first and covered by a test that fails on the commit before this one. Ordinary text was written into Markdown as it stood, so what a page escaped in its source became syntax again. A paragraph saying ```sh reopened as a fence, which runs to the end of the document: it swallowed the rest of the page and the next page's sentinel, and the corpus lost a page with every check still passing. Backtick and tilde runs are now escaped wherever they occur in prose, and a block marker at the start of a line with it. The delimiters this converter emits for HTML elements and code go into the buffer directly and never pass through that path, so they are untouched, and link text is not a line start — its own bracket is written in front of it. Heading identity compared source spelling against built spelling: `_Flow_` is written `*Flow*` in the build, and a linked heading carries a resolved target. Both comparisons failed and the diagram was appended to the end of the page. Both sides are now reduced to what the heading says — custom id, link target and inline markup removed — and the occurrence count is taken from that same form. The containment counter flipped a boolean on every fence-looking line, so a Markdown example displayed inside a longer fence counted as a second contained block and valid documentation failed CI. It now tracks the delimiter: only a run of the same character, at least as long as the opener, closes the block.

Only ASCII punctuation can be backslash-escaped (CommonMark 2.4), so `\1.` suppresses the list but renders the backslash as text — the installed engine reads it back as the literal `\1. text`. `1\.` is the form that both suppresses the list and reads back as what the page wrote.

bloxster · 2026-09-16T13:21:44Z

Four findings addressed across 331589b7b4, 266224470e and 2280393a5d, each reproduced first and covered by a test that fails on the commit before it.

Sentinel count. Top-level URL: paragraphs are filtered through the set of URLs llms.txt lists — the set the fence check above already used — so a prose URL: https://rpc.example is page content, not a boundary. A page that loses its own sentinel still drops below the heading count.

Literal Markdown in prose. Ordinary text went into the corpus as it stood, so what a page escaped became syntax again and a reopened fence ran to the end of the document. Backtick and tilde runs are escaped wherever they occur in prose, and a block marker when the line is at its start; the delimiters emitted for HTML elements and code never pass through that path.

Two boundaries there were worth getting right. Link text is not a line start — [#1516](…) writes the # behind its own bracket — and escaping it there was the one change the real corpus showed, which is how I found it. And only ASCII punctuation can be backslash-escaped (CommonMark 2.4), so an ordered marker is escaped on its delimiter: the installed engine reads \1. text back as literal \1. text, while 1\. text reads back clean.

Heading identity. Both sides are reduced to what the heading says — custom id, link target and inline markup removed — and the occurrence count comes from that same form, so _Flow_ matches *Flow* and a site-relative link matches a resolved one.

Containment counter. It tracks the delimiter now, so a Markdown example inside a longer fence is content rather than a second contained block.

The corpus is byte-identical: all four are latent for these 72 pages, which is why none of them showed up in a regeneration.

--check, 217 Python tests, 13 Node tests, TypeScript and git diff --check pass against a production build.

Copilot

🔵 Needs a closer look

The custom HTML-to-Markdown conversion and corpus-wide CI behavior warrant final human validation despite substantial tests.

Review details

Files not reviewed (1)

docs/site/package-lock.json: Generated file

Files reviewed: 10/13 changed files
Comments generated: 0 new
Review effort level: Balanced

yperbasis

Requesting changes at 2280393a5d3b65d6eeff06a4bb03a8475e249a14 for two P2 regressions described inline. I read all 30 discussion comments, 166 inline comments, and 137 submitted reviews, including the latest Copilot review.

Literal HTML in prose becomes active markup and captures following page boundaries.
Markdown escapes and entities in headings still prevent Mermaid section matching.

Validation of the unmodified head passes: production Docusaurus build on Node 24.3.0, all 217 Python documentation-script tests, 13 Node tests, TypeScript, the four-artifact freshness check for 72 pages, and git diff --check.

The separate MDX fixtures also build successfully. Both heading-placement assertions pass with the merge-base generator (184d76c9c67890e3ba83a3e6c292a128a8066f2e) and fail with this head. Running the five unchanged Node corpus tests against two generated fixture entries passes with the base generator; this head fails because the literal HTML block captures the following page sentinel.

These findings use the revised contract that a Mermaid diagram belongs at the start of its section.

Two defects, each reproduced first and covered by a test that fails on the commit before this one. HTMLParser decodes character references, so a page that wrote `<script>` to display a tag handed the prose path a live `<script>`. Written raw it opens an HTML block that runs to the next blank line and swallows the page boundary inside it — the same way a reopened fence does. A `<` in prose is now escaped, and an `&` when it begins a character reference; a plain ampersand is prose and is left alone. One line of the corpus changes: `where <CL_host> is` now reads `where \<CL_host> is`, which is the text the built page shows. Heading identity compared a source spelling against a rendered one: source `## trace\_call` renders as `trace_call`, and `## Request & response` as `Request & response`. Escapes are now unwound and references decoded before the markup is removed, so both sides reduce to the same text and the occurrence count is taken from it.

bloxster · 2026-09-17T09:06:34Z

Both findings addressed at f7f96bdb1b, each reproduced first and covered by a test that fails on the commit before it.

Literal HTML in prose. HTMLParser decodes character references, so a page that wrote <script> to display a tag handed the prose path a live <script>, which opens an HTML block and swallows the boundary inside it. A < is escaped now, and an & when it begins a character reference; a plain ampersand is prose and is left alone. This one was live in the corpus, not only in your fixture: where <CL_host> is now reads where \<CL_host> is, which is the text the built page shows — one line, and the only change in 72 pages.

Heading escapes and entities. Escapes are unwound and references decoded before the markup is removed, so trace\_call reduces to the same key as the rendered trace_call, and Request & response to the same as Request & response. The order matters: the backslash has to go while it still stands in front of the underscore, or the source side keeps a stray \ where the rendered side keeps nothing. The occurrence count is taken from that same form.

--check, 221 Python tests, 13 Node tests, TypeScript and git diff --check pass against a fresh production build.

Copilot

🔵 Needs a closer look

The large custom HTML-to-Markdown conversion and corpus integrity logic warrant final human validation against a complete production build.

Review details

Files not reviewed (1)

docs/site/package-lock.json: Generated file

Files reviewed: 10/13 changed files
Comments generated: 0 new
Review effort level: Balanced

yperbasis

Requesting changes at f7f96bdb1ba8db14bd0c93e60c3e92be22c92d62 for two P2 regressions described inline:

Literal Markdown punctuation becomes active syntax, hiding a reference-shaped paragraph and changing literal emphasis characters and backslashes.
A generated page H1 shifts the source heading occurrence count and moves a Mermaid diagram outside its section.

I read all 31 discussion comments, 170 inline comments, and 140 submitted reviews.

Validation of the unmodified head passes: production Docusaurus build on Node 24.3.0, all 221 Python documentation-script tests, 13 Node tests, TypeScript, the four-artifact freshness check for 72 pages, and git diff --check.

Five separate MDX fixture pages also build successfully. Four focused assertions fail with this head and pass with the merge-base generator (184d76c9c67890e3ba83a3e6c292a128a8066f2e). The latest literal-HTML, escaped-underscore, and heading-entity fixes pass their controls.

These findings use the revised contract that a Mermaid diagram belongs at the start of its section.

…evel Two defects, each reproduced first and covered by a test that fails on the commit before this one. Prose still wrote brackets, emphasis markers and backslashes live. A paragraph reading `[endpoint]: https://rpc.example` is a link definition, which renders as nothing at all, and a literal backslash escapes whatever follows it. Everything a page can write as text is now escaped in one pass — escaping in two would backslash the backslashes the first pass added. Two limits keep that from rewriting the corpus. An underscore between two alphanumerics opens no emphasis, and half this corpus's RPC methods are written `eth_getProof`, so only one with a boundary beside it is escaped. And text arrives in chunks — `erigon_v{ERIGON_VERSION}_linux` is three of them — so the character already written stands in as the left context and an alphanumeric as the right, which keeps a chunk edge from reading as a word boundary. Four lines of the corpus change: a literal `*`, two literal bracket pairs, and a trailing backslash that was being read as a hard break. Heading occurrences counted every heading with the same text, at any level. Docusaurus gives a page titled from front matter a generated `# Title` with no source heading, so a diagram under `## Title` was placed under that H1 — above its section and above the prose introducing it. The level is recorded with the heading and counted with it, and only a heading at that level can match.

bloxster · 2026-09-17T11:41:23Z

Both findings addressed at 3c66b4ff7f, each reproduced first and covered by a test that fails on the commit before it.

The rest of Markdown in prose. Brackets, emphasis markers and the backslash itself are escaped now, in one pass with the rest — escaping in two puts a backslash in front of the backslash the first pass added. [endpoint]: https://rpc.example stays visible instead of reading as a link definition and rendering as nothing.

Two limits keep that from rewriting the corpus, both found by watching what regeneration actually changed. An underscore between two alphanumerics opens no emphasis and half this corpus writes eth_getProof, so only one with a boundary beside it is escaped — the first attempt put a backslash through every RPC method name. And text arrives in chunks, erigon_v{ERIGON_VERSION}_linux being three of them, so the character already written stands in as the left context and an alphanumeric as the right; without that a chunk edge reads as a word boundary. Four lines change in the end: 3.1.\*, two literal bracket pairs in the trace docs, and a trailing \ that was being read as a hard line break.

Heading occurrences by level. Docusaurus gives a page titled from front matter a generated # Title with no source heading, so counting by text alone placed a ## Title diagram under that H1 — above its section and above the prose introducing it. The level is recorded with the heading and counted with it, and only a heading at that level can match. mermaid_blocks returns (heading, level, occurrence, block); splice_diagram takes the level and still matches at any level when it is not given.

--check, 225 Python tests, 13 Node tests, TypeScript and git diff --check pass against a fresh production build.

Copilot

🔵 Needs a closer look

The large custom HTML-to-Markdown conversion pipeline warrants final human validation despite its extensive test coverage.

Review details

Files not reviewed (1)

docs/site/package-lock.json: Generated file

Files reviewed: 10/13 changed files
Comments generated: 0 new
Review effort level: Balanced

Read heading text and level from the same match so blank and comment-only headings do not crash Mermaid scanning. Add regression coverage for empty headings on pages without diagrams.

bloxster · 2026-09-17T12:33:37Z

Thanks for 88ffe48c66 — that was a real defect in 3c66b4ff7f, not a style point. _ATX_LEVEL_RE.match(...) returns None on a heading _ATX_HEADING_RE accepts but it does not, so mermaid_blocks raised AttributeError on an empty heading; running the previous commit against ## reproduces it. Taking the level from the heading match itself removes the second pattern and the disagreement between them. Verified on the new head: 226 Python tests, 13 Node tests and --check (72 pages) pass, and the corpus is unchanged.

One CI note: hive / test-hive (ethereum/engine, api, parallel) failed on this head with Invalid Missing Ancestor Syncing ReOrg … Unable to customize payload: no transactions available, 1 of 129. This PR touches only docs/site/**, the two docs workflows and the generated artifacts, so it cannot reach engine-API behaviour — it reads as a flake. GitHub would not let me rerun the job while the rest of the run was still going; I will retry once it finishes.

bloxster requested review from AskAlexSharov and yperbasis as code owners August 17, 2026 09:11

bloxster added the docs label Aug 17, 2026

bloxster mentioned this pull request Aug 17, 2026

docs(site): make the llms.txt artifacts discoverable, and fix the landing-card parser #23335

Merged

yperbasis requested a balanced review from Copilot August 17, 2026 11:11

Copilot started reviewing on behalf of yperbasis August 17, 2026 11:11 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/docs/get-started/why-using-erigon.mdx Outdated

Comment thread docs/site/docusaurus.config.ts Outdated

yperbasis requested changes Aug 17, 2026

View reviewed changes

AskAlexSharov approved these changes Aug 17, 2026

View reviewed changes

AskAlexSharov enabled auto-merge August 17, 2026 11:39

AskAlexSharov disabled auto-merge August 17, 2026 11:39

yperbasis requested a balanced review from Copilot August 17, 2026 12:56

Copilot started reviewing on behalf of yperbasis August 17, 2026 12:57 View session

bloxster mentioned this pull request Aug 17, 2026

docs(site): address review on #23335 — describedby relation, guard hole, corpus claims #23341

Merged

Copilot AI reviewed Aug 17, 2026

View reviewed changes

yperbasis requested changes Aug 17, 2026

View reviewed changes

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/docusaurus.config.ts Outdated

Comment thread docs/site/docs/fundamentals/mcp.mdx Outdated

bloxster requested a review from yperbasis August 18, 2026 06:47

yperbasis requested a balanced review from Copilot August 18, 2026 09:37

Copilot started reviewing on behalf of yperbasis August 18, 2026 09:37 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

yperbasis requested changes Aug 18, 2026

View reviewed changes

Comment thread docs/site/docusaurus.config.ts Outdated

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/scripts/generate-llms.py Outdated

yperbasis requested a balanced review from Copilot August 18, 2026 13:41

Copilot started reviewing on behalf of yperbasis August 18, 2026 13:42 View session

Copilot AI reviewed Sep 15, 2026

View reviewed changes

yperbasis requested changes Sep 15, 2026

View reviewed changes

Comment thread docs/site/scripts/test_generate_llms.py

Comment thread docs/site/scripts/generate-llms.py Outdated

Bloxster added 4 commits September 15, 2026 18:50

Merge remote-tracking branch 'origin/main' into docs/llms-txt-discove…

96b3ca9

…rability-main # Conflicts: # docs/site/static/llms-full.txt # llms-full.txt

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Comment thread docs/site/scripts/generate-llms.py

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/scripts/generate-llms.py Outdated

yperbasis requested changes Sep 16, 2026

View reviewed changes

Bloxster added 2 commits September 16, 2026 13:20

Merge remote-tracking branch 'origin/main' into docs/llms-txt-discove…

b218e54

…rability-main # Conflicts: # docs/site/static/llms-full.txt # llms-full.txt

Copilot AI reviewed Sep 16, 2026

View reviewed changes

Comment thread docs/site/src/llms-corpus.test.mjs Outdated

yperbasis requested changes Sep 16, 2026

View reviewed changes

Comment thread docs/site/scripts/generate-llms.py

Comment thread docs/site/scripts/generate-llms.py

Comment thread docs/site/scripts/test_generate_llms.py Outdated

Bloxster added 3 commits September 16, 2026 15:02

Copilot AI reviewed Sep 17, 2026

View reviewed changes

yperbasis requested changes Sep 17, 2026

View reviewed changes

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/scripts/generate-llms.py

Copilot AI reviewed Sep 17, 2026

View reviewed changes

yperbasis requested changes Sep 17, 2026

View reviewed changes

Comment thread docs/site/scripts/generate-llms.py Outdated

Comment thread docs/site/scripts/generate-llms.py

Copilot AI reviewed Sep 17, 2026

View reviewed changes

docs/site: handle empty headings in corpus generation

88ffe48

Read heading text and level from the same match so blank and comment-only headings do not crash Mermaid scanning. Add regression coverage for empty headings on pages without diagrams.

yperbasis approved these changes Sep 17, 2026

View reviewed changes

Conversation

bloxster commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

bloxster commented Aug 17, 2026

Uh oh!

bloxster commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bloxster commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Files not reviewed (1)

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bloxster commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Files not reviewed (1)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

bloxster commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Files not reviewed (1)

Uh oh!

Uh oh!

yperbasis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bloxster commented Aug 17, 2026 •

edited

Loading

bloxster commented Sep 16, 2026 •

edited

Loading