feat: report degraded constructs during markdown parsing - #3239
Conversation
🦋 Changeset detectedLatest commit: ff24b79 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Bundle StatsWarning 1 significant change. @portabletext/markdown🔴 All scenario measurements (7)🗺️
Significant means at least 1.0 KB and 1% gzip, or at least 5 ms and 10% import time. |
2b733a2 to
bb7f293
Compare
bb7f293 to
64aa2f3
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
64aa2f3 to
90ef502
Compare
90ef502 to
3dfa2de
Compare
christianhg
changed the title
3dfa2de to
094bab2
Compare
6610153 to
e863abd
Compare
e863abd to
a836f34
Compare
a836f34 to
b829249
Compare
b829249 to
648172c
Compare
648172c to
93f4ee7
Compare
93f4ee7 to
be0464a
Compare
Schema-gating degrades markdown constructs gracefully by design: an
undeclared decorator drops its formatting, a table flattens into
blocks, an image becomes its markdown source as text. Right for
rendering pipelines, invisible to everyone else. Agent tooling that
patches rich text through `markdownToPortableText` needs the losses
loud, and its accidental feedback channel (schema validation rejecting
output converted against the wrong schema) disappears once conversion
runs against the correct schema.
One new option: `onDegradation`, a plain callback called at most once,
after the walk, only when at least one construct degraded. Its single
report argument (the extensibility joint) holds `degradations`, every
`Degradation` in encounter order (`{type, message, line?, snippet?}`,
both types exported, `type` the literal union to match on, `message`
human-readable and unstable by contract, `snippet` the offending
construct's text truncated to 40 characters), and `message`, the
canonical text written for the LLM retry loop that reads it: effect
stated in content terms, the cause as a constraint naming the missing
schema declaration, quoted snippets as anchors (models match strings
better than they count lines), identical degradations grouped into one
line. Enforcement is ordinary JavaScript, not a library mode: throw
your own error from the callback and it propagates out of the
conversion; pinned by a test. With the option unset, conversion
degrades silently, which also removes the legacy behavior of eleven
style-fallback sites calling `console.warn`: a library does not log,
visibility is opt-in.
Every degradation site in the parser reports, because a callback that
never fires is a totality guarantee, and only real loss reports,
verified from both directions by schema-mutation fuzzing: decorator
and annotation drops, style fallbacks including the double-decline
tiers, list flattening (fired at item resolution, so a task-only
schema converting a pure task list stays silent), task checkbox
stripping at its consumption site, table flattening, code blocks,
horizontal rules, and HTML blocks falling back to text, inline HTML
dropped under the default skip mode, image fallbacks in all three
directions (to text, block-to-inline, inline-to-block, the table-cell
demotion deferred to `td_close` where the sole-image lift is known),
schema field filtering that would otherwise silently destroy content
a declared object cannot carry (`fields-dropped`, reported
generically when a default matcher's field filtering loses supplied
keys), and callout fallbacks deferred to actual text-block creation.
The `json:object` carrier reports its own failures: a fence or tagged
code span whose payload is not a JSON object with a `_type` falls back
to ordinary code and reports `object-carrier-invalid`, naming whether
the JSON or the `_type` was the problem.
Structural-matcher declines mirror the flat paths byte-for-byte and
report only what the flat path itself would report: a decline whose
fallback is the canonical flat form is not a degradation. Snippets
never split surrogate pairs, escape embedded newlines, and grouped
message lists cap at five entries. Graceful block output is
byte-identical to before, verified across default, sparse, and empty
schemas.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ff24b79. Configure here.
| level, | ||
| ...(itemChecked === undefined ? {} : {checked: itemChecked}), | ||
| } | ||
| pushBlock(mergeTarget) |
There was a problem hiding this comment.
List decline drops paragraph provenance
Medium Severity
List decline spreads each stampable paragraph into a new object and never puts that object back in plainParagraphBlocks. An enclosing blockquote decline then skips restyle, so a list inside a blockquote where both matchers return undefined keeps style: 'normal' instead of the flat path's blockquote style. The opposite nesting already re-adds provenance after restyle.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ff24b79. Configure here.
| ...(block as PortableTextTextBlock), | ||
| style: blockquoteStyle, | ||
| }) | ||
| style: fallbackStyle, |
There was a problem hiding this comment.
Blockquote decline uses wrong fallback style
Medium Severity
When both the blockquote and normal matchers decline, this path reports a normal style-fallback and says the quote became normal paragraphs, then assigns the literal style blockquote. The flat blockquote_open path uses normal as that last resort, so the new report names a style the emitted blocks do not have.
Reviewed by Cursor Bugbot for commit ff24b79. Configure here.
| consolidatedOptions.block.blockquote({ | ||
| context: {schema: consolidatedOptions.schema}, | ||
| }) ?? | ||
| // Matcher returned undefined: fall back to exactly what the |
There was a problem hiding this comment.
Accepted list and blockquote skip field reports
Medium Severity
Accepted types.blockquote and types.list objects are pushed without reportFieldsDropped. A default matcher that drops content or items because those fields are absent from the schema therefore stays silent, while the same loss on callout, table, image, code, html, and link is reported.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ff24b79. Configure here.
1 participant
Schema-gating in
markdownToPortableTextdegrades gracefully by design: an undeclared decorator drops its formatting, a table flattens, an image becomes its markdown source as plain text. Right for rendering pipelines, and invisible. Agent tooling patching rich text through this conversion needs the losses loud, an agent that wrote a table must learn the table didn't survive, and the accidental feedback channel it relied on (schema validation rejecting output converted against the wrong schema) disappears as soon as conversion runs against the correct schema.One new option:
onDegradation, a plain callback called at most once, after the walk, only when something degraded. Its report argument holdsdegradations, the encounter-ordered structured list ({type, message, line?, snippet?},typea literal union to match on,messageexplicitly unstable,snippetthe offending construct's text), andmessage, the canonical text written for the retry loop that reads it: each degradation states its effect in content terms, names the missing schema declaration as a constraint, quotes a snippet as the anchor (LLMs match strings far more reliably than they count lines), and identical degradations group into one line, forty undeclared-bold spans read as one rule. Enforcing is ordinary JavaScript rather than a library mode:onDegradation: ({message}) => { throw new Error(message) }propagates out of the conversion, pinned by a test. With the option unset, conversion degrades silently, which also removes the legacyconsole.warnon the eleven style-fallback paths: a library shouldn't log, visibility is opt-in.A callback that never fires is a totality guarantee, so every site in the parser reports, including four the reviews found silent (structural-list paragraph styles, callout content styles, and the double-decline tiers on headings and flat blockquotes), the table-cell image demotion, and the task-checkbox strip at its consumption site so a structural path that preserves
checkedstays silent. Graceful block output is byte-identical to before, verified across default, sparse, and empty schemas, and pinned by the untouched suite.Degradationis the only new export; itstypefield carries the literal union, which grows as new degradation sites report, so consumers compare against the values they handle rather than switching exhaustively. The option lives on the non-exported options type. The API went through four design iterations before this shape; the decision record with the rejected alternatives lives on the ticket.Note
Medium Risk
Touches the core markdown-to-PT conversion path and changes observable logging (warn → silent by default); output shape is intended unchanged, but regression risk is moderate given breadth of degradation sites.
Overview
Adds
onDegradationtomarkdownToPortableText: when markdown can’t be represented in the schema (missing decorators, flattened tables, stripped task checkboxes, invalidjson:objectcarriers, dropped object fields, and similar), conversion still returns the same lossy Portable Text as before, but can report every loss in one post-walk callback.The report includes an encounter-ordered
degradationsarray (exportedDegradationtype with stabletype, optionalline/snippet) and a grouped, line-sortedmessagefor humans or agents. Callers can log, match ontype, or throw from the callback to fail the import. Without the option, behavior stays silent—console.warnon style-fallback paths is removed in favor of opt-in reporting.Implementation wires reporting through the markdown-it walk (new message catalog, snippet truncation, grouping rules) and extends default matchers to tag fields dropped when filtering to schema fields. Docs, README, changeset, and a large test suite cover edge cases (structural list/blockquote declines mirroring the flat path, table-cell images, callouts).
Reviewed by Cursor Bugbot for commit ff24b79. Bugbot is set up for automated code reviews on this repo. Configure here.