Repository navigation
fix(markdown): keep repeated body text on short pages - #601
mangeshraut712 wants to merge 4 commits into
Conversation
A page with too few rows to leave an interior was treated as all margin, so sentences repeated across a large-print document were stripped. Count near-coincident column baselines as one row, and skip that page when the margin zones would meet.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Skipping every line on a short page also kept a real running header. Cluster baselines once, and on a page too short for the full zone treat only the outermost row as furniture.
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Fixes the markdown furniture filter so short pages keep repeated body text: baseline rows are clustered at 0.1pt tolerance and only the outermost row is stripped on short pages. The change is confined to strip_repeated_lines and preserves tall-page handling.
Re-trigger cubic
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| // On a short page the full N-row zones meet and would mark the body | ||
| // as furniture. Keep only the outermost row, which is still enough | ||
| // for a running header or footer. Tall pages keep the N-row zone. | ||
| let edge = if count <= n * 2 { 1 } else { n }; |
There was a problem hiding this comment.
🔴 Eleven-row pages lose repeated prose
On eleven-row pages, edge marks ten rows as margins instead of two. If prose repeats on three pages at fixed baselines, nine or ten body rows disappear from later pages.
Learn more
The repeated-line classifier counts distinct clustered baselines and uses an edge zone to select header/footer candidates. At ten rows, only the first and last rows count as edges; at eleven, five rows from each end count. Repeated ordinary lines at fixed Y coordinates then pass the frequency and Y-consistency checks in strip_repeated_lines and are removed after their first page. This leaves only the central row on subsequent pages.
Example: Three eleven-row pages contain the same eleven-line passage at Y=700, 680, ..., 500. All eleven lines remain on page 1, but pages 2 and 3 each retain only the sixth line.
Recommended fix: Avoid the ten-to-eleven-row jump in is_y_at_edge. Scale or bound the edge zone so it cannot cover nearly all of a short page, and test repeated prose on eleven- and twelve-row pages.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if count <= 2 { | ||
| return false; |
There was a problem hiding this comment.
🟡 Two-row pages retain every repeated header
When every page has a header and one body row, is_y_at_edge rejects every header. No header becomes a candidate, so it appears on every page.
Learn more
The document-wide classifier first gathers candidates from lines it considers at the edge, then removes repeated candidates after their first occurrence. A one- or two-row page contributes no candidates, even when the same header repeats across the document. The later sparse_page exception in strip_repeated_lines can remove only headers that became candidates on other pages, so it cannot handle a document consisting entirely of two-row pages.
Example: Six pages each have Quarterly Bulletin running header text at Y=760 and a different body line at Y=700. The frequency map remains empty, and all six headers survive instead of one.
Recommended fix: Allow sufficiently corroborated repeated headers to be identified on sparse pages without classifying all their rows as furniture. Keep the body line protected by requiring independent evidence, such as stable header position and repetition across pages.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // On a short page the full N-row zones meet and would mark the body | ||
| // as furniture. Keep only the outermost row, which is still enough | ||
| // for a running header or footer. Tall pages keep the N-row zone. | ||
| let edge = if count <= n * 2 { 1 } else { n }; |
There was a problem hiding this comment.
🟡 Short-page headers lose only their first line
On three- to ten-row pages, edge admits only the outermost header row. A repeated two-line header loses its first line, but leaves its second line on every later page.
Learn more
The edge predicate controls both candidate collection and removal in strip_repeated_lines. On pages with three through ten clustered rows, only the first and last rows can enter either pass. A repeated multi-line header therefore has no way to remove its inner header row, even when the outer header row is proven furniture.
Example: Six nine-row pages each have Journal of Testing at Y=760 and Monthly bulletin at Y=740, followed by seven unique body rows. The first header line remains only on page 1; Monthly bulletin remains on all six pages.
Recommended fix: Once an outer row is confirmed as furniture, allow a small adjacent header/footer block to be stripped using repetition and positional evidence, while retaining the short-page guard for ordinary repeated interior prose.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Focused markdown fix in strip_repeated_lines: short pages keep repeated body text because baselines are clustered and only the outermost row is stripped, while tall-page behavior and confirmed headers are preserved; tests pin the corrected cases.
View guided diff | Re-trigger cubic
This branch has not been deployed
1 participant
Fixes #483
Problem
strip_repeated_linestreats a line as margin furniture when it sits in the first or last 5 distinct baselines. When a page has 10 or fewer baselines, those two zones cover every line, and the function used to treat the whole page as margin.A large-print page that repeats the previous page's sentences then loses that body. The same page extracted alone keeps the text. Two-column lines whose baselines differ by a fraction of a point made a short page look tall enough for the full 5-row zone.
Change
Baselines within 0.1pt are one row, and that row list is built once per page. A page with one or two rows is not a margin. On any other short page only the outermost row can be a running header or footer, so the interior body stays and a real header on a short page still strips. Tall pages keep the 5-row zone.
Tests
cargo test --offline --lib markdown::furniture::tests— 27 passed, including a repeated short passage, a two-column page with a 0.05pt baseline offset, a header on a short page, and a header on an eleven-row page.cargo clippy --offline -- -D warningsis clean.