Skip to content

Commit 8494572

Browse files
committed
http-transport: a link's label is decoded once with the page, wherever the link stands
What changed: `inline()` in lib/http-transport.mjs no longer decodes a link's label inside the link pass; the label is kept as written and decoded once at the end with everything else, and its emptiness is judged on the decoded text, so a `&nbsp;`-only link is dropped in a list item as it was in a paragraph. `decodeEntities` decodes hex, decimal and named references in one pass instead of three in turn. The regex oracle in test/fixtures/regex-extract.mjs carries both rules. Why: the review of the G3 fix (output-reliability audit) found the page-level pass still decoding a link label on its own before the tag-stripping pass that follows it, so `<p>See <a href="/x">limit &lt; 10 and burst &gt; 2</a> now.</p>` came out `See [limit 2](/x) now.` and `<a>a &amp;lt; b</a>` came out `[a < b]` - the G3 defect, alive in the one place its test did not look (a paragraph or a div). Two smaller findings from the same review are folded in because they are the same rule: the emptiness test ran on the undecoded label, so `<li><a>&nbsp;</a></li>` kept a `[ ](/x)`; and `&#38;lt;` was decoded to `&lt;` and then to `<`, since the numeric passes ran before the named one. Alternative set aside: decoding the label and escaping its `<` for the stripping pass, which would make the converter's output depend on which pass a character came through. What it touched: research-kit/lib/http-transport.mjs (`decodeEntities`, `inline`), research-kit/test/fixtures/regex-extract.mjs (the oracle, same two rules), research-kit/test/transport.test.mjs (one test: paragraph and div links, the double-encoded label, the blank link, the numeric ampersand), docs/ARCHITECTURE.md (http-transport row), CHANGELOG.md, research-kit/README.md (1548 tests). No new term. What you verified: - the new test red before the fix: `See [limit 2](/x) now.` - verified. - `node research-kit/bin/selftest.mjs transport http-linear`: 106 passed, 0 failed, so the 3,000-page oracle equivalence still holds with the extended rules - verified. - the full suite runs in the commit gate on this commit - verified by the gate before this commit is accepted. - staged diff scanned for keys and the pasted token: 0 - verified. What you got wrong and fixed: - mistake: the new test expected `- item\n\npara` for a paragraph whose first child is a dropped blank link, but the converter leaves the space that followed the link, ` para`. - where: research-kit/test/transport.test.mjs, the blank-link assertion. - impact: a red assertion about behaviour this commit does not change; nothing delivered. - cause: I wrote the expectation from the fix's intent, not from the converter's output. - fix: the assertion matches the list item, the paragraph with an optional leading space, and the absence of any empty link. - verified: `selftest transport http-linear` 106 passed, 0 failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1fT6wSqD2LwRjsUuRvDau
1 parent ad2f5ef commit 8494572

6 files changed

Lines changed: 52 additions & 16 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in
3939
items, table cells and headings: `x &lt; 5 and y &gt; 2` had come out as `x 2`, and a link
4040
label inside an item lost everything from its `<` to the item's end, because the block
4141
pass decoded entities and the page-level pass then stripped the decoded `<...>` as a tag.
42-
Entities are decoded once, at the end (output-reliability audit G3, 2026-10-03).
42+
Entities are decoded once, at the end (output-reliability audit G3, 2026-10-03). The
43+
review of that fix found a link's label still decoded on its own before the tag-stripping
44+
pass, so `<a>limit &lt; 10 and burst &gt; 2</a>` in a paragraph came out `[limit 2](/x)`
45+
and `&amp;lt;` was decoded twice; the label is no longer decoded apart, a `&nbsp;`-only
46+
link is dropped in a list item as in a paragraph, and `&#38;lt;` decodes once, to `&lt;`.
4347
- A search provider whose `search()` throws is a failed search that says so (`<provider>
4448
threw: <message>`), degraded and recorded like any other, so the run and its accounting
4549
finish: re-thrown, it ended `research` and `decompose` mid-run, and the searches paid for

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎research-kit/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -453,7 +453,7 @@ node research-kit/bin/selftest.mjs # all of it
453453
node research-kit/bin/selftest.mjs gate hook # just these files
454454
```
455455

456-
1547 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
456+
1548 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
457457
the assertions settled (ADR-0021), and each test is raced against a watchdog
458458
(`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
459459

‎research-kit/lib/http-transport.mjs‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -193,10 +193,14 @@ function codePoint(n) {
193193

194194
export function decodeEntities(text) {
195195
const named = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', mdash: '—', ndash: '–', hellip: '…', rsquo: '’', lsquo: '‘', ldquo: '“', rdquo: '”' };
196-
return String(text)
197-
.replace(/&#x([0-9a-f]+);/gi, (_, hex) => codePoint(parseInt(hex, 16)))
198-
.replace(/&#(\d+);/g, (_, dec) => codePoint(Number(dec)))
199-
.replace(/&([a-z]+);/gi, (all, key) => named[key.toLowerCase()] ?? all);
196+
// One pass over every reference kind at once: three passes in turn decoded `&#38;lt;` to
197+
// `&lt;` and then to `<`, so a page spelling an ampersand numerically was read twice
198+
// (found 2026-10-03, review of the G3 fix).
199+
return String(text).replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (all, ref) => {
200+
if (/^#x/i.test(ref)) return codePoint(parseInt(ref.slice(2), 16));
201+
if (ref[0] === '#') return codePoint(Number(ref.slice(1)));
202+
return named[ref.toLowerCase()] ?? all;
203+
});
200204
}
201205

202206
/**
@@ -334,13 +338,16 @@ const LINK = { open: /<a\b/gi, tag: { find: /href=/gi, match: /href=["']([^"']*)
334338
/**
335339
* Inline markup to Markdown. `decode: false` leaves entities as written, for a body that a
336340
* later pass will strip tags from and decode (the block passes above); the page-level pass
337-
* decodes once.
341+
* decodes once. A link's label is never decoded on its own: the tag-stripping pass below
342+
* runs over it afterwards, and a `<` decoded first was read there as a tag (`[limit 2](/x)`
343+
* for `limit &lt; 10 and burst &gt; 2`, in a paragraph; found 2026-10-03, review of the G3
344+
* fix). Its emptiness is judged on the decoded text, so `&nbsp;` alone is as empty as a space.
338345
*/
339346
function inline(html, { decode = true } = {}) {
340347
const finish = (text) => (decode ? decodeEntities(text) : text);
341348
let text = replacePairs(String(html), LINK, (p) => {
342-
const label = finish(stripTags(p.body, '').text).trim();
343-
return label ? `[${label}](${p.tag[1]})` : '';
349+
const label = stripTags(p.body, '').text.trim();
350+
return decodeEntities(label).trim() ? `[${label}](${p.tag[1]})` : '';
344351
});
345352
text = replacePairs(text, named('strong|b'), (p) => `**${p.body}**`);
346353
text = replacePairs(text, named('em|i'), (p) => `*${p.body}*`);

‎research-kit/test/fixtures/regex-extract.mjs‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,12 @@ function codePoint(n) {
1616

1717
export function decodeEntities(text) {
1818
const named = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', mdash: '—', ndash: '–', hellip: '…', rsquo: '’', lsquo: '‘', ldquo: '“', rdquo: '”' };
19-
return String(text)
20-
.replace(/&#x([0-9a-f]+);/gi, (_, hex) => codePoint(parseInt(hex, 16)))
21-
.replace(/&#(\d+);/g, (_, dec) => codePoint(Number(dec)))
22-
.replace(/&([a-z]+);/gi, (all, key) => named[key.toLowerCase()] ?? all);
19+
// One pass, like the converter (2026-10-03): `&#38;lt;` is the text `&lt;`, decoded once.
20+
return String(text).replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (all, ref) => {
21+
if (/^#x/i.test(ref)) return codePoint(parseInt(ref.slice(2), 16));
22+
if (ref[0] === '#') return codePoint(Number(ref.slice(1)));
23+
return named[ref.toLowerCase()] ?? all;
24+
});
2325
}
2426

2527
/**
@@ -149,8 +151,8 @@ function inline(html, { decode = true } = {}) {
149151
return finish(
150152
String(html)
151153
.replace(/<a\b[^>]*href=["']([^"']*)["'][^>]*>([\s\S]*?)<\/a>/gi, (_, href, body) => {
152-
const label = finish(body.replace(/<[^>]+>/g, '')).trim();
153-
return label ? `[${label}](${href})` : '';
154+
const label = body.replace(/<[^>]+>/g, '').trim();
155+
return decodeEntities(label).trim() ? `[${label}](${href})` : '';
154156
})
155157
.replace(/<(strong|b)\b[^>]*>([\s\S]*?)<\/\1>/gi, '**$2**')
156158
.replace(/<(em|i)\b[^>]*>([\s\S]*?)<\/\1>/gi, '*$2*')

‎research-kit/test/transport.test.mjs‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,29 @@ test('an entity inside a link label, bold text or code in a list item is decoded
310310
assert.equal(httpKeyless.htmlToMarkdown('<li>&amp;lt;b&amp;gt;</li>'), '- &lt;b&gt;');
311311
});
312312

313+
// Found 2026-10-03 (review of the G3 fix): the G3 change left the page-level pass decoding
314+
// a link's label INSIDE the link pass, before the tag-stripping pass that follows it. So a
315+
// link in a paragraph or a div - the one place G3's test did not look - still lost its
316+
// comparison text (`[limit 2](/x)`) and still double-decoded `&amp;lt;`. The label is never
317+
// decoded on its own: entities are decoded once, at the end, wherever the link stands.
318+
test('comparison text inside a link label in a paragraph or a div survives, decoded once', () => {
319+
assert.equal(httpKeyless.htmlToMarkdown('<p>See <a href="/x">limit &lt; 10 and burst &gt; 2</a> now.</p>'),
320+
'See [limit < 10 and burst > 2](/x) now.');
321+
assert.equal(httpKeyless.htmlToMarkdown('<div>See <a href="/x">limit &lt; 10</a> now.</div>'), 'See [limit < 10](/x) now.');
322+
assert.equal(httpKeyless.htmlToMarkdown('<p><a href="/x">a &amp;lt; b</a></p>'), '[a &lt; b](/x)');
323+
// A label that is only `&nbsp;` is as empty as one that is only a space, in a list item as
324+
// in a paragraph: the link is dropped in both, so a page's blank icon links do not become
325+
// `[ ](/x)` in its bulleted text.
326+
const blank = httpKeyless.htmlToMarkdown('<ul><li><a href="/x">&nbsp;</a> item</li></ul><p><a href="/y">&nbsp;</a> para</p>');
327+
assert.match(blank, /^- item$/m, 'the list item keeps its text and drops the blank link');
328+
assert.match(blank, /^ ?para$/m, 'the paragraph the same');
329+
assert.doesNotMatch(blank, /\[ *\]\(/, 'no empty link survives anywhere');
330+
// A numeric reference that spells an ampersand is decoded once too: `&#38;lt;` is the text
331+
// `&lt;`, not `<`.
332+
assert.equal(httpKeyless.decodeEntities('&#38;lt; &#x26;gt; &amp;amp;'), '&lt; &gt; &amp;');
333+
assert.equal(httpKeyless.htmlToMarkdown('<p>&#38;lt;b&#38;gt;</p>'), '&lt;b&gt;');
334+
});
335+
313336
test('a page that marks its content with <main> is believed over a denser fragment', () => {
314337
// Found 2026-09-27: on a GitHub Docs page the keyless extractor kept the 262-character
315338
// summary and dropped the 17,385-character <main> around it - words-per-tag favours a

0 commit comments

Comments
 (0)