Skip to content

Tavily's exclusion holds, and using the kit to prove it found a bug in the kit - #68

Merged
StepenkoAnatoli merged 2 commits into
mainfrom
fix/topic-signal-limits
Sep 22, 2026
Merged

StepenkoAnatoli merged 2 commits into
mainfrom
fix/topic-signal-limits

Conversation

@StepenkoAnatoli

Copy link
Copy Markdown
Owner

What was asked

"First research Tavily's current terms with the kit." Constraint C-6 put Tavily out of scope on 2026-09-19 with an expiry condition — revisit if a no-training tier ships — so the question of replacing SerpAPI with Tavily is answered by the terms, not by preference.

The answer

The trigger is not met. C-6 stands.

§6.5 and §6.7 are the same sections C-6 cited, unchanged: Tavily and its AI providers may retain inputs and outputs "for purposes of training", and those providers "may not be required to maintain the confidentiality" of either.

Because the finding is an absence, it was counted rather than skimmed:

term occurrences
opt out 3 — all arbitration
enterprise 0
zero data retention 0
no model training 0

A prior was registered before collecting, and it held. That is worth stating: a protocol that only reports its surprises is advertising.

The bug, found by obeying the tool

preflight warned that one capture prints 176 render-failure notices, and told me to record my review "in a renderReview: front-matter line". I did exactly that, on the live corpus:

  • before: PASS, 0 blocking, 1 warning
  • after following the instruction: fail provenance/body-unmodified, and the warning still standing

verifyLedger hashes the capture whole, front-matter included. The check's own source comment, ten lines above the message, says the note must not go there — the message contradicted the code. It now names the EVIDENCE row and the [render-reviewed: ...] syntax the parser actually reads.

An existing test pinned that front-matter does not satisfy the review. Nothing pinned that the message does not send you there. Now something does, confirmed red first.

Also

  • The topic signal scored 0.92 on a corpus that was 1/8 relevant. Not broken — the topic was seven common words, so the search matched words and the signal measured the same words back. topicMatch now documents that a high score is worth nothing without distinctive terms. It decides nothing, which is why it did no damage.
  • A missing newline ran spent and topic together in the run summary.
  • ARCHITECTURE said twelve checks; CHECK_NAMES holds thirteen. corroboration landed without it.
  • SerpAPI timed out mid-run and the merge seam degraded to Firecrawl instead of collapsing. Unplanned, and the first time that path took a real provider failure in production.

Honest limits

  • Seven of eight captures are off topic and cited by nothing. They are marked as such rather than deleted, because the ledger records that the credits were spent.
  • Both unknowns rest on tavily.com/terms alone, recorded as accepted single-witness judgements. A vendor's terms have one author, and no third party can testify that a clause is absent.
  • The terms carry no effective date and no version string, so nothing in the document can confirm it is unchanged — only that what it says today is what it said three days ago. That is why the trigger has to be a re-read.

Verification

  • 874 passed, 0 failed (offline suite)
  • Corpus: PASS — 0 blocking, 0 warnings, 14 passing, and under --strict
  • Chain verifies; 8 captures, 8 ledger entries

🤖 Generated with Claude Code

StepenkoAnatoli and others added 2 commits September 22, 2026 18:14
The Tavily collection printed `0.92 best of 13 topic terms, 8 capture(s) >= 0.50`
on a corpus where one capture in eight was about Tavily. The signal is not
broken: the topic was "data retention training zero enterprise opt out", seven
common words, so the search matched the words and the signal then measured the
same words back. A high score means nothing unless the topic carried
distinctive terms, and `topicMatch` now says so where someone reading the
number will find it. It still decides nothing, which is why it did no harm.

The same summary ran two lines together - `spent 8 (...)topic 0.92 ...` - for a
missing newline.

The third is a real bug, and I found it by obeying the tool. `partial-render`
told me to record my review "in a `renderReview:` front-matter line". Doing
exactly that on a live corpus turned a 1-warning PASS into a blocking
`provenance/body-unmodified` failure and left the warning standing, because
`verifyLedger` hashes the capture whole. The check's own comment, ten lines
above the message, explains why the note must NOT go there - the message
contradicted it. It now names the EVIDENCE row and the `[render-reviewed: ...]`
syntax the parser actually reads, and says not to edit the capture.

The existing test pinned that front-matter does not SATISFY the review. Nothing
pinned that the message does not SEND you there. Now something does; confirmed
red first.

ARCHITECTURE said twelve checks and CHECK_NAMES holds thirteen - corroboration
went in without it. Fixed while here, along with the README's test count.

874 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C-6 put Tavily out of scope on 2026-09-19 and wrote an expiry condition:
revisit if a no-training tier ships. The operator asked whether Tavily should
replace SerpAPI, so the condition was checked before any key was wired.

It is not met. §6.5 and §6.7 are the same sections C-6 cited, with the same
numbering and the same clauses: Tavily and its AI providers may retain inputs
and outputs "for purposes of training", and those providers "may not be
required to maintain the confidentiality" of either. Because the finding is an
absence I counted rather than skimmed - "opt out" appears three times and all
three are arbitration; "enterprise", "zero data retention" and "no model
training" appear zero times.

A prior was registered before collecting and it held. Saying so matters: a
protocol that only reports its surprises is advertising.

The corpus is honest about being thin. Seven of eight captures are generic
industry writing about zero data retention and are cited by nothing; they are
marked off topic rather than deleted, because the ledger records the credits.
The topic signal scored 0.92 on it anyway, which is the limitation the previous
commit documents.

Two things went right unplanned: SerpAPI timed out mid-run and the merge seam
degraded to Firecrawl instead of collapsing, and the corpus passes --strict
with 0 warnings.

PASS - 0 blocking, 0 warnings, 14 passing, chain verifies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@StepenkoAnatoli
StepenkoAnatoli merged commit 657375d into main Sep 22, 2026
3 checks passed
@StepenkoAnatoli
StepenkoAnatoli deleted the fix/topic-signal-limits branch September 22, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant