Repository navigation
Tavily's exclusion holds, and using the kit to prove it found a bug in the kit - #68
Merged
Merged
Conversation
The Tavily collection printed `0.92 best of 13 topic terms, 8 capture(s) >= 0.50` on a corpus where one capture in eight was about Tavily. The signal is not broken: the topic was "data retention training zero enterprise opt out", seven common words, so the search matched the words and the signal then measured the same words back. A high score means nothing unless the topic carried distinctive terms, and `topicMatch` now says so where someone reading the number will find it. It still decides nothing, which is why it did no harm. The same summary ran two lines together - `spent 8 (...)topic 0.92 ...` - for a missing newline. The third is a real bug, and I found it by obeying the tool. `partial-render` told me to record my review "in a `renderReview:` front-matter line". Doing exactly that on a live corpus turned a 1-warning PASS into a blocking `provenance/body-unmodified` failure and left the warning standing, because `verifyLedger` hashes the capture whole. The check's own comment, ten lines above the message, explains why the note must NOT go there - the message contradicted it. It now names the EVIDENCE row and the `[render-reviewed: ...]` syntax the parser actually reads, and says not to edit the capture. The existing test pinned that front-matter does not SATISFY the review. Nothing pinned that the message does not SEND you there. Now something does; confirmed red first. ARCHITECTURE said twelve checks and CHECK_NAMES holds thirteen - corroboration went in without it. Fixed while here, along with the README's test count. 874 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
C-6 put Tavily out of scope on 2026-09-19 and wrote an expiry condition: revisit if a no-training tier ships. The operator asked whether Tavily should replace SerpAPI, so the condition was checked before any key was wired. It is not met. §6.5 and §6.7 are the same sections C-6 cited, with the same numbering and the same clauses: Tavily and its AI providers may retain inputs and outputs "for purposes of training", and those providers "may not be required to maintain the confidentiality" of either. Because the finding is an absence I counted rather than skimmed - "opt out" appears three times and all three are arbitration; "enterprise", "zero data retention" and "no model training" appear zero times. A prior was registered before collecting and it held. Saying so matters: a protocol that only reports its surprises is advertising. The corpus is honest about being thin. Seven of eight captures are generic industry writing about zero data retention and are cited by nothing; they are marked off topic rather than deleted, because the ledger records the credits. The topic signal scored 0.92 on it anyway, which is the limitation the previous commit documents. Two things went right unplanned: SerpAPI timed out mid-run and the merge seam degraded to Firecrawl instead of collapsing, and the corpus passes --strict with 0 warnings. PASS - 0 blocking, 0 warnings, 14 passing, chain verifies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was asked
"First research Tavily's current terms with the kit." Constraint C-6 put Tavily out of scope on 2026-09-19 with an expiry condition — revisit if a no-training tier ships — so the question of replacing SerpAPI with Tavily is answered by the terms, not by preference.
The answer
The trigger is not met. C-6 stands.
§6.5 and §6.7 are the same sections C-6 cited, unchanged: Tavily and its AI providers may retain inputs and outputs "for purposes of training", and those providers "may not be required to maintain the confidentiality" of either.
Because the finding is an absence, it was counted rather than skimmed:
opt outenterprisezero data retentionno model trainingA prior was registered before collecting, and it held. That is worth stating: a protocol that only reports its surprises is advertising.
The bug, found by obeying the tool
preflightwarned that one capture prints 176 render-failure notices, and told me to record my review "in arenderReview:front-matter line". I did exactly that, on the live corpus:fail provenance/body-unmodified, and the warning still standingverifyLedgerhashes the capture whole, front-matter included. The check's own source comment, ten lines above the message, says the note must not go there — the message contradicted the code. It now names the EVIDENCE row and the[render-reviewed: ...]syntax the parser actually reads.An existing test pinned that front-matter does not satisfy the review. Nothing pinned that the message does not send you there. Now something does, confirmed red first.
Also
topicMatchnow documents that a high score is worth nothing without distinctive terms. It decides nothing, which is why it did no damage.spentandtopictogether in the run summary.CHECK_NAMESholds thirteen.corroborationlanded without it.Honest limits
Verification
874 passed, 0 failed(offline suite)PASS — 0 blocking, 0 warnings, 14 passing, and under--strict🤖 Generated with Claude Code