Skip to content

Make the live workflow's green mean something, and stop python hiding behind python - #17

Merged
StepenkoAnatoli merged 1 commit into
mainfrom
fix/live-workflow-and-python-probe
Sep 20, 2026
Merged

StepenkoAnatoli merged 1 commit into
mainfrom
fix/live-workflow-and-python-probe

Conversation

@StepenkoAnatoli

Copy link
Copy Markdown
Owner

All eight findings. All eight were mine — every one introduced by the commit that closed the previous audit.

A workflow written to verify things, which then discarded the exit status of every check it ran, is the sharpest version of the lesson this repo keeps relearning: a signal that looks like verification and is not is worse than no signal.

F-16 — an old python hid a usable python3 (the only one in shipped code)

checkPython tried python then python3 and returned failure at the first old interpreter. On the most common layout there is — python an alias for 2.7, python3 the real one — it reported "no usable Python" with a perfectly good 3.12 one name away.

The test almost failed to prove this. Reordering to try python3 first satisfies the common case, so reintroducing the early return left it green. The case that actually exercises the branch — the first name tried is old, a good one behind it — is asserted separately now, and goes red on its own.

F-13 / F-17 — the workflow swallowed the checks that decide its result

handoff and preflight both ran with || true. A green run proved the repository was unchanged and nothing else.

One success predicate now, stated at the top of the file and enforced by the steps:

  1. the vendor CLI is a major this adapter supports
  2. collection exited 0
  3. a ledger exists, parses, and records a successful fetch
  4. handoff exits 0
  5. preflight exits 0
  6. the repository is unchanged

Diagnostics still print on failure — via if ! cmd; then, which keeps the output and the exit status. doctor stays non-required because it legitimately reports a missing key on the keyless route; that is now said, beside the step, rather than hidden in a suffix.

F-14, F-15, F-18, F-19, F-20

F-14 Paid runs serialised. max_pages bounds one run, not the meter — two dispatches a second apart spend concurrently while a reader believes 10 is the ceiling. cancel-in-progress: false is deliberate: cancelling a paid collection wastes credits already spent.
F-15 Installation failure and incompatibility are different facts; `
F-18 The public summary carried 60 characters of each URL — enough to disclose the subject being researched, not enough to be useful. Hostnames only.
F-19 CI reads the absence of the result file as "crashed before reporting". An unwritable path produced the same absence, so a typo read as a crash. Says so on stderr now, while still never failing the run it reports on.
F-20 "No dependencies" is true of the repository, not of collection. The README draws that line.

Verified

node research-kit/bin/selftest.mjs   -> 644 passed, 0 failed, exit 0  (642 + 2)
node research-kit/bin/preflight.mjs  -> PASS, 0 blocking, 0 warnings

Both new guards were reproduced. The workflow guard pins the property — no swallow on a command that decides the result — rather than the wording.

The kit's own gate blocked my first attempt at the commit: runtime.mjs and selftest.mjs are declared code paths and the map was not staged with them.

🤖 Generated with Claude Code

… behind python

All eight findings, and all eight were mine - every one introduced by the commit
that closed the previous audit. A workflow written to verify things, that then
discarded the exit status of every check it ran, is the sharpest version of the
lesson this repository keeps relearning: a signal that looks like verification
and is not is worse than no signal.

F-16  AN OLD `python` HID A USABLE `python3`         (the only one in shipped code)
  checkPython tried `python` then `python3` and returned FAILURE at the first old
  interpreter. On the most common layout there is - `python` an alias for 2.7,
  `python3` the real one - it reported "no usable Python" with a perfectly good
  3.12 one name away.

  Two changes: `python3` is tried first, and the search CONTINUES past an old
  interpreter, remembering it only so the message can name what was found rather
  than the less useful "no python on PATH".

  The test almost failed to prove this. Reordering alone satisfies the common
  case, so reintroducing the early return left it green. The case that actually
  exercises the branch - the FIRST name tried is old, a good one behind it - is
  asserted separately now, and goes red on its own.

F-13, F-17  THE WORKFLOW SWALLOWED THE CHECKS THAT DECIDE ITS RESULT
  handoff and preflight both ran with `|| true`. A green run proved the
  repository was unchanged and nothing else - not that the corpus arrived whole,
  not that it passed the checks a synthetic corpus passes, not that anything was
  collected at all.

  One success predicate now, stated at the top of the file and enforced by the
  steps: supported CLI major, collection exited 0, a ledger exists and parses and
  records a successful fetch, handoff 0, preflight 0, repository unchanged.
  Diagnostics still print on failure - through `if ! cmd; then`, which keeps the
  output AND the exit status.

  `doctor` stays non-required, because it legitimately reports a missing key on
  the keyless route. That is now SAID beside the step rather than hidden in a
  suffix.

F-14  PAID RUNS ARE SERIALISED
  max_pages bounds one run, not the meter: two dispatches a second apart spend
  concurrently while a reader reasonably believes 10 is the ceiling.
  `cancel-in-progress: false` is deliberate - cancelling a paid collection wastes
  credits already spent and can leave a scratch ledger torn.

F-15  INSTALLATION FAILURE AND INCOMPATIBILITY ARE DIFFERENT FACTS
  `npm i -g a || npm i -g b || true` suppressed both, so "the package moved" and
  "this is a major we do not support" looked identical. A `cli_version` input
  chooses the policy explicitly: `tested` pins the version the adapter was
  written against and is the reproducible integration test; `latest` is
  surveillance and says so, where a refusal is the workflow working.

F-18  THE PUBLIC SUMMARY CARRIED URL PREFIXES
  60 characters of each URL - exactly enough to disclose the subject being
  researched and the query terms, not enough to be useful. Hostnames only.

F-19  A CONFIGURED RESULT FILE FAILED SILENTLY
  CI reads the ABSENCE of the result file as "crashed before reporting". An
  unwritable path produced the same absence, so a bad permission or a typo was
  reported as a crash that did not happen. It says so on stderr now, while still
  never failing the run it reports on.

F-20  "NO DEPENDENCIES" IS TRUE OF THE REPOSITORY, NOT OF COLLECTION
  The kit pulls nothing from npm. The metered route spawns a globally installed
  CLI versioned outside this repository, and the live workflow installs it at
  dispatch. The README draws that line now.

VERIFIED
  node research-kit/bin/selftest.mjs   -> 644 passed, 0 failed, exit 0 (642 + 2)
  node research-kit/bin/preflight.mjs  -> PASS, 0 blocking, 0 warnings

  Both new guards were reproduced. The workflow guard pins the PROPERTY - no
  swallow on a command that decides the result - rather than the wording.

  The kit's own gate blocked the first attempt at this commit: runtime.mjs and
  selftest.mjs are declared code paths and the map was not staged with them.
@StepenkoAnatoli
StepenkoAnatoli merged commit a49c793 into main Sep 20, 2026
3 checks passed
@StepenkoAnatoli
StepenkoAnatoli deleted the fix/live-workflow-and-python-probe branch September 20, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant