Skip to content

Commit 88a7da1

Browse files
StepenkoAnatoliclaude
StepenkoAnatoli
and
claude
committed
The privacy warning was wrong: measure what a public run exposes, and stay public
Step 2 was "decide the private-repo question before it decides itself". Deciding it meant first checking a claim I had written from reasoning and never tested. The collector's own header said: `workflow_dispatch` inputs are themselves visible on the run page to anyone with read access. Unauthenticated probing of run 35608301287, searching every response body for the topic: 200 run metadata, jobs and step names, timing, artifact LISTING topic absent 401 repository secrets refused 401 artifact DOWNLOAD refused 403 job LOGS refused A stranger learns the SHAPE of a run - that one happened, when, how long, what the steps were called, that an artifact of N bytes exists - and nothing about the subject or the contents. Wrong in the direction that feels safe is still wrong. An overstated warning is the same defect as an understated one: a reader who finds a warning false once discounts the next, and the next may be the true one. THE DECISION: THIS REPOSITORY STAYS PUBLIC Checked rather than inferred - `gh api user --jq .plan.name` returns `free`. Per E-06, converting a Free repository to private makes protection rules AND environment secrets IGNORED rather than refused. The collector's credential lives in the research-collection environment, so going private would break it, and the obvious "fix" - moving the key to a repository secret - is correctly refused by the scope-check canary. Privacy for a sensitive subject is a PLAN change, not a visibility toggle, and the measurement says it is not needed for what is actually exposed. That also promotes the naming rules from a nicety to the primary control: the names are the disclosure surface, which makes client_ref - it becomes both the run name and the artifact name - the sharpest edge in the design. THE CLAIM IS EXECUTABLE NOW, NOT REWORDED node research-kit/bin/disclosure.mjs --repository OWNER/REPO --run <id> --topic "..." Unauthenticated by construction: no token parameter, and an unknown flag matching token/auth/key/secret is refused with that reason, because a probe that quietly authenticated would answer a different question while looking like this one. Exit codes are the finding - 0 shape only, 1 content readable, 2 subject leaked, 3 could not measure. A paragraph goes stale silently; a command does not. AND THE SAME MISTAKE, GUARDED ONE LEVEL DOWN `needleChecked` exists so that "the subject was not exposed" cannot print when no subject was searched for - without --topic the report says NOT CHECKED. An unreachable endpoint is a refusal to conclude rather than a pass. The probe must not do to its reader what the header did to me, and a test asserts both. WHAT IS NOT MEASURED IS NOT CLAIMED What a signed-in user sees in the web UI is a different surface from the REST API and could not be tested from here. The logs do contain the topic and the API refuses them to a stranger with 403; whether the UI is equally strict is UNKNOWN, and unknown is what the documentation now says. Both corrected passages quote the claim they replaced. A correction that erases what it corrected teaches nobody - and this is the second time in two days that something built from documentation turned out untrue when run. 824 passed, 0 failed. Repository preflight PASS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 8572cda commit 88a7da1

8 files changed

Lines changed: 545 additions & 18 deletions

File tree

‎.github/workflows/collect.yml‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,33 @@ name: collect
3939
# and artifact listing all return 200 unauthenticated; secrets 401 and raw logs 403). A
4040
# name leaks before anybody opens anything, so neither carries the topic.
4141
#
42-
# BUT READ THIS BEFORE RESEARCHING ANYTHING SENSITIVE ON A PUBLIC REPOSITORY.
43-
# `workflow_dispatch` inputs are themselves visible on the run page to anyone with read
44-
# access. Keeping the topic out of the names reduces incidental exposure in listings; it
45-
# does NOT make a public run private. Sensitive research belongs in a private repository
46-
# on a plan that supports environments there - and per E-06, converting a GitHub Free
42+
# WHAT A PUBLIC RUN ACTUALLY EXPOSES, MEASURED RATHER THAN ASSUMED.
43+
#
44+
# An earlier version of this comment said dispatch inputs are visible on the run page to
45+
# anyone with read access. That was asserted, not checked, and checking it found the
46+
# opposite. Unauthenticated, on this repository, on 2026-09-21:
47+
#
48+
# 200 run metadata, jobs and step names, timing, artifact LISTING readable
49+
# 401 artifact DOWNLOAD refused
50+
# 403 job LOGS refused
51+
# 401 repository secrets refused
52+
#
53+
# The topic appeared in none of the readable responses. So a stranger learns the SHAPE of
54+
# a run - that one happened, when, how long it took, what the steps were called, that an
55+
# artifact of N bytes exists - and not its subject or its contents.
56+
#
57+
# That is why the naming rules above are the primary control rather than a minor one: the
58+
# names are the disclosure surface, and they are the part this workflow governs.
59+
#
60+
# Re-run the measurement instead of trusting this comment, which can go stale:
61+
# node research-kit/bin/disclosure.mjs --repository OWNER/REPO --run <id> --topic "..."
62+
#
63+
# What is NOT measured here: what a signed-in user with read access sees in the web UI,
64+
# which is a different surface from the REST API. Treat that as unknown rather than safe.
65+
#
66+
# And the reason going private is not the reflex fix: per E-06, converting a GitHub Free
4767
# repository from public to private makes its protection rules and environment secrets
48-
# IGNORED rather than refused, so that is a plan change, not a visibility toggle.
68+
# IGNORED rather than refused. That is a plan change, not a visibility toggle.
4969
#
5070
# NOTHING IS INTERPOLATED INTO A SHELL
5171
#

‎docs/ARCHITECTURE.md‎

Lines changed: 5 additions & 2 deletions
Large diffs are not rendered by default.

‎docs/adr/0035-a-privacy-claim-is-measured-not-asserted.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# ADR-0035 — A privacy claim is measured, not asserted; and this repository stays public
2+
3+
- **Date:** 2026-09-21
4+
- **Status:** accepted
5+
- **Area:** disclosure, plan, credentials
6+
- **Depends on:** ADR-0031 (Actions-first delivery), ADR-0033 (the collector refuses rather than degrades)
7+
- **Evidence:** `docs/decisions/2026-09-21-delivery-architecture/` E-06 and E-08, plus the measurement below
8+
9+
## Context
10+
11+
Two questions were open and had been answered in prose rather than by checking.
12+
13+
**"Should this repository go private for sensitive research?"** ADR-0031 recorded the
14+
constraint and nothing acted on it. **"What does a public run expose?"** The collector's own
15+
header answered it, confidently and wrongly: *"`workflow_dispatch` inputs are themselves
16+
visible on the run page to anyone with read access."*
17+
18+
That sentence was written by me, from reasoning, and never tested.
19+
20+
## The measurement
21+
22+
Unauthenticated requests against this repository, run `35608301287`, on 2026-09-21. The
23+
topic was `GitHub Actions workflow artifact retention period`, and every response body was
24+
searched for it:
25+
26+
| probe | status | topic present |
27+
|---|---|---|
28+
| run metadata | `200` readable | no |
29+
| jobs and step names | `200` readable | no |
30+
| run timing | `200` readable | no |
31+
| artifact **listing** | `200` readable | no |
32+
| repository secrets | `401` refused | — |
33+
| artifact **download** | `401` refused | — |
34+
| job **logs** | `403` refused | — |
35+
36+
**A stranger learns the shape of a run and not its subject.** That one happened, when, how
37+
long it took, what its steps were called, that an artifact of N bytes exists with a given
38+
name — and nothing about what was researched or what came back.
39+
40+
The claim in the header was wrong in the direction that feels safe, which is still wrong.
41+
An overstated warning is the same defect as an understated one: a reader who finds a
42+
warning false once discounts the next, and the next may be the true one.
43+
44+
## Decision
45+
46+
### 1. This repository stays public
47+
48+
Going private is the reflex answer and it is the wrong one here, for a reason that is
49+
recorded rather than assumed. This account is on the **free** plan — checked, not inferred:
50+
`gh api user --jq .plan.name` returns `free`. Per E-06, converting a GitHub Free repository
51+
from public to private makes its protection rules **and environment secrets** *ignored*
52+
rather than refused.
53+
54+
The collector's credential lives in the `research-collection` environment. Going private
55+
would make it unreadable, the collector would fail its credential check, and the failure
56+
would look like a broken workflow rather than a plan limitation. The next move — adding the
57+
key as a repository secret to "fix" it — is refused by the `scope-check` canary, correctly
58+
and confusingly.
59+
60+
So privacy for sensitive subjects is a **plan change**, not a visibility toggle, and it is
61+
not needed for what is measured above.
62+
63+
### 2. The naming rules are the primary control, not a minor one
64+
65+
The measurement says the names are the disclosure surface. Everything the workflow does to
66+
keep the topic out of `run-name` and the artifact name is therefore the main defence rather
67+
than a nicety, and `client_ref` being public (it becomes both names) is the sharpest edge
68+
in the design.
69+
70+
### 3. The claim becomes executable
71+
72+
`lib/disclosure.mjs` and `bin/disclosure.mjs` make the measurement repeatable by anyone:
73+
74+
```
75+
node research-kit/bin/disclosure.mjs --repository OWNER/REPO --run <id> --topic "..."
76+
```
77+
78+
Unauthenticated by construction — there is no token parameter, and an unknown flag matching
79+
token/auth/key/secret is refused with that reason, because a probe that quietly
80+
authenticated would answer a different question while looking like this one.
81+
82+
Exit codes are the finding: `0` shape only, `1` content readable, `2` subject leaked, `3`
83+
could not measure. A documentation paragraph goes stale silently; a command does not.
84+
85+
### 4. An unchecked thing is never reported as a clean thing
86+
87+
`needleChecked` exists so that "the subject was not exposed" cannot be printed when no
88+
subject was searched for. Without `--topic`, the report says **not checked**. An
89+
unreachable endpoint is likewise a refusal to conclude, not a pass. This is the same
90+
failure the ADR is about, one level down: the probe must not do to its reader what the
91+
header did to me.
92+
93+
## What is NOT measured, and is therefore not claimed
94+
95+
**What a signed-in user sees in the web UI.** That is a different surface from the REST
96+
API, and it could not be tested from here — the browser available to this session cannot
97+
reach github.com. The logs contain the topic (the step prints `TOPIC:` in its `env:`
98+
group), and the API refuses them to a stranger with `403 Must have admin rights to
99+
Repository`. Whether the UI is equally strict for a signed-in reader is **unknown**, and
100+
"unknown" is what the documentation now says rather than "safe".
101+
102+
## Alternatives considered
103+
104+
**Go private anyway, to be safe.** Breaks the collector on this plan, for an exposure the
105+
measurement does not show.
106+
107+
**Reword the warning.** Cheaper, and it would have left the next claim just as unchecked.
108+
The problem was not the wording.
109+
110+
**Refuse to run when the topic looks sensitive.** A judgement about meaning, which a regex
111+
cannot make — the same reason `client_ref` warns rather than refuses.
112+
113+
## Consequences
114+
115+
- `collect.yml`, `README.md` and `research-kit/README.md` now carry the measured table and
116+
the command, and each states what was not measured.
117+
- Both corrected passages **quote the claim they replaced**. A correction that erases what
118+
it corrected teaches nobody, and this one is the second time in two days that something
119+
built from documentation turned out untrue when run.
120+
- `bin/disclosure.mjs` is read-only and unauthenticated, so it is safe to run against a
121+
repository you do not own — which is also the only way to check somebody else's claim
122+
about their own exposure.

‎docs/adr/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,5 +50,6 @@ directory and are treated as ADR-equivalent.
5050

5151
| [0033](0033-the-collector-refuses-rather-than-degrades.md) | **The collector refuses rather than degrades, and no dispatch input reaches a shell.** `collect.yml` implements ADR-0031 against ADR-0032's contract, separate from `live-collection.yml` because "does the adapter work" and "collect this, for a person, now" deserve different approvers. The environment is the spend gate — and is **checked for existence**, because GitHub silently *creates* an unprotected environment when a workflow names one that is missing (E-06), which would remove the approval gate without failing anything. The honest limit is written in the file: GitHub exposes no scope information, so the marker proves the environment was configured and **cannot** prove a repository-level value did not serve the secret. No `http-keyless` fallback: a missing credential refuses rather than shipping a quietly worse corpus under the same artifact name. Every input arrives through `env:` and then an argv array — `${{ inputs.x }}` in a `run:` block is substitution *before* the shell parses, so a backtick in a topic executes. The test enforcing that scans every workflow and found `live-collection.yml` had been doing it since it was written; fixed here **Amended the same day**: the required reviewer is dropped and the environment kept. A reviewer approving a *dispatch* is a spend gate, not a review — nothing is collected yet to judge — and it makes autonomous dispatch impossible. The environment stays because it was never about approval: it is the only place the credential is unreadable by every other workflow. Spend is bounded by `max_pages`, the depth tier and the vendor's 402-at-zero cap. The control that replaces the reviewer is **token scope**: a fine-grained PAT with `Actions: read and write` on one repository, driven by `bin/collect-remote.mjs`, which takes **no `--token` flag** and redacts by pattern rather than by remembering its own value |
5252
| [0034](0034-the-collector-speaks-mcp-over-stdio.md) | **The collector speaks MCP over stdio** — and this is the first module in the repository built from research that PASSED THE GATE rather than from a plan; every decision cites an `E-` row in [`docs/decisions/2026-09-21-agent-interface/`](../decisions/2026-09-21-agent-interface/research/BRIEF.md). stdio because semantics are identical on both transports, so the difference is operational, and remote costs the entire OAuth 2.1 chain. Two tools, not one, because a server cannot initiate a request — so the alternative is a call held open for minutes and an agent that loses a paid run when its process dies. A `resource_link`, never embedded bytes: the spec defines embedding and **states no maximum size**, so "permitted" is not "known to work". The credential stays in the environment, which for stdio is what the spec itself prescribes. **The gate refused the first draft of that brief**, naming a load-bearing GAP; two more pages went into the same ledger to close it, and `resource_link` is what came back |
53+
| [0035](0035-a-privacy-claim-is-measured-not-asserted.md) | **A privacy claim is measured, not asserted — and this repository stays public.** The collector's own header said dispatch inputs are visible to anyone with read access. I wrote that from reasoning and never tested it; unauthenticated probing found the topic in **none** of the readable responses — run metadata, step names, timing and the artifact *listing* are `200`, while the artifact **download** is `401` and the job **logs** are `403`. A stranger learns a run's shape, not its subject. Wrong in the direction that feels safe is still wrong: a reader who finds one warning false discounts the next. Staying public because this account is on the **free** plan (checked, not inferred) where going private makes environment secrets *ignored* — a plan change, not a toggle. The claim is now executable: `bin/disclosure.mjs`, unauthenticated by construction, exit codes as the finding, and `needleChecked` so an **unchecked** thing can never print as a clean one |
5354

5455
Domain vocabulary lives in [`CONTEXT.md`](../../CONTEXT.md).

‎research-kit/README.md‎

Lines changed: 33 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ node research-kit/bin/handoff.mjs # did the corpus arrive whole?
9898
| `artifact.mjs` | the portable package: `create` (derives authorization, never takes it) and `validate` (offline, read-only) |
9999
| `collect-remote.mjs` | run the collector on GitHub and bring the result back (`--repository`, `--topic`, `--json`) |
100100
| `mcp-server.mjs` | the collector as an MCP server over stdio, for an agent that speaks the protocol |
101+
| `disclosure.mjs` | what a stranger can read of a workflow run (`--repository`, `--run`, `--topic`), unauthenticated and read-only |
101102
| `selftest.mjs` | the whole suite, offline |
102103

103104
## The portable artifact
@@ -245,16 +246,38 @@ name, both readable by anyone who can read the repository. Use an opaque job id
245246
(`job-0417`); never put the subject in it. `layoff-plan-q3` fits the permitted shape
246247
perfectly, which is why the workflow warns rather than pretending a regex could judge it.
247248

248-
**Privacy, stated honestly.** The topic is in neither the run name nor the artifact name,
249-
because an anonymous caller can read both listings on a public repository. But
250-
`workflow_dispatch` inputs *are* visible on the run page to anyone with read access — this
251-
reduces incidental exposure, it does not make a public run private. Sensitive research
252-
belongs in a private repository on a plan that supports environments there; converting a
253-
GitHub Free repository to private makes its protection rules and environment secrets
254-
**ignored** rather than refused.
249+
**Privacy, measured rather than assumed.** An earlier version of this paragraph said
250+
dispatch inputs are visible on the run page to anyone with read access. That was asserted,
251+
not checked, and checking it found the opposite. Measure it yourself:
255252

256-
**Still unproven:** the runner-side `npm install -g firecrawl@<version>` path has never
257-
executed. Closing it needs the environment, the secret, and one real dispatch.
253+
```bash
254+
node research-kit/bin/disclosure.mjs --repository OWNER/REPO --run <id> --topic "..."
255+
```
256+
257+
Unauthenticated, on this repository, 2026-09-21:
258+
259+
| probe | result |
260+
|---|---|
261+
| run metadata, step names, timing, artifact **listing** | `200` readable |
262+
| artifact **download** | `401` refused |
263+
| job **logs** | `403` refused |
264+
| repository secrets | `401` refused |
265+
266+
The topic appeared in **none** of the readable responses. A stranger learns the *shape* of
267+
a run — that one happened, when, how long, what the steps were called, that an artifact of
268+
N bytes exists — and not its subject or its contents.
269+
270+
That is why keeping the topic out of the run name and the artifact name is the *primary*
271+
control rather than a minor one: those names are the disclosure surface, and they are the
272+
part the workflow governs.
273+
274+
**Not measured, and so not claimed:** what a signed-in user sees in the web UI, which is a
275+
different surface from the REST API. Treat it as unknown rather than as safe.
276+
277+
**Going private is not the reflex fix.** On a GitHub Free plan — which this account is on —
278+
converting a repository to private makes its protection rules and environment secrets
279+
**ignored** rather than refused. The collector would stop working, safely but confusingly.
280+
That is a plan change, not a visibility toggle.
258281

259282
**An artifact is transport, not archival storage.** Workflow artifacts last at most 90
260283
days on a public repository and vanish with the run that produced them. Anything that must
@@ -372,7 +395,7 @@ node research-kit/bin/selftest.mjs # all of it
372395
node research-kit/bin/selftest.mjs gate hook # just these files
373396
```
374397

375-
815 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
398+
824 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
376399
the assertions settled (ADR-0021), and each test is raced against a watchdog
377400
(`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
378401

‎research-kit/bin/disclosure.mjs‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/usr/bin/env node
2+
// bin/disclosure.mjs - measure what a public run exposes, instead of asserting it.
3+
//
4+
// Sends unauthenticated requests to a repository you name and reports what a stranger can
5+
// read. There is no token flag and no credential is read from the environment: a probe
6+
// that quietly authenticated would answer a different question and look like this one.
7+
//
8+
// Exit codes are the finding, so this can gate something later:
9+
// 0 only the SHAPE is public - names, sizes, timings
10+
// 1 CONTENT is public - artifacts or logs can be read by a stranger
11+
// 2 the SUBJECT is public - the topic you passed was found in a public response
12+
// 3 could not measure
13+
14+
import { parseFlags } from '../lib/core.mjs';
15+
import { requireRuntime } from '../lib/runtime.mjs';
16+
import { probeRun, render } from '../lib/disclosure.mjs';
17+
18+
const HELP = `disclosure - what can a stranger see of a workflow run?
19+
20+
node research-kit/bin/disclosure.mjs --repository OWNER/REPO --run <id> [--topic "<text>"]
21+
22+
--repository OWNER/REPO required
23+
--run <id> the workflow run to probe (required)
24+
--topic "<text>" the subject whose appearance would mean it leaked
25+
--json machine-readable report
26+
27+
Every request is UNAUTHENTICATED, on purpose: the question is what somebody with no
28+
relationship to the repository can read. There is no token flag.
29+
30+
Exit: 0 shape only, 1 content readable, 2 subject leaked, 3 could not measure.
31+
`;
32+
33+
const { flags } = parseFlags(process.argv.slice(2));
34+
if (flags.help) { process.stdout.write(HELP); process.exit(0); }
35+
requireRuntime({ node: true });
36+
37+
const known = new Set(['repository', 'run', 'topic', 'json', 'help']);
38+
const unknown = Object.keys(flags).filter((f) => !known.has(f));
39+
if (unknown.length) {
40+
for (const f of unknown) {
41+
process.stderr.write(`unknown option --${f}\n`);
42+
if (/token|auth|key|secret/i.test(f)) {
43+
process.stderr.write('This probe is unauthenticated by design. Authenticating it would answer a different question.\n');
44+
}
45+
}
46+
process.stdout.write(HELP);
47+
process.exit(3);
48+
}
49+
50+
const missing = ['repository', 'run'].filter((n) => flags[n] === undefined || flags[n] === true);
51+
if (missing.length) {
52+
process.stderr.write(`disclosure needs ${missing.map((m) => `--${m}`).join(' and ')}\n\n`);
53+
process.stdout.write(HELP);
54+
process.exit(3);
55+
}
56+
57+
let report;
58+
try {
59+
report = await probeRun({
60+
repository: String(flags.repository),
61+
runId: Number(flags.run),
62+
needle: flags.topic === undefined || flags.topic === true ? null : String(flags.topic),
63+
});
64+
} catch (error) {
65+
process.stderr.write(`could not measure: ${error.message}\n`);
66+
process.exit(3);
67+
}
68+
69+
if (flags.json) process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
70+
else process.stdout.write(`${render(report)}\n`);
71+
72+
process.exit(report.exposedSubject ? 2 : report.exposedContent ? 1 : 0);

0 commit comments

Comments
 (0)