Skip to content

docs: record measurement/instrumentation discipline in CLAUDE.md - #876

Merged
jkitchin merged 1 commit into
mainfrom
docs/claude-md-measurement-discipline
Jul 26, 2026
Merged

docs: record measurement/instrumentation discipline in CLAUDE.md#876
jkitchin merged 1 commit into
mainfrom
docs/claude-md-measurement-discipline

Conversation

@jkitchin

Copy link
Copy Markdown
Owner

What

Adds a Measurement & instrumentation discipline section to CLAUDE.md, plus targeted rules in Working on an issue, Workflow, and Key Constraints.

Documentation only — no code, no behavior change.

Why

Every rule is derived from a specific failure in recent sessions, not from general principle. Reviewing those sessions, the dominant failure was not a wrong measurement — it was a measurement that never happened and reported success anyway. One habit, many faces:

Incident What it reported
Probe used Constraint.lhs/.expr; the real attribute is .body traversed nothing, printed results
deepcopy(Model) raises TypeError (can't pickle PyModelRepr), caught by a bare except a whole fallback was an invisible no-op, "working"
Attribution probe gated only the root dive its "neither" arm was meaningless
Sparse patch landed in the algebraic extractor, not the repr one "parity confirmed" was dense-vs-dense
Parity test never asserted issparse() on the forced-sparse arm proved nothing, believed
pytest imported discopt from the main tree, not the worktree 19 bogus failures in a PR review
A background sweep declared dead while still running two wasted runs, the second timing out on contention the first created

Two published claims were retracted on remeasurement: a "leftover model state slows relaxation ~40%" that was really +2% (pooled sd 4.55, with an unrelated pytest holding ~87% CPU), and a gastrans040 regression that was a timing artifact. Hence the load-gate / interleaved-control / report-a-spread rule, which explicitly covers stray load you created yourself.

Contents

New section (items 6–11, continuing the Development Philosophy numbering): prove the probe fired (executed-assertion counter, non-zero exit at zero); never swallow an exception in an instrument; verify which code you loaded (__file__ and a version-unique marker, asserted absent on baseline runs); timing claims need a load gate + interleaved control + spread; a long job needs incremental output and "no output" is not "dead"; retract your own published claims in writing.

Working on an issue gains item 6, finish rather than document — an investigation with three artifacts and no code change is the signal to start building.

Workflow gains three operational rules: no issue-closing keyword near a negation (GitHub matched close #863 inside "Does NOT close #863" and closed it); look up an API rather than guessing attribute names; re-derive order-of-magnitude figures before stating them (91.5 TB was written for 91.5 GB).

Key Constraints gains the two silent-corruption traps that have each already cost a wrong conclusion: np.asarray() on a scipy sparse matrix returns a 0-d object array without raising, and INF in the Rust LP layer is the sentinel 1e20, not f64::INFINITY — so unboundedness must be tested on the bound, never on a product. That second one is not hypothetical: it is the root cause of the false optimality certificate fixed in the companion PR.

Verification

pre-commit hooks ran on commit (ruff / ruff-format / mypy / cargo fmt — all skipped or passed; no code files touched).

Every rule added here is derived from a specific failure in recent sessions, not
from general principle. The dominant one is a single habit wearing many faces:
an instrument that measured nothing and reported success anyway.

New "Measurement & instrumentation discipline" section (items 6-11, continuing
the Development Philosophy numbering):

* prove the probe fired -- emit an executed-assertion count and exit non-zero at
  zero. Three separate probes silently traversed nothing (wrong attribute name,
  a gate applied only at the root, a sparse-parity test that compared dense
  against dense) and each was believed.
* never swallow an exception in an instrument -- a bare except around
  copy.deepcopy(Model) hid a TypeError and left an entire fallback an invisible
  no-op while it was reported as working.
* verify which code you loaded -- assert __file__ AND a version-unique marker;
  a pytest run imported from the main tree instead of the worktree and produced
  19 bogus failures.
* timing claims need a load gate, an interleaved control and a spread -- two
  claims were published and retracted (a "40% slowdown" that measured +2% with
  pooled sd 4.55 under an unrelated pytest at 87% CPU, and a regression that was
  a timing artifact).
* a long job needs incremental output, and "no output" is not "dead" -- a
  background sweep was twice declared dead while still running, the second time
  with a competing job launched that then timed out on the contention.
* retract your own published claims in writing when a measurement contradicts
  them.

Also:

* "Working on an issue" gains item 6, finish rather than document: an
  investigation with three artifacts and no code change is the signal to start
  building. Filing a follow-up is the 3 exception, not the deliverable.
* Workflow gains three operational rules: no issue-closing keyword near a
  negation (GitHub matched "close #863" inside "Does NOT close #863" and closed
  it), look up an API rather than guessing attribute names, and re-derive
  order-of-magnitude figures before stating them (91.5 TB was reported for
  91.5 GB).
* Key Constraints gains the two silent-corruption traps that have each already
  cost a wrong conclusion: np.asarray() on a scipy sparse matrix returns a 0-d
  object array without raising, and INF in the Rust LP layer is the sentinel
  1e20 rather than f64::INFINITY, so unboundedness must be tested on the bound
  and never on a product.

Documentation only; no code or behavior change.
@jkitchin
jkitchin merged commit 8627069 into main Jul 26, 2026
9 checks passed
@jkitchin
jkitchin deleted the docs/claude-md-measurement-discipline branch July 26, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

watercontamination0202 overruns time_limit by >8x (107k rows, 7 binaries, parses in 1s)

1 participant