docs: record measurement/instrumentation discipline in CLAUDE.md - #876
Merged
Conversation
Every rule added here is derived from a specific failure in recent sessions, not from general principle. The dominant one is a single habit wearing many faces: an instrument that measured nothing and reported success anyway. New "Measurement & instrumentation discipline" section (items 6-11, continuing the Development Philosophy numbering): * prove the probe fired -- emit an executed-assertion count and exit non-zero at zero. Three separate probes silently traversed nothing (wrong attribute name, a gate applied only at the root, a sparse-parity test that compared dense against dense) and each was believed. * never swallow an exception in an instrument -- a bare except around copy.deepcopy(Model) hid a TypeError and left an entire fallback an invisible no-op while it was reported as working. * verify which code you loaded -- assert __file__ AND a version-unique marker; a pytest run imported from the main tree instead of the worktree and produced 19 bogus failures. * timing claims need a load gate, an interleaved control and a spread -- two claims were published and retracted (a "40% slowdown" that measured +2% with pooled sd 4.55 under an unrelated pytest at 87% CPU, and a regression that was a timing artifact). * a long job needs incremental output, and "no output" is not "dead" -- a background sweep was twice declared dead while still running, the second time with a competing job launched that then timed out on the contention. * retract your own published claims in writing when a measurement contradicts them. Also: * "Working on an issue" gains item 6, finish rather than document: an investigation with three artifacts and no code change is the signal to start building. Filing a follow-up is the 3 exception, not the deliverable. * Workflow gains three operational rules: no issue-closing keyword near a negation (GitHub matched "close #863" inside "Does NOT close #863" and closed it), look up an API rather than guessing attribute names, and re-derive order-of-magnitude figures before stating them (91.5 TB was reported for 91.5 GB). * Key Constraints gains the two silent-corruption traps that have each already cost a wrong conclusion: np.asarray() on a scipy sparse matrix returns a 0-d object array without raising, and INF in the Rust LP layer is the sentinel 1e20 rather than f64::INFINITY, so unboundedness must be tested on the bound and never on a product. Documentation only; no code or behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a Measurement & instrumentation discipline section to
CLAUDE.md, plus targeted rules in Working on an issue, Workflow, and Key Constraints.Documentation only — no code, no behavior change.
Why
Every rule is derived from a specific failure in recent sessions, not from general principle. Reviewing those sessions, the dominant failure was not a wrong measurement — it was a measurement that never happened and reported success anyway. One habit, many faces:
Constraint.lhs/.expr; the real attribute is.bodydeepcopy(Model)raisesTypeError(can't picklePyModelRepr), caught by a bareexceptissparse()on the forced-sparse armpytestimporteddiscoptfrom the main tree, not the worktreeTwo published claims were retracted on remeasurement: a "leftover model state slows relaxation ~40%" that was really +2% (pooled sd 4.55, with an unrelated pytest holding ~87% CPU), and a
gastrans040regression that was a timing artifact. Hence the load-gate / interleaved-control / report-a-spread rule, which explicitly covers stray load you created yourself.Contents
New section (items 6–11, continuing the Development Philosophy numbering): prove the probe fired (executed-assertion counter, non-zero exit at zero); never swallow an exception in an instrument; verify which code you loaded (
__file__and a version-unique marker, asserted absent on baseline runs); timing claims need a load gate + interleaved control + spread; a long job needs incremental output and "no output" is not "dead"; retract your own published claims in writing.Working on an issue gains item 6, finish rather than document — an investigation with three artifacts and no code change is the signal to start building.
Workflow gains three operational rules: no issue-closing keyword near a negation (GitHub matched
close #863inside "Does NOT close #863" and closed it); look up an API rather than guessing attribute names; re-derive order-of-magnitude figures before stating them (91.5 TB was written for 91.5 GB).Key Constraints gains the two silent-corruption traps that have each already cost a wrong conclusion:
np.asarray()on a scipy sparse matrix returns a 0-d object array without raising, andINFin the Rust LP layer is the sentinel1e20, notf64::INFINITY— so unboundedness must be tested on the bound, never on a product. That second one is not hypothetical: it is the root cause of the false optimality certificate fixed in the companion PR.Verification
pre-commithooks ran on commit (ruff / ruff-format / mypy / cargo fmt — all skipped or passed; no code files touched).