Caching stage1 - #1686
Open
sjdv1982 wants to merge 14 commits into
Open
Conversation
The caching feature touches three concerns that are best judged separately, by different people, against different criteria: whether the reuse on offer is the reuse a HADDOCK user would ask for, whether the implementation is a reasonable thing to carry in HADDOCK3, and whether the reproducibility the whole idea rests on actually holds. A user asked to review a diff would be reviewing the wrong artifact; a core developer asked whether reusing a refinement result across a changed cluster rank is scientifically acceptable would be answering the wrong question. `caching-publication-plan.md` writes down the review order that follows: four cumulative branches, each the previous one plus further work, arranged so that the promised behaviour is fixed before the implementation that has to satisfy it exists. This branch is stage 1, and the document states the bar it is held to: it must stand up **even if caching is never merged**. Bitwise-reproducible CNS results are worth having in their own right, and the canonicalization library at the end of this branch is a self-contained component that no production code calls. Stage 1 is not "caching, part one". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`libparallel.Scheduler` collected worker results in completion order. Each
`Worker` puts its whole chunk on the queue when it finishes, and `Scheduler.run`
flattened `all_results` in arrival order, so a slow first chunk pushed its
results behind every chunk that overtook it:
submission order : [0, 1, 2, 3, 4, 5, 6, 7]
Scheduler.results: [2, 3, 4, 5, 6, 7, 0, 1]
This is a pre-existing defect. It is present on `main` and was not introduced
by the reproducibility work; what changed is that it stopped being harmless.
Callers that pair `Scheduler.results` positionally with their submitted job
metadata were previously self-correcting, because everything else in the tuple
-- combination, restraint file, seed, and the index used to name the expected
PDB -- comes from submission order and stayed mutually consistent, and the
generated CNS script is self-describing. Once a caller needs the pairing to
actually hold, the same reordering silently mispairs jobs with their metadata:
declared output script actually writes
rigidbody_1.pdb <-> rigidbody_3.pdb
rigidbody_3.pdb <-> rigidbody_5.pdb
rigidbody_5.pdb <-> rigidbody_7.pdb
rigidbody_7.pdb <-> rigidbody_1.pdb
Measured on a real run with `sampling = 8`, `ncores = 4` and a delay injected
into the first chunk's work; all eight jobs were mispaired. Five unskewed runs
at the same settings showed no divergence, so this does not surface by luck --
it needs load imbalance, which ensembles and real systems supply.
The fix belongs in the scheduler rather than at any call site, since anything
that consumes `Scheduler.results` positionally has the same bug. `Worker` now
carries the submission index of each of its tasks and returns `(index, result)`
pairs; `Scheduler` sorts on that index before exposing `results`. Task
execution, chunking and exception handling are unchanged, and the public shape
of `Scheduler.results` is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both modules read `sampling_factor` into a local, warn when it is zero, and
clamp it to 1:
sampling_factor = self.params["sampling_factor"]
if sampling_factor == 0:
self.log("[Warning] sampling_factor cannot be 0, setting it to 1")
sampling_factor = 1
They then ignored the clamped local and looped over the raw parameter instead,
so `sampling_factor = 0` logged the warning, announced the correction, and
produced no CNS jobs at all -- the module reported that 100% of its output was
missing rather than refining each model once. The same local is already used
correctly a few lines earlier to compute `nmodels` for the sampling guardrail,
so the two disagreed about how many models the step would produce.
Use the clamped local at the loop as well. `flexref` already did.
Behaviour is unchanged for every configuration with `sampling_factor >= 1`,
which is every shipped example.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_add_cg_backmapping_arguments` builds the `$input_aa_psf_filename_N`,
`$input_aa_pdb_filename_N` and `$input_cgtbl_filename_N` families that cgtoaa
uses to map a coarse-grained complex back to all-atom. It built the two lists
over different populations and then zipped them:
aa_psf_list <- every component, shape molecules included
cgtoaa_tbl_list <- non-shape components only
`zip` truncates to the shorter list, so with a shape molecule present the two
indexings only coincide when the shape happens to come last. Verified by
running the shipped shape example with the molecule order rotated:
input_aa_psf_filename_1 = shape_haddock.psf input_cgtbl_filename_1 = 2r15_A..._cg_to_aa.tbl
input_aa_psf_filename_2 = 2r15_A_haddock.psf input_cgtbl_filename_2 = 2r15_B..._cg_to_aa.tbl
2r15_B_haddock.psf dropped by zip
Molecule B was never back-mapped, and molecule A was restrained by B's
back-mapping restraints. Silently: nothing downstream can tell that the
restraint applied to a structure it was not generated for.
This is pre-existing and independent of the reproducibility work; it was found
while reading a generated cgtoaa input. It is fixed here rather than left
in place because canonicalization pins these paths into a job's identity, and
pinning a wrong pairing would make the defect reproducible rather than
intermittent.
Both branches now iterate topology, restraint and shape flags together and
append to both lists only for non-shape components, so a shape molecule is
skipped consistently wherever it appears in `molecules`. The single-entry
branch is corrected the same way; previously it appended unconditionally and
would dereference `None` for a lone shape input. A test pins the leading-shape
ordering, which is the case that was wrong.
Also corrects the "Coarse-Crain" spelling in one of the two error messages.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`topocg` placed the SCD dummy beads of a coarse-grained model using a random
vector drawn from the process-global, unseeded `random` module
(`libaa2cg.add_dummy`). Two runs of a byte-identical configuration, same inputs,
same filenames, `ncores = 1`, produced different coordinates:
- ATOM 5 SCD1 SER A1462 -29.859 -14.171 8.823
+ ATOM 5 SCD1 SER A1462 -29.813 -14.290 8.862
Five runs gave five distinct outputs. The generated `.inp` was identical between
runs and re-running one `.inp` three times gave identical output, so CNS is not
involved: the nondeterminism is on the Python side, upstream of CNS, and no
amount of output normalization can reach it. It then propagates into every
downstream CNS job in a coarse-grained workflow, since rigidbody, cgtoaa and
emref all read the coarse-grained PDB -- an all-atom pair of runs agreed at
every step while the coarse-grained pair diverged from topocg onward.
Determinism of the computation is the precondition for everything else in this
branch. A canonical job identity is only meaningful if the same job, run again,
produces the same result; where it does not, a cache hit and a re-execution
disagree for reasons that have nothing to do with the identity being wrong.
`add_dummy` and `map_cg` now take an explicit generator, and `martinize` accepts
a `seed` and threads a `random.Random(seed)` through. `topocg` passes the
configured `iniseed`, so coarse-graining is reproducible from the run
configuration alone rather than from ambient interpreter state, and two runs of
the same config now produce 18 of 18 identical artifacts across the workflow.
`caprieval` and `caprifilter` also coarse-grain, but the reference structure
rather than a model, and they have no `iniseed` of their own. They pass a fixed
seed, which makes reported CAPRI metrics for coarse-grained runs reproducible;
the alternative -- leaving them on `random.Random(None)` -- keeps exactly the
defect fixed here one module over, in the numbers users quote.
`add_dummy(..., rng=None)` still falls back to an unseeded generator, so
callers outside HADDOCK3 keep the previous behaviour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A CNS job's random seed was a function of everything except the job. `libcns`
held a module-level `RandomNumberGenerator` and drew from it in two places, so a
seed depended on how many CNS inputs had been prepared earlier in the same
process. Where a model carried an inherited seed instead, that seed came from
the index its producer happened to occupy in an earlier schedule. Neither is a
property of the computation, and both make the same job unrecognisable as the
same job.
One rule replaces both, for the five modules whose recipes read `$seed` --
`rigidbody`, `flexref`, `emref`, `mdref`, `mdscoring`:
a job's seed is a function of `iniseed`, of the content of what the job
reads, and of which repeat of that job it is -- and of nothing else.
Not of the job's index, not of the schedule's length, not of how many molecules
or conformers the run happens to contain.
## What was wrong, and what it cost
**The ambient generator.** `prepare_single_input` emitted `eval ($seed=...)` for
topology jobs, so a topology job's seed followed its molecule's position in the
process-wide draw order:
molecules = [molA, molB] molA seed=62729 molB seed=68893
molecules = [molC, molA, molB] molA seed=68893 molB seed=63673
The seed does not reach a topology artifact at all -- the same `molA.inp` re-run
at seeds 1, 68893 and 99999 produced byte-identical PDB and PSF -- because the
topology recipes read `$iniseed`, not `$seed`
(`topoaa/cns/generate-topology.cns`, `topocg/cns/generate-topology.cns`,
`topoaa/cns/build-missing.cns`). `$seed` appears nowhere in either CNS tree; the
line was dead text. Dead text in the input still changes the input, so the same
receptor topologized in two workflows with different molecule counts or orders
could never be recognised as the same computation. The line is removed.
`prepare_cns_input` drew a second fallback whenever a model carried no seed.
`PDBFile.seed` defaults to `None` and topoaa never assigns it, so every model
going straight from topoaa into a refinement or scoring module took that
fallback. There the seed is not dead. For minimisation-only modules it makes no
difference, but for MD-based ones it changes the science:
mdscoring, same .inp, seed varied:
seed=63673 HADDOCK score -57.8214
seed=88327 HADDOCK score -47.6475 normalized PDB differs
seed=99999 HADDOCK score -47.1345 normalized PDB differs
Both shipped `refine-*` examples take that path, so an MD refinement's seed
there was a function of process-wide draw order. `RandomNumberGenerator` is left
with no caller, and `libs/libmath.py` is removed along with its test and its
documentation entries.
**Refinement replicas were duplicates.** A refinement model inherited its input
model's seed -- `prepare_expected_pdb` copied `model_obj.seed` -- so every
replica of one input was the same computation with the same seed. Measured on a
`sampling_factor = 2` run: `flexref_1.pdb` and `flexref_3.pdb` are both
`2a931732`, and `flexref_2.pdb` and `flexref_4.pdb` are both `0b8db467`, with
seeds 918 and 919 inherited from `rigidbody_1` and `rigidbody_2`. Downstream of
`rigidbody`, `sampling_factor` bought duplicate models rather than more
sampling. This is pre-existing and not introduced by this branch: the code
before it passed `seed=model.seed` in the same way. The inheritance is removed;
a replica's seed now follows its replica index, which is what
`sampling_factor` was always supposed to mean.
**Rigid-body reuse did not survive an ensemble edit.** `rigidbody` seeded job k
with `iniseed + k` and bound it to `combinations[k % n]`, so both halves were
stable when `sampling` grew and neither was when `n` changed. Measured on a
40-job run over a ten-member ensemble:
perturbation jobs whose content survives at their own index
add one member 2 of 40
remove one member 9 of 40
Those are genuine recomputations rather than a measurement artefact: a job can
only match a job with the same seed, the seed pinned it to the same index, and
at that index the combination differed. Adding one conformer to a ten-member
ensemble therefore discarded 38 of 40 docking jobs. A second effect compounds
it: the member order presented downstream is *string*-sorted, so the topoaa
keys of an eleven-member run read `'0', '1', '10', ...` and an added eleventh
member does not append -- it inserts between members 1 and 2 and shifts every
combination after it. Deriving a combination's identity from its members'
content rather than from its index is what lets a job survive that.
## Why it is the seed that had to change
The seed is the sole remaining channel through which a schedule's numbering
reaches a job's identity. That is why no emission order could be right in both
directions: input-major numbering is stable when the input set grows, round-
major when the replica count grows, and neither is stable under both, because a
flat counter was doing work that belongs to the job's content. Close the channel
and the question stops arising.
## What is hashed
For refinement and scoring, `(iniseed, the input model's content, the replica
index)`. For rigid-body sampling, `(iniseed, the content of the combination's
members in order, the repeat index of that combination)` -- in order, because
two molecules swapped between pins is a different docking job. Content checksums
are memoized per process, since a run otherwise re-reads the same topology once
per job that docks it, and a model stored compressed hashes the same as the same
model stored plain.
Two details are settled here rather than left to be discovered. The derived
value stays below 2**31: CNS holds numbers as double-precision floats, and
`iniseed` admits values up to 10**16, which is past the point where a CNS value
is still an exact integer. And seed collisions between *different* jobs are
harmless -- seeds need not be unique, only stable, and distinct across repeats
of one job, which the repeat index guarantees.
`iniseed` keeps its meaning exactly: changing it still changes every seed in the
run.
## Consequences
Results change for every run of the five seeded modules. That is permitted here.
Reproducing what HADDOCK3 produced yesterday is not a goal of this branch --
making results reproducible from here on is -- and it is a reason to change the
scheme once and deliberately rather than in instalments, since each instalment
spends the same disruption again. One integration test's fnat band moves onto
the band its two sibling tests already use.
`cgtoaa` and `emscoring` read neither `$seed` nor `$iniseed`. `cgtoaa` stops
assigning a seed to the models it is handed, and neither module passes one,
rather than carrying a dead inherited value into their input. `emscoring` has no
`iniseed` parameter, which is consistent, since it never reads one.
A side benefit worth recording: a content-derived seed makes a job
self-contained. Its seed follows from its declared inputs, so a job dumped to a
working directory carries everything needed to reproduce it, without knowing the
schedule it came from.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`rigidbody` derived a per-combination repeat count from the requested sampling:
sampling_factor = int(sampling / len(models_to_dock))
and then ran each model combination that many times, combination-major. Two
consequences follow, and the second is the reason for this change.
The visible one is truncation. Integer division discards the remainder, so the
number of jobs actually run was `sampling_factor * n_combinations`, not
`sampling`:
sampling=1000 combinations=3: old jobs=999 new jobs=1000
sampling= 10 combinations=4: old jobs= 8 new jobs= 10
sampling= 5 combinations=4: old jobs= 4 new jobs= 5
The structural one is that the schedule was not prefix-stable. Because jobs were
laid out combination-major with a count derived from the total, changing
`sampling` renumbered every job: job 5 of a 10-model run and job 5 of an 11-model
run were different computations on different inputs with different seeds. Nothing
about job k could be decided from k alone.
Sampling is now a flat round-robin over the combinations, so job k depends only
on k:
job k <- models_to_dock[k % n_combinations]
and exactly `sampling` jobs are scheduled. Raising `sampling` appends jobs and
leaves every earlier one -- input, restraint file and seed -- untouched; lowering
it truncates. `ambig_fnames[k % n_diff]` is prefix-stable for the same reason.
A pure-function test pins the property, since it is not visible in any single
run's output and a future refactor could silently restore the renumbering.
The seed does not appear in that mapping, and it is the reason the mapping can
be this simple. A seed is derived from the combination being docked and from
which repeat of that combination the job is, never from k, so flattening the
nested loop into a counter does not smuggle the schedule's numbering back into
job identity. The repeat index is recovered from the schedule itself --
`_repeats_of_sampled_combinations` counts how many times each combination has
already been scheduled -- rather than computed as `k // n_combinations`, so the
two stay in step by construction if the round-robin is ever replaced by another
prefix-stable order.
This changes user-visible results. The job count changes as tabulated above, and
the assignment of model combinations changes from combination-major to
round-robin, so a combination is now docked in a different order than before.
That is a change of scheduling, not of scientific method: a macroscopic change
in result quality would indicate a separate problem. Documented in the changelog
and in the `sampling` parameter description.
`check_combination_chains` is hoisted out of the per-job path. It validates a
model combination, so it needs to run once per distinct combination rather than
once per sampled job -- under the previous code it ran `n_combinations` times,
and a naive port of the round-robin schedule would have made it run `sampling`
times, adding O(sampling) redundant PDB parses on the default local path. The
resolved chain IDs are computed once per source combination and passed to
`prepare_cns_input` through a new `chainid_list` argument.
`sampling_factor` is gone from both `prepare_cns_input_*` signatures; it was
being passed as a literal at every call site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`flexref`, `emref` and `mdref` emitted every replica of input 1 before input 2, so raising `sampling_factor` renumbered every job belonging to an input after the first: with two inputs, `flexref_2.pdb` is input 2's only replica at `sampling_factor = 1` and input 1's second replica at `sampling_factor = 2`. Replicas are now emitted in rounds -- every input gets its first replica before any input gets its second -- as `rigidbody` already does across its combinations. This is a numbering property, and it is worth arguing as one rather than as a reuse property. Seeds are derived from content, so the emission order changes neither what any job computes nor whether it can be recognised again; a job is found wherever it sits. What it changes is whether `flexref_3.pdb` means the same thing in two runs that differ only in `sampling_factor` -- for a person comparing two runs, and for every downstream step that carries a model's number. All three modules, in one commit. They are one job shape -- one input model in, one refined model out -- and test sets that cover job shapes cover this one through a single representative. Changing one and leaving the other two would not leave a coverage gap, which would at least be visible in a case list; it would leave the representative representing nothing, with every test still green. The schedule is therefore one shared pure function rather than three copies of a loop, and `tests/test_refinement_shape_family.py` writes the constraint down and pins the sharing. A pure-function test covers the ordering itself, in the shape of the rigid-body one: prefix stability in `sampling_factor` is not visible in any single run's output, and a future refactor could silently restore the renumbering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every module parameter was written into the generated `.inp`, including the
purely Python-side ones. `tolerance` is the clearest case: it is documented as
"percentage of allowed failures for a module to successfully complete", is
consumed only by `BaseHaddockModule.export_io_models`, and is referenced by no
CNS script -- yet it appeared as `eval ($tolerance=5)` in every topology,
sampling and refinement input. So did the twelve execution globals (`ncores`,
`max_cpus`, `mode`, `batch_type`, `queue`, `queue_limit`, `concat`,
`self_contained`, `clean`, `offline`, `debug`, `cns_exec`). Changing how a run
is scheduled changed the text of the computation it performed.
`BaseCNSModule.cns_params()` selects, from the module's parameters, those its
own CNS recipe tree can actually read. Two admission rules, both derived from
the recipes rather than maintained by hand:
1. a literal `$name` occurring anywhere under the module's `cns/` tree;
2. a name a recipe can *construct* by splicing a loop variable into a symbol.
The second rule is not optional. CNS recipes address whole parameter families
by symbol splice -- `$int_$nmol1_$nmol2`, `$nrair_$nchain1`,
`$seg_sta_$nchain1_$nseg`, `$c2sym_sta$ncs_$nsym` -- so a literal-name scan
alone sees the tokens `int_`, `nmol1` and `nmol2` but never `int_1_2`. Admitting
only literal names would silently drop 210 interaction-matrix parameters plus
the random-AIR, semi-flexible-segment, symmetry and NCS families from every
sampling, refinement and back-mapping module, and CNS would fail every job:
%XRMULT-ERR: Illegal data types:
eval($scalfac = $kinter * $scale.int_$nchain1_$nchain2)
So `cns_params()` also collects the splice prefixes each recipe tree contains
and admits any parameter matching `prefix<digits>(_<digits>)*`. `mol_`, `fle_`
and `ncs_` are admitted by prefix: the first two are the expandable molecule and
flexible-segment families, and `ncs_*` is consumed by CNS' built-in NCS data
structure rather than named in the recipe text.
The direction of this rule matters more than its precision. A deny-list of known
orchestration settings fails open: a parameter that is not on the list leaks into
the input by default, which costs a spurious recomputation. A recipe-derived
include rule fails the other way -- a parameter CNS needs but the rule cannot see
is silently dropped, and the job computes something else or nothing at all. The
splice expansion exists to keep that failure mode closed, and a per-module test
asserts that constructible families survive.
Three related corrections to the generated input:
- `prepare_cns_input` assigned `$ambig_fname` twice when the configuration named
a restraint archive: once from the raw configuration value (`ambig.tbl.tgz`)
and once with the per-job extracted table. CNS honours the last assignment, so
the first was dead text naming a file the job never reads. It is dropped
before `load_workflow_params`.
- The per-model `$ligand_top_fname` assignment is dropped. Only the two topology
recipes read that variable, and neither of them is generated through
`prepare_cns_input`, so for every module that is, the assignment was dead text
naming a file the job never opens -- and the topology it describes is in the
PSF by then. Its value is a step-folder path, and being unread it is not a
declared dependency, so nothing would rewrite it and it would carry that
locator into the job's identity. The ligand *parameter* file, which every
recipe reached through this function does read, is still emitted per model.
A test pins the premise, so a recipe that starts reading the topology variable
fails loudly rather than silently losing a real input.
- Direct callers of `prepare_cns_input` bypass `BaseCNSModule`'s molecule
parameter expansion and so supplied only `mol_*_1`. Each `mol_*` family is now
completed from its molecule-1 default up to the component count the input
structure actually has.
`$ini_count` and `$structures` are removed from `rigidbody.cns`. `$structures`
is referenced nowhere in the codebase and `$ini_count` survives only as a
self-assignment elsewhere; both were derived from `sampling`, which rigidbody's
recipe does not otherwise read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CNS stamps each artifact it writes with where and when it was written. A PDB
carries the run directory and the output filename; a PSF carries its own
filename and a wall-clock date:
REMARK FILENAME= /home/user/run1/1_rigidbody/rigidbody_1.pdb
REMARK DATE:26-Aug-2026 14:03:11 created by user: user
REMARK HADDOCK stats for rigidbody_1.pdb
REMARK initial structure 1 - ../0_topoaa/molA_haddock.pdb
; FILENAME="molA_haddock.psf"
None of it is scientific content, and no Python code in HADDOCK3 reads any of
it. All of it changes the bytes of the file. Two runs of the same workflow
differing only in the directory they ran in, or in what the input molecules were
called, produced artifacts that no byte comparison could recognise as the same
result -- and the `initial structure` line carries the upstream step folder and
input filename forward, so the difference propagates into everything downstream
that reads the model.
`libs/libcnsoutput.py` normalizes both artifact kinds. Volatile PDB REMARK lines
are dropped; the PSF date stamp is dropped and its `; FILENAME=` title is
rewritten to a fixed literal rather than removed, so the file keeps a title and
the erasure stays visible to a reader. Normalization is byte-level throughout:
records are split on LF only and never decoded, because CNS output is
scientific data that may contain bytes that are not valid UTF-8, and Python's
`str.splitlines()` would additionally treat `\x0b`, `\x0c`, `\x85` and U+2028 as
line breaks. `.gz` artifacts are decompressed, normalized and recompressed with
`mtime=0`. Rewrites go through a temporary file and `os.replace`, so a hardlinked
source is never modified in place.
For a job to be normalized, its outputs have to be known, so `CNSJob` now
accepts `output_files` and `output_pdb_files` and every one of the nine CNS
module call sites declares what it expects. Two consequences worth naming:
- Outputs are resolved against `work_dir`, captured at construction, rather than
against the process's current directory at run time. Jobs are always built in
their step directory; that was already load-bearing and is now explicit.
- `_assert_declared_output_bindings` runs in `__init__` and rejects a job whose
declared outputs disagree with the `$output_pdb_filename` /
`$output_psf_filename` its own script assigns. This is cheap insurance against
a whole class of bug rather than a hypothetical: a job that normalizes a file
it did not write fails silently, because normalizing a missing path is a
no-op and the real artifact keeps its volatile headers. The assertion turns
that into a loud failure at job construction, before CNS runs and before any
scheduler is involved. The invariant it relies on holds unconditionally --
across 23 jobs covering all nine shapes, the declared output PDB equals
`$output_pdb_filename`, a declared PSF equals `$output_psf_filename`, and
where no PSF is declared the script has no such assignment.
Verified: a real `rigidbody_1.inp` run twice with only the output filename
literal changed produces artifacts whose raw bytes differ and whose normalized
bytes are identical; renaming both input molecules leaves all four topoaa PSFs
byte-identical; and across a full seven-module all-atom workflow the normalized
artifacts are byte-identical to the merge base's artifacts with the same
normalization applied -- that is, the removals listed above are exactly the
delta, and nothing else changes.
This removes provenance information from published scientific output. That is
the point, but it is user-visible, so it is recorded in the changelog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CNS wrote directly to the final HADDOCK filename. If CNS or its worker died
mid-write, a truncated PDB was left at the path a completed model is supposed to
occupy, and `Persistent.is_present()` checks only that the path exists -- so
downstream module logic would treat the fragment as a generated result. `run()`
compounded this by never consulting the subprocess exit status: failure was
inferred from stderr being non-empty and from scanning stdout for known CNS
error markers, so a hard failure that wrote nothing to stderr looked like
success.
Each declared output is now written to a hidden staging name in the same step
directory, keeping the logical suffix, because CNS recipes derive auxiliary
filenames from it:
.rigidbody_1.partial.pdb
.molecule_haddock.partial.psf
After CNS terminates, publication requires all of: an acceptable exit status, no
known CNS error marker in stdout, every declared output present, and every
declared output non-empty. Only then are the artifacts normalized and moved to
their public names with same-filesystem `os.replace()`. An interrupted job leaves
only the hidden staging file; the public path stays absent, which is a state
downstream logic already handles. A retry clears stale staging files as a set.
Normalization moves ahead of publication with them, rewriting the staging file
rather than one already sitting at its final name, so a public path never exists
in unnormalized form.
PDB and PSF cannot be made one atomic filesystem transaction while the step
directory stays flat, but they are validated as a pair before either is
published and cleared as a pair before a retry, and no downstream module starts
until the producing module has finished.
The staging name cannot reach a result's identity, which is what makes this safe
to do at all. In the artifact, the `REMARK FILENAME=` and `HADDOCK stats for`
lines that would otherwise carry the temporary name are already stripped by
normalization. In the identity, the canonical representation added two commits
later erases `$output_pdb_filename` to a fixed literal, so no output filename --
staged or public -- reaches a checksum. What is rewritten here is only the
script CNS executes; the canonical form is not built on this path, and CNS never
sees it.
Beyond the staging itself, `run()` is corrected in five places:
- The exit status is now a failure condition.
- A known CNS error marker in stdout now raises. Previously it wrote the `.err`
file and returned normally unless stderr was also non-empty, so a job that
announced its own failure the way CNS actually announces one was recorded as a
success.
- GNU Fortran writes `IEEE_DENORMAL` to stderr for otherwise complete CNS
calculations. Treating any stderr output as failure would fail those jobs, and
ignoring stderr entirely would hide real ones, so exactly that one note is
filtered and anything else on stderr still fails the job.
- `run()` no longer has separate string-input and path-input branches. Both
materialize the executed script and pipe it on stdin, so a `debug = true` run
retains an `.inp` that is the script CNS actually received rather than one that
differs from it in the output filenames. The `.out` file continues to be
written for a path-backed input only, which is where it was written before.
- CNS is started with `cwd` set to the job's `work_dir`, and the `.err`, `.out`
and `.seed` files are resolved through `_output_path()`. All of these paths
are relative and a multiprocessing worker does not necessarily sit in the step
directory, so they were only ever correct by accident of the caller's current
directory. The `.seed` path is also repaired: the old
`Path(Path(self.output_file).stem).with_suffix(".seed")` made a path out of the
output file's stem and dropped its directory, so `compress_seed` looked for a
file that was not there.
`prepare_execution_input()` is split out for backends that do not call `run()`,
and `publish_outputs()` is the matching finalization step; batch and grid use
them in the following commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ance Neither the batch nor the grid backend went through `CNSJob.run()`, so neither normalized or atomically published anything. `HPCWorker` writes a shell script that pipes the `.inp` into CNS directly; `libgrid` packages jobs itself and copies artifacts back with `shutil.copy`. Artifacts were therefore execution-mode dependent: the same job produced different bytes on `mode = local` and `mode = batch`, and an interrupted grid transfer could leave a truncated file at a public output path. Batch: each task's script is materialized through `prepare_execution_input()` into a per-task temporary input, and each task is finalized through `publish_outputs(check_output_log=True)` once the worker terminates. Staging inputs are removed afterwards. Publication is per task, and deliberately not per worker. With `concat > 1` a single worker carries several models, and a worker that ends in any state other than `finished` can still hold tasks that succeeded; publishing only for finished workers would discard those alongside the one that failed. Equally, `publish_outputs` raises `CNSRunningError` for a task that produced nothing, and `HPCScheduler.run` catches only `KeyboardInterrupt` -- so letting that propagate would abort the entire workflow on a single faulty CNS job, where the local path absorbs it (`libparallel.Worker.run` catches per task) and `export_io_models` decides via `tolerance` whether enough models were produced. Each task is therefore validated, published and logged independently, and `tolerance` keeps making that decision everywhere. The job file also keeps its per-invocation diagnostics: a task whose input was never materialized still emits the plain `cns < input > output` line, so a dry-run or hand-inspected job file remains readable and the shell's own missing-input error is not swallowed. Grid: retrieved artifacts are copied to a hidden temporary name in the destination step directory, checked for existence and non-emptiness, normalized there, and then moved into place with `os.replace()`. An interrupted copy cannot expose a truncated public output. Compound suffixes are recognised, so a returned `foo.pdb.gz` is normalized rather than silently skipped. Also drains the `ThreadPoolExecutor.map` iterators in `GRIDScheduler`. `map` returns a lazy generator, so an exception raised inside `process_job`, `package()` or `submit()` was discarded unread -- including the incompleteness check added above, which could not have reported anything. This is a pre-existing defect; it is fixed here because the new checks depend on grid worker exceptions actually surfacing. The batch and grid paths are covered by unit tests only. There is no SLURM, Torque or DIRAC in the test environment, so the end-to-end behaviour of both is reasoned from the code rather than measured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A CNS job is a pure function: a script, every file it reads, and the executable, producing a PDB and optionally a PSF. Nothing about how HADDOCK3 arrived at that job -- which run directory, which step number, what the files were called, which module or which HADDOCK3 version authored it -- is part of the computation. But all of it is in the generated `.inp` and in the paths it names, so two byte- identical computations have no representation in which they look alike. `libs/libcnscanonical.py` builds one. For a `CNSJob` it produces a canonical script with every locator erased, together with a pin table binding each dependency to a canonical name and its content checksum. From those a stable identity can be computed for the job. The distinction the module is built around is between a **locator**, an **input**, and a **binding**: - a locator -- run directory, step ordinal, install path, filename, structure index where it only selects a file -- is erased, because the file's content is already its identity; - an input's content is hashed; - the *binding* -- which canonical pin an input occupies, and the shape of the declared output -- is part of the identity, because swapping two molecules between pins is a different computation even though the bytes are the same. The **executable** is the one thing named in a CNS job that is deliberately not hashed. It is not an input to the computation; it is the machine that evaluates it. The computation a job declares is its script together with the data that script reads, and the binary is the interpreter of that declaration. The `canonical-cns` pin therefore exists and keeps its position -- the executable is a named part of the mapping, not something the representation forgets -- but it is bound to a policy constant rather than to the executable's own bytes. Binding it to the bytes would produce an identity that no two installations can ever share, because no two of them compile or download the same binary. That is not a corner case; sharing identities across installations is the principal reason to compute one at all -- a lab-wide store, a store published alongside a paper, a workstation result carried to a cluster. Nor would the bytes buy safety: they cannot detect a CNS build that computes *different* results, only prevent two builds that agree from being recognised as agreeing, which is the overwhelmingly common case. A build that genuinely disagrees is a reproducibility problem this representation cannot fix and should not pretend to, and the honest treatment is to record which executable produced a result as provenance, where a mixture of builds is visible and auditable without being part of identity. That record is not part of this branch. So paths are not simply ignored on the input side. A path is erased and replaced by a canonical pin name that is itself part of the representation, which is what makes location independence and rank independence properties of a *stable* name-to-pin mapping rather than of names not mattering. Concretely the module resolves what a script reads -- including `@`/`@@` references, `MODULE:` and `TOPPAR:` environment-relative spellings, indexed symbol splices such as `@@$input_aa_psf_filename_$nchain`, and dynamic `$base + "_" + encode($count)` constructions -- and rewrites every path spelling to a canonical pin name, leaving CNS variable names as they are. The declared output is rewritten with them: `$output_pdb_filename` and `$output_psf_filename` are bound to the fixed `canonical-output.pdb` and `canonical-output.psf`, since the name a job writes to is a locator while the shape of what it writes is identity. It normalizes the logging-only `$log_level` and locator-only `$count` values to fixed literals, and checksums each dependency from its logical, uncompressed bytes so that compressed and uncompressed storage of the same content share an identity. A completeness guard rejects any canonical script still containing a work directory, run path, step folder, module root or toppar root, and asserts that the outputs the script binds are exactly the declared canonical ones. **This representation is virtual and has no production caller.** Ordinary runs are unchanged: CNS executes the generated input in the normal step layout, under the normal HADDOCK filenames, through the existing schedulers. `canonical_mapping()` is reachable only from tests. It is deliberately dead production code, retained because the caching stage will make cache-key construction its first real consumer, and it will remain virtual then -- CNS will not execute it. Executing the canonical form directly was investigated and rejected as a production architecture: it is technically feasible, but per-job workspaces add substantial metadata pressure on shared HPC filesystems, and node-local scratch would require a new staging, content-pooling, lifecycle and cross-filesystem publication subsystem spanning the local, MPI, batch and grid backends. The full analysis is recorded separately. One consequence should be stated plainly rather than discovered later: because nothing executes this representation, the tests here can establish that it is stable, location-independent and free of recognised leaks, but they cannot establish that the declared dependency set is *complete*. An undeclared dependency is invisible to a checksum-side test by construction. Proving completeness requires executing a job in an environment containing only what it declares, which is deferred to the later audit stage that dumps a job as a self-contained runnable command. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every canonicalization test so far builds a hand-written three-to-ten line CNS
script. Those pin the rules the library was written against, which is why they
all passed while real generated inputs failed: absolute install paths surviving
into topology keys, a topocg output binding consumed by a same-basename input,
and the `cgtoaa` symbol splice being rejected outright were all invisible to the
suite and had to be found by probing the library by hand. A parameter regression
that silently dropped 250 `eval` lines from a real `rigidbody_1.inp` was
likewise invisible.
Toy fixtures cannot see these, because the thing under test is not the rule but
what the rule does to input HADDOCK3 actually generates.
This adds one real generated input per shape -- topoaa, topocg, rigidbody,
flexref, emref, mdref, emscoring, mdscoring, cgtoaa -- canonicalized and
compared against a committed golden form. Two properties follow:
- A change in generated input surfaces as a reviewable diff. The parameter set,
the seeds, the pin assignment and the erasures are all visible in the golden
file, so a change to any of them has to be looked at and accepted rather than
silently absorbed.
- The naming rule itself becomes reviewable. Canonical pin assignment is a
dependency of every identity the library will ever compute: if the rule
changes -- pins numbered by order of first reference rather than by sorted
filename, say -- then every key changes, with no change to any content,
read-set or science. Freezing one canonical form per shape is what turns that
from an invisible event into a diff.
A golden form records what is *derived*, and nothing that is already in the
tree:
[pins] every canonical name, the file that occupies
it, and that file's content checksum
[outputs] the declared output shape
[recipe rewrites] what canonicalization does to the module's CNS
recipe, as before/after pairs with the number
of places each occurs
[canonical parameter header] the generated part of the input, verbatim
The recipe itself is deliberately not copied in. It is two thirds of a canonical
script, and canonicalization touches two to six of its lines; copying it would
mirror every recipe edit into a golden file that has no opinion about the
change, and teach a reviewer to regenerate without reading. Recording the
rewrites instead keeps the diff to what this test is about: editing a recipe
moves nothing here unless it changes what canonicalization has to do to it,
while a new `@@` read or path spelling in a recipe still appears, because that
is a rewrite.
The checksums close the other half of the same question. A canonical script
names a shared include such as `bestener.cns` by pin name, so editing one
changes every identity that depends on it while the script text stays
byte-identical; the pin table is where that becomes visible.
Two premises are asserted rather than assumed, since the sections are cut apart
by line: a module recipe is spliced in unchanged as the tail of the generated
input, and canonicalization rewrites lines in place rather than adding or
removing any.
The seeded shapes are given the seed production derives for them rather than a
literal, so the golden forms pin the seeding rule as well as the layout: a
change to how a seed is derived from a job shows up as a diff in five files
rather than as a silent change of every future identity. `mdscoring` gains a
`$seed` line it was always given in production and the fixture had been
omitting.
A companion test asserts, per module, that the parameter families a recipe can
construct by symbol splice are present in the generated input, which is the
specific failure the golden forms would otherwise only report as a large
unexplained diff.
The golden forms are generated artifacts, committed deliberately. They are
regenerated with `HADDOCK_UPDATE_CNS_GOLDENS=1` when the generated input
legitimately changes, and the diff is the review.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sjdv1982
requested review from
VGPReys,
amjjbonvin and
rvhonorato
and removed request for
rvhonorato
September 3, 2026 14:05
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do and why?
This is the first stage of the caching feature. The purpose is to fix several issues that make HADDOCK less reproducible (and therefore less cacheable) than it could be. For example, if we increase the sampling from 100 to 101, we want the first 100 structures to stay the same: as it is, the first 100 structures are often different because of different seeds.
It consists of 14 individual commits, each with a long commit message that describe what is being changed.
The first commit contains a design plan, essentially the long version of the description of the 4 stages.
How was this tested?
Most commits add their own tests. The full test set and integration test set passes.
AI assistance
AI (Claude Opus) wrote all the commit messages and AI (Codex and Claude) did the coding. The code changes were triggered by reproducibility errors identified by the AI when implementing stage 3 and testing it on stage 2.
Checklist
CHANGELOG.mdupdated for user-facing changesRelated issues
Notes for reviewers
This is preparatory work on the HADDOCK internals: user-facing work is in stage 2-3. See https://github.com/haddocking/haddock3/compare/caching-stage1?expand=1#diff-22967752f3bf7f43d4c2f7e748d12c8b2ff61c340e6fe499ff45023de5d8acba fot details.