osprey: Made the Stage-7 join fold run by run on every route, and retired --input-scores

pwiz_tools/Osprey/Documentation/Help/en/CommandLine.html

@@ -68,7 +68,6 @@ <h1>Osprey command-line usage</h1>
 <table>
 <tr><th>Argument</th><th>Description</th></tr>
 <tr><td><nobr>--task</nobr> &lt;SpectraCache | PerFileScoring | FirstPassFDR | PerFileRescoring | SecondPassFDR | ModelDiagnostics&gt;</td><td>HPC: run exactly one pipeline task (one node = one task). Omit for the full pipeline. SpectraCache stages the .spectra.bin caches; ModelDiagnostics regenerates only the <nobr>--model-diagnostics</nobr> report for a COMPLETED run, writing no other artifact.</td></tr>
-<tr><td><nobr>--input-scores</nobr> &lt;paths|dir&gt;</td><td>HPC: one or more .scores.parquet files, or a single directory (<nobr>non-recursive</nobr>). Mutex with <nobr>--input</nobr>.</td></tr>
 </table>
 <div class="RowType">Logging</div>
 <table>
@@ -87,22 +86,22 @@ <h1>Osprey command-line usage</h1>
 <tr><td><nobr>-h</nobr>,&nbsp;<nobr>--help</nobr></td><td>Show this help message ([ascii|unicode|sections|html|&lt;Section&gt;])</td></tr>
 <tr><td><nobr>-v</nobr>,&nbsp;<nobr>--version</nobr></td><td>Show version</td></tr>
 </table>
-<p>EXAMPLES:</p><p>  osprey -i sample.mzML -l library.tsv -o results.blib</p><p>  osprey -i *.mzML -l library.tsv -o results.blib --resolution hram</p><p>HPC SPLIT (one node = one --task): see --task / --input-scores above.</p><div class="RowType">Distributed execution (HPC)</div>
+<p>EXAMPLES:</p><p>  osprey -i sample.mzML -l library.tsv -o results.blib</p><p>  osprey -i *.mzML -l library.tsv -o results.blib --resolution hram</p><p>HPC SPLIT (one node = one --task): see --task above.</p><div class="RowType">Distributed execution (HPC)</div>
 <p>Run with no <code>--task</code> for the whole pipeline in one process. For distributed (HPC / workflow-engine) execution the pipeline splits at its join / fan-out boundaries into four single-task workers &mdash; one node = one <code>--task</code>: <code>PerFileScoring</code> (split, per file) &rarr; <code>FirstPassFDR</code> (join, all files) &rarr; <code>PerFileRescoring</code> (split, per file) &rarr; <code>SecondPassFDR</code> (join, all files). Pass the same <code>--library</code> and search options to every task; the parquet integrity check rejects inputs whose search/library hash does not match.</p>
 <pre>
 # split 1 - one process per mzML (writes &lt;stem&gt;.scores.parquet, &lt;stem&gt;.calibration.json beside each input)
 Osprey --task PerFileScoring -i s1.mzML -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
 
-# join 1 - one process over ALL parquets (pass a directory so the order is deterministic)
-Osprey --task FirstPassFDR --input-scores ./scores_dir -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
+# join 1 - one process over ALL runs (pass a sorted list so the order is deterministic)
+Osprey --task FirstPassFDR --input-list runs.txt -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
 #   writes beside each parquet: &lt;stem&gt;.1st-pass.fdr_scores.bin, &lt;stem&gt;.reconciliation.json
 
 # split 2 - one process per file (parquet + its two sidecars co-located)
-Osprey --task PerFileRescoring --input-scores s1.scores.parquet -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
+Osprey --task PerFileRescoring -i s1.mzML -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
 #   writes: &lt;stem&gt;.scores-reconciled.parquet
 
-# join 2 - one process over ALL reconciled parquets (writes out.blib)
-Osprey --task SecondPassFDR --input-scores ./reconciled_dir -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
+# join 2 - one process over ALL runs, reading their reconciled parquets (writes out.blib)
+Osprey --task SecondPassFDR --input-list runs.txt -l hela.tsv -o out.blib --resolution unit --protein-fdr 0.01
 </pre>
-<p><code>--input-scores</code> takes a directory (globbed and sorted internally) or an explicit file list (used in the order given). FirstPassFDR reconciliation is order-sensitive, so for <code>FirstPassFDR</code> and <code>SecondPassFDR</code> pass a directory or a deterministically sorted list. The rehydration sidecars must travel with their parquet into each worker's working directory. Let the scheduler do the fan-out (one file per split process) rather than <code>--parallel-files</code>, which is the single-node multi-file mode.</p>
+<p>EVERY task takes <code>-i</code>, naming the DATA files - the same names the first split was given. A join task derives each run's parquet and sidecars from the input stem, so the data file itself need not still exist: what has to be in the worker's working directory (or under <code>--output-dir</code>) is that run's artifacts. FirstPassFDR reconciliation is order-sensitive, so pass a deterministically sorted list - <code>--input-list</code> takes one path per line and is what a cohort past a few hundred runs needs, since <code>-i</code> spends the command line at O(files). Let the scheduler do the fan-out (one file per split process) rather than <code>--parallel-files</code>, which is the single-node multi-file mode.</p>
 </body></html>

pwiz_tools/Osprey/Osprey-workflow.html

@@ -395,7 +395,7 @@ <h1>Osprey DIA pipeline workflow</h1>
   <rect class="task-hdr-bg" x="120" y="0" width="960" height="96" rx="8"/>
   <text x="140" y="22" class="task-hdr-name">&#9656; --task FirstPassFDR</text>
   <text x="1060" y="22" text-anchor="end" class="task-hdr-role">join &#183; 1 node &#183; holds O(distinct), never O(runs x entries)</text>
-  <text x="140" y="40" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores.parquet</tspan>, <tspan class="sc-run">&lt;stem&gt;.calibration.json</tspan>  (every run, via --input-scores)</text>
+  <text x="140" y="40" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores.parquet</tspan>, <tspan class="sc-run">&lt;stem&gt;.calibration.json</tspan>  (every run, derived from --input)</text>
   <text x="140" y="56" class="task-hdr-io"><tspan class="task-hdr-key">out&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.1st-pass.fdr_scores.bin</tspan>, <tspan class="sc-run">.1st-pass.stratum.json</tspan>, <tspan class="sc-run">&lt;stem&gt;.reconciliation.json</tspan>   &#183;   validity  <tspan class="sc-run">&lt;out&gt;.FirstPassFDR.osprey.task</tspan></text>
   <text x="140" y="72" class="task-hdr-io"><tspan class="task-hdr-key">out&#160;</tspan><tspan class="sc-exp">&lt;blib-stem&gt;.1st-pass.fdr_experiment.bin</tspan>, <tspan class="sc-exp">.1st-pass.retained_base_ids.bin</tspan> (compaction key), <tspan class="sc-exp">&lt;stem&gt;.1st-pass.model.json</tspan></text>
   <text x="1060" y="88" text-anchor="end" class="task-hdr-relay">relay: every experiment-wide file to EVERY downstream node</text>
@@ -462,7 +462,7 @@ <h1>Osprey DIA pipeline workflow</h1>
   <rect class="task-hdr-bg" x="120" y="0" width="960" height="96" rx="8"/>
   <text x="140" y="22" class="task-hdr-name">&#9656; --task PerFileRescoring</text>
   <text x="1060" y="22" text-anchor="end" class="task-hdr-role">per-run fan-out &#183; reads its own runs + the experiment baseline only</text>
-  <text x="140" y="40" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores.parquet</tspan> (via --input-scores) + <tspan class="sc-run">.1st-pass.fdr_scores.bin</tspan>, <tspan class="sc-run">.reconciliation.json</tspan>, <tspan class="sc-run">.calibration.json</tspan>, <tspan class="sc-cache">.spectra.bin</tspan></text>
+  <text x="140" y="40" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores.parquet</tspan> (derived from --input) + <tspan class="sc-run">.1st-pass.fdr_scores.bin</tspan>, <tspan class="sc-run">.reconciliation.json</tspan>, <tspan class="sc-run">.calibration.json</tspan>, <tspan class="sc-cache">.spectra.bin</tspan></text>
   <text x="140" y="56" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-exp">&lt;blib-stem&gt;.1st-pass.fdr_experiment.bin</tspan>, <tspan class="sc-exp">.1st-pass.retained_base_ids.bin</tspan>, <tspan class="sc-exp">&lt;stem&gt;.1st-pass.model.json</tspan>   &#183;   resident baseline</text>
   <text x="140" y="72" class="task-hdr-io"><tspan class="task-hdr-key">out&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores-reconciled.parquet</tspan>, <tspan class="sc-run">.2nd-pass.fdr_decoys.bin</tspan>, <tspan class="sc-run">.2nd-pass.fdr_scores.bin</tspan> (pass-2 worker)   &#183;   validity  <tspan class="sc-run">&lt;out&gt;.PerFileRescoring.osprey.task</tspan></text>
   <text x="1060" y="88" text-anchor="end" class="task-hdr-relay">relay: the run's own set + every experiment-wide file</text>
@@ -506,7 +506,7 @@ <h1>Osprey DIA pipeline workflow</h1>
   <rect class="task-hdr-bg" x="120" y="0" width="960" height="96" rx="8"/>
   <text x="140" y="22" class="task-hdr-name">&#9656; --task SecondPassFDR</text>
   <text x="1060" y="22" text-anchor="end" class="task-hdr-role">join &#183; 1 node &#183; final aggregation</text>
-  <text x="140" y="40" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores-reconciled.parquet</tspan> (via --input-scores; falls back to <tspan class="sc-run">&lt;stem&gt;.scores.parquet</tspan>), <tspan class="sc-run">.2nd-pass.fdr_scores.bin</tspan> (worker)</text>
+  <text x="140" y="40" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-run">&lt;stem&gt;.scores-reconciled.parquet</tspan> (every run - one parquet per run, no <tspan class="sc-run">.scores.parquet</tspan> fallback), <tspan class="sc-run">.2nd-pass.fdr_scores.bin</tspan> (worker)</text>
   <text x="140" y="56" class="task-hdr-io"><tspan class="task-hdr-key">in&#160;&#160;</tspan><tspan class="sc-exp">&lt;blib-stem&gt;.1st-pass.fdr_experiment.bin</tspan>, <tspan class="sc-exp">&lt;stem&gt;.1st-pass.model.json</tspan>, <tspan class="sc-exp">.1st-pass.stratum.json</tspan>, <tspan class="sc-run">&lt;stem&gt;.reconciliation.json</tspan>, <tspan class="sc-run">.calibration.json</tspan></text>
   <text x="140" y="72" class="task-hdr-io"><tspan class="task-hdr-key">out&#160;</tspan><tspan class="sc-exp">&lt;output&gt;.blib</tspan>, <tspan class="sc-exp">&lt;blib-stem&gt;.2nd-pass.fdr_experiment.bin</tspan>, <tspan class="sc-run">&lt;stem&gt;.2nd-pass.fdr_scores.bin</tspan> (where no worker ran)   &#183;   validity  <tspan class="sc-exp">&lt;out&gt;.SecondPassFDR.osprey.task</tspan></text>
   <text x="140" y="88" class="task-hdr-relay">holds O(distinct) &#183; folds the runs one at a time, rebuilding each from its own artifacts and dropping it</text>

pwiz_tools/Osprey/Osprey.Core/OspreyConfig.cs

@@ -328,22 +328,15 @@ public class OspreyConfig
         /// Pipeline-membership flag (read by each task's <c>IsIncluded</c>):
         /// include only the per-file fan-out, not the joining tasks. Set by both
         /// <c>--task PerFileScoring</c> and <c>--task PerFileRescoring</c>; the
-        /// concrete behavior depends on the input type. With <c>-i</c> mzML it
-        /// is the Stage 1-4 worker — each input produces a
-        /// <c>{stem}.scores.parquet</c> next to it, no FDR, no blib. With
-        /// <see cref="InputScores"/> it is the Stage 6 rescore worker. The two
-        /// are told apart by input type (see <see cref="SelectedTask"/>).
+        /// concrete behavior depends on which of the two selected it.
+        /// <c>PerFileScoring</c> is the Stage 1-4 worker - each input produces a
+        /// <c>{stem}.scores.parquet</c> next to it, no FDR, no blib;
+        /// <c>PerFileRescoring</c> is the Stage 6 rescore worker. The two are told apart
+        /// by <see cref="SelectedTask"/>, which is the only thing that ever decided it -
+        /// they used to be told apart by input KIND as well, and that second seam is gone.
         /// </summary>
         public bool NoJoin { get; set; }
 
-        /// <summary>
-        /// HPC scoring split: when set (non-null, non-empty), skip Stages 1-4
-        /// entirely and load these per-file scoring caches as the starting
-        /// point for Stage 5+. Set by <c>--input-scores</c>. When set,
-        /// <see cref="InputFiles"/> is ignored.
-        /// </summary>
-        public List<string> InputScores { get; set; }
-
         /// <summary>
         /// HPC: when true, exit after Stage 5 + reconciliation planning,
         /// having written the boundary files
@@ -371,10 +364,10 @@ public class OspreyConfig
         /// membership flags above (<see cref="NoJoin"/>,
         /// <see cref="StopAfterStage5"/>, <see cref="ExpectReconciledInput"/>)
         /// are derived from this and drive each task's <c>IsIncluded</c>; this
-        /// property additionally lets argument validation enforce the
-        /// task&#8596;input-type contract (e.g. PerFileScoring takes mzML,
-        /// PerFileRescore takes <see cref="InputScores"/>) and name the task the
-        /// user actually typed in error messages.
+        /// property additionally lets argument validation name the task the user actually
+        /// typed in error messages. It no longer has an input-KIND contract to enforce:
+        /// every task takes the same data files, and the second seam that said "you handed
+        /// me parquets, so Stage 1-4 is done" has retired into these flags.
         /// </summary>
         public HpcTask? SelectedTask { get; set; }
 

pwiz_tools/Osprey/Osprey.Core/ResidentPaths.cs

@@ -132,11 +132,15 @@ public static class ResidentPaths
         /// disclosure table had to be invented to see it at all.</para>
         ///
         /// <para>It admits ONLY the chosen case. A run that takes the resident join because no
-        /// streamed one was admissible - a straight-through run, where
-        /// <c>ExpectReconciledInput</c> is false - is not refused by this token, because there
-        /// is nothing for the operator to choose. That case goes when
-        /// <c>CanStreamStage7Join</c>'s admission stops being expressed as a CLI flag; then this
-        /// token is the only way to be resident, and it can be deleted with the switch.</para>
+        /// streamed one was admissible is not refused by this token, because there is nothing
+        /// for the operator to choose. That set has SHRUNK to one: the straight-through run
+        /// used to be in it - <c>CanStreamStage7Join</c>'s admission was the CLI flag
+        /// <c>ExpectReconciledInput</c>, so the ordinary run could not stream by construction -
+        /// and it is now derived from the reconciled parquets on disk, which every route
+        /// satisfies. What is left is a pass-2 mode whose per-file half has no worker
+        /// (<c>OSPREY_PASS2_QVALUE=transfer</c>), and when that half moves to
+        /// <c>Pass2PerFileWorker</c> this exemption has no subject, the guard can refuse
+        /// unconditionally, and this token becomes the only way to be resident.</para>
         /// </summary>
         public static readonly string STAGE7_STREAM_OFF = @"stage7-stream-off";