Conversation
nrminor
force-pushed
the
v3.5.0-rc
branch
3 times, most recently
from
August 11, 2026 19:03
2777717 to
11204bc
Compare
nrminor
force-pushed
the
v3.5.0-rc
branch
4 times, most recently
from
August 16, 2026 12:41
6e6dcab to
bd42114
Compare
nrminor
marked this pull request as ready for review
August 25, 2026 15:21
Remove the post-BLAST query-group gates that required a non-phage viral hit after MEGABLAST and BLASTN. Target enrichment and depletion continue to control which sequences reach BLAST, while every annotated top-hit group now proceeds directly to merging, LCA annotation, and reporting. MEGABLAST partitioning and existing hit-retention limits remain unchanged. Delete the obsolete Nextflow processes, Python filter and summary scripts, unit tests, filter decision artifacts, and sequence-flow rows. Rename affected channels around annotated-hit semantics, publish annotated MEGABLAST and BLASTN TSVs at their result-family roots, and remove the virus-only result directories. Describe the existing enrichment parameters as target-enrichment controls in executable help without renaming the 3.x API. Normalize the sequence-flow process input to the codebase's scalar-or-list convention because removing the filter summaries allows valid runs with a single evidence file.
Use the v3.4.0 checkout in the NVD_REPO installation example after the v3.4 release line, while preserving the existing launcher and compatibility guidance.
Bump __version__, the Nextflow manifest, pyproject, and uv.lock to 3.5.0, and publish schemas/nvd-params.v3.5.0.schema.json as the parameter contract for the release. The rolling nvd-params.latest.schema.json symlink and params.SCHEMA_URL both move to it, so IDE completion and generated templates describe 3.5. The v3.5.0 schema is a copy of the v3.4.0 contract with only $id changed. That contract already reflects the v3.4.0 merge: the six sourmash reference and threshold params are gone, labkey_insert_batch_size and skip_unassembled_read_queries are present, and merge_pairs defaults to true. The rapid-screen skip option that originally introduced this schema was dropped during the rebase onto the v3.4.0 merge, since sourmash rapid screening no longer exists to skip. Only its version plumbing survives here. Also drops bin/filter_non_virus_blast_nodes.py, bin/summarize_blast_filtering.py, and bin/test_filter_non_virus_blast_nodes.py. "feat: retain BLAST hits across taxa" deleted all three, a bad merge resurrected them here (the summarize_blast_filtering.py copy predating its qseqid column check), and nothing in the pipeline referenced them.
Carry validated input and retained counts on Deacon outputs, route zero-retention samples directly to completion, and keep downstream reporting aware of samples that cannot produce contigs. Cover the empty route through workflow and MultiQC boundaries.
Upgrade Deacon to 0.16 and use its explicit interleaved-input mode for paired streams and files. Add an opt-in NVD check-pairs parameter that asks Deacon to reject mismatched CASAVA or /1 and /2 record names without changing the default compatibility policy. Propagate the validation setting through direct, streamed, and SRA-backed paired filtering while leaving single-read, contig, and post-merge per-record filtering unchanged.
Apply the configured Deacon target index independently to both BBMerge output classes when target enrichment and pair merging are enabled. This removes nonmatching former mates retained by the pair-atomic initial pass while preserving genuinely interleaved, merge-disabled, and originally single-read paths. Both post-merge classes are refiltered: overlap_merged_pair, and the single_read class BBMerge emits for pairs that did not overlap. Reads that never entered MERGE_PAIRS, meaning anything that is not Illumina interleaved, keep the initial pass untouched, because pair-atomic retention was never a question for them. merge_pairs defaults to true as of v3.4.0, so merging and then refiltering is ordinary behaviour for every run that supplies a target index, not an opt-in experiment. The gate is target enrichment plus merging, so --no-merge-pairs or --no-enrichment turns it off. Emit one Deacon summary per post-merge query class and publish those summaries beneath the existing target-enrichment results at summaries/postmerge. MultiQC reporting is unchanged; the summaries are retained for later reporting decisions. Exercise the semantic premise directly against the locked Deacon CLI: a single FASTQ containing one matching and one nonmatching former mate retains only the match and reports two sequences in and one out. Channel routing remains intentionally reserved for end-to-end validation.
"feat!: remove sourmash rapid screening" deleted test_sourmash_reference_params along with the six reference and threshold params it exercised, and took the following `class TestNvdParamsDefaults:` header with it. The 24 default-value tests below that point silently became TestNvdParamsToNextflowArgs methods. Nothing stopped running, so nothing failed and nothing warned. pytest collects the same 110 tests in lib/py_nvd/test_models.py before and after this change; they are only reparented back to the class whose docstring describes them. Reinstate the header immediately before test_default_cutoff_percent, which is where it sat before the removal.
"feat: retain BLAST hits across taxa" reframed the virus_* parameters as generic target-enrichment controls in models.py, deliberately leaving the 3.x parameter names alone. schemas/nvd-params.v3.5.0.schema.json did not exist at that point, so it inherited the v3.4.0 wording and still described a prebuilt "vertebrate-infecting virus deacon index" driving "virus read enrichment". That is the text IDEs show on hover, so the published contract described a narrower feature than the pipeline implements. Rewords eight descriptions to match the model. The schema's longer phrasing is kept deliberately: these descriptions are hover documentation and say more than the terse CLI help strings, so this is not a blanket sync of the two. schemas/nvd-params.v3.4.0.schema.json is left alone. It is the published contract for a released version whose enrichment really was described that way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After #31, this release is much smaller. Its many feature addition is direct streaming of reads from SRA into our deacon enrichment, with continued support for no-enrichment runs as well. This means entire bioprojects of accessions, for example, can be put through
nvdwithout persisting the entirety of each FASTQ onto disk first. Instead, reads are streamed directly into deacon filtering. Reads are thus only written to disk if they pass filtering with the provided index. This feature does have a downside: if you wish to download the entire FASTQs and then use nvd resumes to filter with different indexes,nvddoes not currently support that. Reads must be re-downloaded from SRA each time. This was a design choice that can be adjust in the future if it poses problems, though ultimately, we do not recommend using nvd resumes to reprocess with different parameters because each resumed run will overwrite previous runs' results!This release also introduces a
skip_rapid_screenparameter, which skips our currently sourmash-based rapid screening subworkflow.Both of these features were ultimately crucial in dholab's ability to scan our entire wastewater bioproject on SRA for Cyclospora reads during the summer 2026 outbreak.
Additional other improvements include zero-read samples more clearly being reported, development tooling moving to Mise instead of a Nix Flake+direnv, some updated pixi dependencies, and improved repository-override documentation.