RepeatMasker: update to 4.2.4 and add cached Dfam (FamDB) library support - #8215
Open
mvdbeek wants to merge 6 commits into
Open
RepeatMasker: update to 4.2.4 and add cached Dfam (FamDB) library support#8215mvdbeek wants to merge 6 commits into
mvdbeek wants to merge 6 commits into
Conversation
…port RepeatMasker 4.2.4 uses the partitioned FamDB (v3) library format, no longer bundles a repeat library, and drops the -libdir option, so the previous "DFam (bundled)" and "DFam (full/specific version, single .h5)" source types no longer work. - Bump the tool to RepeatMasker 4.2.4 and add a famdb 3.0.0 requirement. - Replace the dfam / dfam_up source types with a "cached" source that reads a Dfam library from the new repeatmasker_famdb data table (populated by data_manager_repeatmasker). Families for the requested clade are exported with `famdb.py ... families -a -d -f fasta_name --include-class-in-name` and passed to RepeatMasker as a custom -lib. The custom-FASTA "library" source is unchanged. - Add the repeatmasker_famdb tool data table (sample + test configs, loc sample). - Replace the dfam/dfam_up tests with a cached test backed by a tiny FamDB v3 fixture whose curated-consensus families carry the repeats.fasta sequences on clade 'Genus', so the cached path masks the same positions as the library test.
github-project-automation
Bot
moved this to Triage/Discuss
in Galaxy Tool Developers SIG Board
…dled famdb.py) The two-requirement (repeatmasker + famdb) tool forced a mulled multi-package biocontainer that is not built, so no container resolved and every test failed with exit 127 (RepeatMasker not found). The repeatmasker package already depends on famdb, so famdb.py is present in the single-package repeatmasker biocontainer.
… outputs RepeatMasker 4.2.4 probes its configured FamDB at startup even in -lib mode, and the Conda package's FamDB has no library data, so masking died with "FamDB data directory not found". Clearing FAMDB_DIR for the RepeatMasker invocation skips that probe (the cached source already ran famdb.py directly via -i to build the -lib FASTA). Also regenerate the expected test outputs against the repeatmasker 4.2.4 biocontainer: masked/log/align/poly/gff are unchanged from 4.1.5; stats and cat differ only in version strings. Add dedicated expected files for the cached test (same masked positions as the library test, but families are classified via the FamDB export). Verified by running famdb.py + RepeatMasker in the quay.io repeatmasker:4.2.4 biocontainer.
…sker -v) The 4.2.4 biocontainer ships only 'RepeatMasker' (capital), not a lowercase 'repeatmasker', so the old version_command 'repeatmasker --version' failed with exit 127 and aborted every job. 'RepeatMasker -v' prints the version and exits 0. Verified the full job (version command + famdb.py export + masking) end to end in the quay.io repeatmasker:4.2.4 biocontainer as uid 1001 under set -e.
6 tasks
…names Both repeat library sources now run RepeatMasker with -lib, so -cutoff applies to the cached Dfam export as well; move it out of the 'library' branch to a top-level parameter. Add common taxon suggestions to the cached species field. The valid values are every taxon name in the installed FamDB partition, so a select is not possible, but a text parameter with options offers the common ones while still accepting any name.
The data manager now records which FamDB components were installed and which search engines they support. Track the extra columns, and offer only libraries usable with rmblast, since that is the engine this tool searches with -- an HMM-only library would otherwise be selectable and then fail.
Contributor
|
This looks ready to me. I would merge after: #8214 (comment) is resolved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Updates RepeatMasker 4.1.5 → 4.2.4 and reworks its repeat-library handling for the partitioned FamDB (v3) format.
Why
RepeatMasker 4.2.4 uses the partitioned FamDB (v3) library format, no longer bundles a repeat library, and drops the
-libdiroption. The tool's previousdfam(bundled curated) anddfam_up(single.h5) source types therefore no longer work: the bundled library is gone, and the single-file.h5format is obsolete.How
famdb3.0.0 requirement.dfam/dfam_upwith acachedsource type that reads a Dfam library from the newrepeatmasker_famdbdata table (populated bydata_manager_repeatmasker, companion PR Add data_manager_repeatmasker #8214). Families for the requested clade are exported withfamdb.py -i <lib> families -a -d -f fasta_name --include-class-in-name '<species>'and passed to RepeatMasker as a custom-lib. The custom-FASTAlibrarysource is unchanged.repeatmasker_famdbtool data table (tool_data_table_conf.xml.sample+.test,tool-data/*.loc.sample).dfam/dfam_uptests with a cached test backed by a tiny FamDB v3 fixture (root + curated-consensus, ~70 KB) whose families carry therepeats.fastasequences on cladeGenus, so the cached path masks the same positions as thelibrarytest.Notes for review
repeatmasker_famdb).FamDBRoot/FamDBLeafclasses +build_pruned_tree; I verified locally that the exact wrapperfamdb.pyexport command returns all families with RepeatMasker-style headers, byte-identical in sequence torepeats.fasta..masked/.stats/.cat/.log, generated under 4.1.5) likely need regeneration against 4.2.4 in CI — thecachedtest useslines_difftolerances in the meantime.