Add data_manager_repeatmasker - #8214
Conversation
Downloads a Dfam repeat library in the partitioned FamDB (v3) format and registers it in the new repeatmasker_famdb data table, so RepeatMasker (>= 4.2.4) can screen sequences against a real library instead of the empty placeholder shipped in the RepeatMasker Conda package. The root partition is always fetched; curated/uncurated consensus and HMM components are selectable. Each file is md5-verified against its Dfam sidecar and decompressed into a single -libdir directory. Partition counts are discovered by probing rather than hard-coded per release.
| <option value="curated_consensus" selected="true">Curated consensus sequences (rmblast; recommended)</option> | ||
| <option value="uncurated_consensus">Uncurated consensus sequences (rmblast; very large)</option> | ||
| <option value="curated_hmm">Curated profile HMMs (nhmmer)</option> | ||
| <option value="uncurated_hmm">Uncurated profile HMMs (nhmmer; very large)</option> |
There was a problem hiding this comment.
Do tools consuming the data need to know which options were selected? Then it might be nice to store it in the data table.
There was a problem hiding this comment.
Good point — they do. Added components and engines columns to repeatmasker_famdb in 05ec01f, so the table is now value, name, version, components, engines, path.
components records the selected keys (curated_consensus,curated_hmm), engines is derived from them (rmblast for consensus partitions, nhmmer for HMMs). The derived column earns its place because a consuming tool wants to ask "can this library serve rmblast?", and that's an OR over two component keys, which a data table filter can't express. Both are comma-separated for use with multiple_splitter.
The companion PR #8215 now uses it:
<options from_data_table="repeatmasker_famdb">
<filter type="multiple_splitter" column="engines" separator="," />
<filter type="static_value" column="engines" value="rmblast" />
</options>so an HMM-only library is no longer offered to a tool that searches with rmblast.
While in there I also made the components parameter non-optional: root-only is not a usable install, and an empty multi-select rendered as the literal string None, which failed with a confusing Unknown component(s): None.
(Claude Opus 5 here, working with @mvdbeek.)
There was a problem hiding this comment.
I do not understand (in particular the last part)
While in there I also made the components parameter non-optional: root-only is not a usable install, and an empty multi-select rendered as the literal string None, which failed with a confusing Unknown component(s): None.
Is this referring to a specific test?
- Record the admin's selection in the data table: new `components` and `engines` columns, so a consuming tool can filter libraries by the search engine their partitions actually support (consensus -> rmblast, HMM -> nhmmer). Both are comma-separated for use with a multiple_splitter filter. - Move the README content into the tool help, where admins installing the data manager will see it, and drop README.md. - Require at least one component: the root partition alone contains almost no families, and an empty multi-select rendered as the string "None", which failed with a confusing "Unknown component(s): None".
What
A new data manager that downloads a Dfam repeat library in the partitioned FamDB (v3) format and registers it in a new
repeatmasker_famdbdata table.Why
The RepeatMasker Conda package no longer ships a populated repeat library — the bundled
Dfam.h5is a placeholder and RepeatMasker fails withSpecies "..." is not known to RepeatMaskerbecause the library has no families. In Galaxy's mulled biocontainers the historical install-time download never runs (no network, post-link scripts not executed), so the library must be provisioned separately. This data manager does that.How
-libdir.0,1,2,…until a 404, so it adapts to any release; adding a Dfam release is a one-line dict entry.Data table
repeatmasker_famdbwith columnsvalue, name, version, path(path → the FamDB directory).Testing
--testmode downloads only the smallest (~0.3 MB) partition to keep CI fast. Verified locally end to end against the live Dfam server: download + md5 verification + gunzip +data_manager_jsonoutput, producing a valid HDF5 file; md5-mismatch aborts; partition enumeration stops at the first 404.The consuming tool change (RepeatMasker 4.2.4 + a
cachedsource type that reads this table) is a companion PR.