Skip to content

Add data_manager_repeatmasker - #8214

Open
mvdbeek wants to merge 2 commits into
galaxyproject:mainfrom
mvdbeek:data-manager-repeatmasker
Open

Add data_manager_repeatmasker#8214
mvdbeek wants to merge 2 commits into
galaxyproject:mainfrom
mvdbeek:data-manager-repeatmasker

Conversation

@mvdbeek

@mvdbeek mvdbeek commented Jul 22, 2026

Copy link
Copy Markdown
Member

What

A new data manager that downloads a Dfam repeat library in the partitioned FamDB (v3) format and registers it in a new repeatmasker_famdb data table.

Why

The RepeatMasker Conda package no longer ships a populated repeat library — the bundled Dfam.h5 is a placeholder and RepeatMasker fails with Species "..." is not known to RepeatMasker because the library has no families. In Galaxy's mulled biocontainers the historical install-time download never runs (no network, post-link scripts not executed), so the library must be provisioned separately. This data manager does that.

How

  • Always downloads the required FamDB root partition; the admin selects which components to add (curated/uncurated consensus for rmblast, curated/uncurated HMM for nhmmer). Curated consensus is checked by default.
  • Each file is verified against its Dfam md5 sidecar, decompressed, and placed in a single directory that RepeatMasker (>= 4.2.4) consumes via -libdir.
  • Partition counts are discovered by probing 0,1,2,… until a 404, so it adapts to any release; adding a Dfam release is a one-line dict entry.

Data table

repeatmasker_famdb with columns value, name, version, path (path → the FamDB directory).

Testing

--test mode downloads only the smallest (~0.3 MB) partition to keep CI fast. Verified locally end to end against the live Dfam server: download + md5 verification + gunzip + data_manager_json output, producing a valid HDF5 file; md5-mismatch aborts; partition enumeration stops at the first 404.

The consuming tool change (RepeatMasker 4.2.4 + a cached source type that reads this table) is a companion PR.

Downloads a Dfam repeat library in the partitioned FamDB (v3) format and
registers it in the new repeatmasker_famdb data table, so RepeatMasker
(>= 4.2.4) can screen sequences against a real library instead of the empty
placeholder shipped in the RepeatMasker Conda package.

The root partition is always fetched; curated/uncurated consensus and HMM
components are selectable. Each file is md5-verified against its Dfam sidecar
and decompressed into a single -libdir directory. Partition counts are
discovered by probing rather than hard-coded per release.
Comment on lines +27 to +30
<option value="curated_consensus" selected="true">Curated consensus sequences (rmblast; recommended)</option>
<option value="uncurated_consensus">Uncurated consensus sequences (rmblast; very large)</option>
<option value="curated_hmm">Curated profile HMMs (nhmmer)</option>
<option value="uncurated_hmm">Uncurated profile HMMs (nhmmer; very large)</option>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do tools consuming the data need to know which options were selected? Then it might be nice to store it in the data table.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — they do. Added components and engines columns to repeatmasker_famdb in 05ec01f, so the table is now value, name, version, components, engines, path.

components records the selected keys (curated_consensus,curated_hmm), engines is derived from them (rmblast for consensus partitions, nhmmer for HMMs). The derived column earns its place because a consuming tool wants to ask "can this library serve rmblast?", and that's an OR over two component keys, which a data table filter can't express. Both are comma-separated for use with multiple_splitter.

The companion PR #8215 now uses it:

<options from_data_table="repeatmasker_famdb">
  <filter type="multiple_splitter" column="engines" separator="," />
  <filter type="static_value" column="engines" value="rmblast" />
</options>

so an HMM-only library is no longer offered to a tool that searches with rmblast.

While in there I also made the components parameter non-optional: root-only is not a usable install, and an empty multi-select rendered as the literal string None, which failed with a confusing Unknown component(s): None.

(Claude Opus 5 here, working with @mvdbeek.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand (in particular the last part)

While in there I also made the components parameter non-optional: root-only is not a usable install, and an empty multi-select rendered as the literal string None, which failed with a confusing Unknown component(s): None.

Is this referring to a specific test?

Comment thread data_managers/data_manager_repeatmasker/README.md Outdated
- Record the admin's selection in the data table: new `components` and
  `engines` columns, so a consuming tool can filter libraries by the
  search engine their partitions actually support (consensus -> rmblast,
  HMM -> nhmmer). Both are comma-separated for use with a
  multiple_splitter filter.
- Move the README content into the tool help, where admins installing
  the data manager will see it, and drop README.md.
- Require at least one component: the root partition alone contains
  almost no families, and an empty multi-select rendered as the string
  "None", which failed with a confusing "Unknown component(s): None".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage/Discuss

Development

Successfully merging this pull request may close these issues.

2 participants