Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions CITATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,50 @@

> Shen W, Le S, Li Y, Hu F. SeqKit: A Cross-Platform and Ultrafast Toolkit for FASTA/Q File Manipulation. PLoS One. 2016 Oct 5;11(10):e0163962. doi: 10.1371/journal.pone.0163962. PMID: 27706213; PMCID: PMC5051824.

## Read preprocessing tools

Only the tools you select through the preprocessing parameters are run. See `docs/usage.md`.

- [AdapterRemoval](https://pubmed.ncbi.nlm.nih.gov/26868221/)

> Schubert M, Lindgreen S, Orlando L. AdapterRemoval v2: rapid adapter trimming, identification, and read merging. BMC Res Notes. 2016 Feb 12;9:88. doi: 10.1186/s13104-016-1900-2. PMID: 26868221; PMCID: PMC4751634.

- [BBMap/BBDuk/Clumpify](https://sourceforge.net/projects/bbmap/)

> Bushnell B. (2022) BBMap. sourceforge.net/projects/bbmap/.

- [Cutadapt](https://doi.org/10.14806/ej.17.1.200)

> Martin M. Cutadapt removes adapter sequences from high-throughput sequencing reads. EMBnet.journal. 2011;17(1):10-12. doi: 10.14806/ej.17.1.200.

- [Deacon](https://github.com/bede/deacon)

> Constantinides B. Deacon: fast alignment-free host depletion. github.com/bede/deacon.

- [fastp](https://pubmed.ncbi.nlm.nih.gov/30423086/)

> Chen S, Zhou Y, Chen Y, Gu J. fastp: an ultra-fast all-in-one FASTQ preprocessor. Bioinformatics. 2018 Sep 1;34(17):i884-i890. doi: 10.1093/bioinformatics/bty560. PMID: 30423086; PMCID: PMC6129281.

- [Hostile](https://pubmed.ncbi.nlm.nih.gov/38191289/)

> Constantinides B, Hunt M, Crook DW. Hostile: accurate decontamination of microbial host sequences. Bioinformatics. 2023 Dec 1;39(12):btad728. doi: 10.1093/bioinformatics/btad728. PMID: 38191289; PMCID: PMC10746824.

- [leeHom](https://pubmed.ncbi.nlm.nih.gov/25100869/)

> Renaud G, Stenzel U, Kelso J. leeHom: adaptor trimming and merging for Illumina sequencing reads. Nucleic Acids Res. 2014 Oct;42(18):e141. doi: 10.1093/nar/gku699. PMID: 25100869; PMCID: PMC4191382.

- [PRINSEQ++](https://doi.org/10.7287/peerj.preprints.27553v1)

> Cantu VA, Sadural J, Edwards R. PRINSEQ++, a multi-threaded tool for fast and efficient quality control and preprocessing of sequencing datasets. PeerJ Preprints. 2019;7:e27553v1. doi: 10.7287/peerj.preprints.27553v1.

- [Trim Galore!](https://www.bioinformatics.babraham.ac.uk/projects/trim_galore/)

> Krueger F. Trim Galore!: a wrapper around Cutadapt and FastQC. bioinformatics.babraham.ac.uk/projects/trim_galore/.

- [Trimmomatic](https://pubmed.ncbi.nlm.nih.gov/24695404/)

> Bolger AM, Lohse M, Usadel B. Trimmomatic: a flexible trimmer for Illumina sequence data. Bioinformatics. 2014 Aug 1;30(15):2114-20. doi: 10.1093/bioinformatics/btu170. PMID: 24695404; PMCID: PMC4103590.

## Software packaging/containerisation tools

- [Anaconda](https://anaconda.com)
Expand Down
1 change: 1 addition & 0 deletions conf/containers_conda_lock_files_amd64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { conda = 'modules/nf-core/fastp/.conda-lock/linux_amd64-bd-4df8d6c11b471bde_1.txt' } }
process { withName: 'FASTQC' { conda = 'modules/nf-core/fastqc/.conda-lock/linux_amd64-bd-5cb1a2fa2f18c7c2_1.txt' } }
process { withName: 'MULTIQC' { conda = 'modules/nf-core/multiqc/.conda-lock/linux_amd64-bd-c17fb751507e9dfc_1.txt' } }
process { withName: 'RGI_BWT' { conda = 'modules/nf-core/rgi/bwt/.conda-lock/linux_amd64-bd-e905ecb8305e2609_1.txt' } }
1 change: 1 addition & 0 deletions conf/containers_conda_lock_files_arm64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { conda = 'modules/nf-core/fastp/.conda-lock/linux_arm64-bd-95b49074d3fe9875_1.txt' } }
process { withName: 'FASTQC' { conda = 'modules/nf-core/fastqc/.conda-lock/linux_arm64-bd-e455e32f745abe68_1.txt' } }
process { withName: 'MULTIQC' { conda = 'modules/nf-core/multiqc/.conda-lock/linux_arm64-bd-5c84a5000a226ab5_1.txt' } }
process { withName: 'RGI_BWT' { conda = 'modules/nf-core/rgi/bwt/.conda-lock/linux_arm64-bd-fab5219836c1d880_1.txt' } }
1 change: 1 addition & 0 deletions conf/containers_docker_amd64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { container = 'community.wave.seqera.io/library/fastp:1.3.6--4df8d6c11b471bde' } }
process { withName: 'FASTQC' { container = 'community.wave.seqera.io/library/fastqc:0.12.1--5cb1a2fa2f18c7c2' } }
process { withName: 'MULTIQC' { container = 'community.wave.seqera.io/library/multiqc:1.35--c17fb751507e9dfc' } }
process { withName: 'RGI_BWT' { container = 'community.wave.seqera.io/library/rgi_kma:e905ecb8305e2609' } }
1 change: 1 addition & 0 deletions conf/containers_docker_arm64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { container = 'community.wave.seqera.io/library/fastp:1.3.6--95b49074d3fe9875' } }
process { withName: 'FASTQC' { container = 'community.wave.seqera.io/library/fastqc:0.12.1--e455e32f745abe68' } }
process { withName: 'MULTIQC' { container = 'community.wave.seqera.io/library/multiqc:1.35--5c84a5000a226ab5' } }
process { withName: 'RGI_BWT' { container = 'community.wave.seqera.io/library/rgi_kma:fab5219836c1d880' } }
1 change: 1 addition & 0 deletions conf/containers_singularity_https_amd64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/d0/d013aad5427d824afe472e6607ea47685ff0181f1fb09e52a179e0ec39e43e88/data' } }
process { withName: 'FASTQC' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/f2/f20b021476d1d87658820f971ebecc1e8cdbde0f338eb0d9cea2b0a8fc54a54b/data' } }
process { withName: 'MULTIQC' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/c8/c8e346f4f6080eadf1253505e6ff09ef004454fc18e8d672006fd7b222cc412e/data' } }
process { withName: 'RGI_BWT' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/3f/3f452c8e124ee58ab6b26442d15401c57d471cb753f53921570dc484df4e7620/data' } }
1 change: 1 addition & 0 deletions conf/containers_singularity_https_arm64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/42/42ea3a248c30a6eb0ab5d5ca35ce8b50aab6ca75e982d43420779214517a516b/data' } }
process { withName: 'FASTQC' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/46/46daf2dad0169afd2ae047c3e50ed3776259f664bf07e5e06b045dc23449e994/data' } }
process { withName: 'MULTIQC' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/e4/e48aa28aebc881254a499b24c3e1ce77b8df1b85a5432699ed6f72eb17ac7fb5/data' } }
process { withName: 'RGI_BWT' { container = 'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/c8/c8e83b71b5dae10e1d771490c36f49d314718888c3cc8214350eb7f149090f0e/data' } }
1 change: 1 addition & 0 deletions conf/containers_singularity_oras_amd64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { container = 'oras://community.wave.seqera.io/library/fastp:1.3.6--63f52bbe4fa9aee1' } }
process { withName: 'FASTQC' { container = 'oras://community.wave.seqera.io/library/fastqc:0.12.1--5c4bd442468d75dd' } }
process { withName: 'MULTIQC' { container = 'oras://community.wave.seqera.io/library/multiqc:1.35--c680f2aea25ccec2' } }
process { withName: 'RGI_BWT' { container = 'oras://community.wave.seqera.io/library/rgi_kma:06c9375141c15d16' } }
1 change: 1 addition & 0 deletions conf/containers_singularity_oras_arm64.config
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
process { withName: 'FASTP' { container = 'oras://community.wave.seqera.io/library/fastp:1.3.6--469416be288311d1' } }
process { withName: 'FASTQC' { container = 'oras://community.wave.seqera.io/library/fastqc:0.12.1--127a87fc06499035' } }
process { withName: 'MULTIQC' { container = 'oras://community.wave.seqera.io/library/multiqc:1.35--c0468833d65b2f81' } }
process { withName: 'RGI_BWT' { container = 'oras://community.wave.seqera.io/library/rgi_kma:a6eaadaaffbeaf85' } }
64 changes: 64 additions & 0 deletions conf/modules.config
Original file line number Diff line number Diff line change
Expand Up @@ -89,4 +89,68 @@ process {
mode: params.publish_dir_mode
]
}

// FASTQ_SHORTREADS_PREPROCESS_QC ships these in its own nextflow.config, which Nextflow
// does not load for an installed subworkflow. Its meta.yml asks the pipeline to copy them
// here, and without them the complexity filters and the deduplication step do nothing.
withName: ".*:FASTQ_COMPLEXITY_FILTER:BBMAP_BBDUK" {
ext.args = [
'entropy=0.7',
'minlength=0'
].join(' ')
ext.prefix = { "${meta.id}.trim" }
}

withName: ".*:FASTQ_COMPLEXITY_FILTER:FASTP" {
ext.args = [
'--low_complexity_filter',
'--disable_adapter_trimming',
'--disable_trim_poly_g',
'--disable_quality_filtering',
'--disable_length_filtering'
].join(' ')
}

withName: "BBMAP_CLUMPIFY" {
ext.args = 'dedupe=t'
}

// Intermediate FASTQ files are only worth keeping when the user asks for them; the logs and
// reports are small and always published so MultiQC and the reader can see what each step did.
withName: ".*:FASTQ_SHORTREADS_PREPROCESS_QC:.*" {
publishDir = [
[
path: { "${params.outdir}/preprocessing" },
mode: params.publish_dir_mode,
enabled: params.save_preprocessed_reads,
pattern: '*.{fastq,fq}.gz'
],
[
path: { "${params.outdir}/preprocessing" },
mode: params.publish_dir_mode,
pattern: '*.{log,json,txt,tsv,html,settings}'
],
]
}

// FastQC runs twice per library inside FASTQ_SHORTREADS_PREPROCESS_QC, once before and once
// after preprocessing, and at that point reads are still per-run. Without these prefixes all
// four sets of reports would collide on the same file names. Declared after the block above
// so that its publishDir wins.
withName: ".*:PRE_STATS:FASTQC" {
ext.prefix = { "${meta.id}_${meta.run_accession}_raw" }
publishDir = [
path: { "${params.outdir}/fastqc" },
mode: params.publish_dir_mode,
pattern: '*.{html,zip}'
]
}
withName: ".*:POST_STATS:FASTQC" {
ext.prefix = { "${meta.id}_${meta.run_accession}_processed" }
publishDir = [
path: { "${params.outdir}/fastqc" },
mode: params.publish_dir_mode,
pattern: '*.{html,zip}'
]
}
}
15 changes: 10 additions & 5 deletions docs/output.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The directories listed below will be created in the results directory after the

The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps:

- [FastQC](#fastqc) - Raw read QC and preprocessing
- [Read preprocessing](#read-preprocessing) - FastQC, adapter trimming, complexity filtering and host removal
- [HUMAnN v3 / v4](#humann-v3--v4) - Functional profiling via MetaPhlAn + HUMAnN
- [FMH FunProfiler](#fmh-funprofiler) - Sketch-based functional profiling
- [mifaser](#mifaser) - Read-level functional profiling
Expand All @@ -24,13 +24,18 @@ The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes d
Other than FastQC, MultiQC and pipeline information, all other steps (the profilers) are off by default, and must be switched on manually.
:::

### FastQC
### Read preprocessing

[FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content, adapter contamination, and overrepresented sequences. For further reading and documentation see the [FastQC help pages](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/).
Preprocessing runs through the nf-core [`fastq_shortreads_preprocess_qc`](https://nf-co.re/subworkflows/fastq_shortreads_preprocess_qc/) subworkflow, per sequencing run and before runs of the same sample are merged. Only FastQC is on by default; the steps that modify reads are switched on individually, see [read preprocessing in the usage docs](usage.md#read-preprocessing).

[FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content, adapter contamination, and overrepresented sequences. For further reading and documentation see the [FastQC help pages](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/). It runs twice, once before and once after preprocessing.

- `fastqc/`
- `*_fastqc.html`: FastQC report containing quality metrics for each sample.
- `*_fastqc.zip`: Zip archive containing the FastQC report and data files.
- `*_raw_fastqc.{html,zip}`: FastQC report for the reads as they arrived.
- `*_processed_fastqc.{html,zip}`: FastQC report for the reads that go on to the profilers.
- `preprocessing/`
- `*.{log,json,settings,txt}`: Logs and reports from the trimming, complexity filtering, deduplication and host removal steps that ran. Always published, and picked up by MultiQC.
- `*.fastq.gz`: The intermediate FASTQ files themselves, only with `--save_preprocessed_reads`.

### HUMAnN v3 / v4

Expand Down
46 changes: 46 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,38 @@ The pipeline will only run the profilers you explicitly turn on, and for which a
Each `--run_` flag requires a matching database entry in the `--databases` CSV. Database rows for tools that are not enabled will be ignored.
:::

## Read preprocessing

Preprocessing is handled by the nf-core [`fastq_shortreads_preprocess_qc`](https://nf-co.re/subworkflows/fastq_shortreads_preprocess_qc/) subworkflow. It runs per sequencing run, before runs of the same sample are merged, so each run is trimmed against its own adapters and quality profile.

Only read QC is on by default: FastQC runs and reports into MultiQC, and the reads reach the profilers untouched. Every step that modifies reads is opt-in.

FastQC runs twice per run, once on the raw reads and once after preprocessing, so the reports are suffixed `_raw` and `_processed`. With no preprocessing steps enabled the two are identical by construction.

| Flag | Default | Effect |
| -------------------------------------- | --------- | ------------------------------------------------------------------------------------------------ |
| `--skip_preprocessing_qc` | `false` | Skip FastQC. Nothing else in this table depends on it |
| `--perform_shortread_qc` | `false` | Turn on adapter trimming and quality filtering |
| `--shortread_qc_tool` | `fastp` | One of `fastp`, `adapterremoval`, `trimmomatic`, `cutadapt`, `trimgalore`, `bbduk`, `leehom` |
| `--shortread_qc_skipadaptertrim` | `false` | Keep the quality filtering but skip adapter removal |
| `--shortread_qc_adapterlist` | `null` | Custom adapter file. FASTA for `bbduk`/`fastp`, plain text for `adapterremoval` |
| `--shortread_qc_mergepairs` | `false` | Emit merged read pairs instead of the trimmed pairs. `fastp` and `adapterremoval` only |
| `--shortread_qc_savetrimmedfail` | `false` | Also keep the reads that failed the `fastp` filters |
| `--shortread_qc_dedup` | `false` | Deduplicate with BBMap `clumpify` |
| `--perform_shortread_complexityfilter` | `false` | Turn on low-complexity filtering |
| `--shortread_complexityfilter_tool` | `bbduk` | One of `bbduk`, `prinseqplusplus`, `fastp` |
| `--perform_shortread_hostremoval` | `false` | Turn on host decontamination |
| `--shortread_hostremoval_tool` | `hostile` | Either `hostile` or `deacon` |
| `--shortread_hostremoval_reference` | `null` | Host genome FASTA to build an index from |
| `--shortread_hostremoval_index` | `null` | Pre-built index directory, used instead of building one from the reference |
| `--shortread_hostremoval_index_name` | `null` | Name of the pre-built index |
| `--save_preprocessed_reads` | `false` | Publish the intermediate FASTQs to `<outdir>/preprocessing`. Logs and reports publish regardless |

Tool-specific arguments (minimum read length, entropy thresholds and similar) are set through `ext.args` in `conf/modules.config` rather than as pipeline parameters. See [Custom Tool Arguments](#custom-tool-arguments).

> [!NOTE]
> The pipeline still does no read preprocessing by default, so reads are expected to arrive already cleaned unless you enable the steps above.

## Databases input

```bash
Expand All @@ -78,6 +110,20 @@ Use the `db_name` column to record the database release or version used for the
| `db_params` | No | Additional parameters to pass to the profiler (no quotes allowed). |
| `db_path` | Yes | Absolute path to the database file or directory. Gzipped TAR archives (`.tar.gz`) are automatically decompressed. |

### Database versions and compatibility

The pipeline passes `db_path` straight to the profiler and never checks it against the tool version, so pairing a database with a compatible tool release is up to you. The versions a run actually used are recorded in `<outdir>/pipeline_info/nf_core_funcprofiler_software_mqc_versions.yml`.

These combinations were exercised on a human gut metagenome cohort during development; RGI, DIAMOND and HUMAnN v4 were only tested against the small CI databases.

| Tool | Tool version | Database |
| --------------- | ------------ | ----------------------------------------------------------------------------------------------------------------- |
| HUMAnN v3 | 3.6.1 | ChocoPhlAn full `v201901_v31`, UniRef90 `201901b` full, `utility_mapping` full |
| MetaPhlAn | 4.0.6 | `mpa_vJan21_CHOCOPhlAnSGB_202103` |
| FMH FunProfiler | 1.1.1 | KO sketches from [Zenodo record 10045253](https://zenodo.org/records/10045253), scaled 1000 / k=11 and scaled 500 |
| mi-faser | 1.64 | `GS-24-all`, shipped inside the mi-faser biocontainer rather than downloaded separately |
| eggNOG-mapper | 2.1.13 | eggNOG 5.0.2 data directory |

### HUMANn databases

HUMANn requires four database components per named database, each as a separate row with the same `db_name`. The example below uses a HUMANn v3-compatible UniRef90 database set; replace `uniref90_v3` with the exact release or version used in your analysis.
Expand Down
Loading
Loading