Operating System
Ubuntu 22.04
Other Linux
No response
Workflow Version
v1.5.2
Workflow Execution
EPI2ME Desktop (Local)
Other workflow execution
No response
EPI2ME Version
No response
CLI command run
No response
Workflow Execution - CLI Execution Profile
None
What happened?
I am re-basecalling a mix of old and new POD5s with the latest models and ran into an error partway through the alignment part of the workflow.
A quick web search of the first error message (Inconsistency detected by ld.so: ../elf/dl-tls.c: 481: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!) comes up with this race condition bug in glibc: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1863162
Could it be related to having a large number of files with small execution time per file? Our reads are from cell-free DNA so they are comparatively short. The old data was obtained before we optimised settings affecting the number of reads per file, so the workflow splits off into more than a thousand dorado basecaller calls that each take maybe a minute, with subsequent alignment steps being even faster to complete.
I have (manually) resumed the Nextflow run and it seems to be working so far. (As an aside, being able to resume failed runs in the GUI would be nice!)
Relevant log output
N E X T F L O W ~ version 23.04.2
Launching `/scratch/epi2me/workflows/epi2me-labs/wf-basecalling/main.nf` [cfDNA_batch_1-hac52-5mC_5hmC-6mA] DSL2 - revision: a7538c3219
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
�[0;92m|||||||||| �[0m�[2m_____ ____ ___ ____ __ __ _____
�[0;92m|||||||||| �[0m�[2m| ____| _ \_ _|___ \| \/ | ____|
�[0;33m||||| �[0m�[2m| _| | |_) | | __) | |\/| | _|
�[0;33m||||| �[0m�[2m| |___| __/| | / __/| | | | |__
�[0;94m|||||||||| �[0m�[2m|_____|_| |___|_____|_| |_|_____|
�[0;94m|||||||||| �[0m�[1mwf-basecalling v1.5.2�[0m
�[2m--------------------------------------------------------------------------------�[0m
�[1mCore Nextflow options�[0m
�[0;34mrunName : �[0;32mcfDNA_batch_1-hac52-5mC_5hmC-6mA�[0m
�[0;34mcontainerEngine: �[0;32mdocker�[0m
�[0;34mlaunchDir : �[0;32m/scratch/epi2me/instances/wf-basecalling_01JZB34XJJ25H5VNFMJNDH1CYY�[0m
�[0;34mworkDir : �[0;32m/scratch/epi2me/instances/wf-basecalling_01JZB34XJJ25H5VNFMJNDH1CYY/work�[0m
�[0;34mprojectDir : �[0;32m/scratch/epi2me/workflows/epi2me-labs/wf-basecalling�[0m
�[0;34muserName : �[0;32mikleijn�[0m
�[0;34mprofile : �[0;32mstandard�[0m
�[0;34mconfigFiles : �[0;32m/scratch/epi2me/workflows/epi2me-labs/wf-basecalling/nextflow.config�[0m
�[1mInput Options�[0m
�[0;34minput : �[0;32m/data/symlinks_minknow/cfDNA_batch_1_runs12�[0m
�[0;34mref : �[0;32m/scratch/GRCh38-no_alt.fasta�[0m
�[1mOutput Options�[0m
�[0;34mout_dir : �[0;32m/scratch/epi2me/instances/wf-basecalling_01JZB34XJJ25H5VNFMJNDH1CYY/output�[0m
�[0;34msample_name : �[0;32mcfDNA_batch_1�[0m
�[0;34mstore_dir : �[0;32m/scratch/epi2me/data�[0m
�[1mBasecalling options�[0m
�[0;34mbasecaller_cfg : �[0;32mdna_r10.4.1_e8.2_400bps_hac@v5.2.0�[0m
�[0;34mbarcode_kit : �[0;32mSQK-NBD114-24�[0m
�[1mAdvanced basecalling options�[0m
�[0;34mbasecaller_args: �[0;32m--modified-bases 5mC_5hmC 6mA�[0m
!! Only displaying parameters that differ from the pipeline defaults !!
�[2m--------------------------------------------------------------------------------�[0m
If you use epi2me-labs/wf-basecalling for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
�[2m--------------------------------------------------------------------------------�[0m
This is epi2me-labs/wf-basecalling v1.5.2.
�[2m--------------------------------------------------------------------------------�[0m
[25/afd250] Submitted process > prepare_reference:cram_cache (1)
[25/19bcf5] Submitted process > getVersions
[e2/71f90d] Submitted process > prepare_reference:make_mmi (1)
[8c/ca7a56] Submitted process > getParams
[e7/88a1cb] Submitted process > wf_dorado:dorado (2)
[82/f92143] Submitted process > wf_dorado:dorado (1)
[30/ed9a9c] Submitted process > wf_dorado:dorado (3)
[...]
[54/042dd8] Submitted process > wf_dorado:dorado (1130)
[e7/d59bf7] Submitted process > wf_dorado:dorado (1131)
[d1/f6eb69] Submitted process > wf_dorado:dorado (1132)
[77/41c001] Submitted process > wf_dorado:align_and_qsFilter (1)
[bf/a87e9a] Submitted process > wf_dorado:align_and_qsFilter (2)
[83/24288e] Submitted process > bamstats (3)
[82/a50877] Submitted process > progressive_stats (3)
[41/3a2039] Submitted process > makeReport (3)
[33/48b1ce] Submitted process > wf_dorado:align_and_qsFilter (3)
[95/e6a61a] Submitted process > bamstats (4)
[71/c7a66a] Submitted process > progressive_stats (4)
[69/b8526f] Submitted process > makeReport (4)
[fd/ecacc7] Submitted process > wf_dorado:align_and_qsFilter (4)
[eb/4df191] Submitted process > bamstats (5)
[ce/786b0f] Submitted process > wf_dorado:align_and_qsFilter (5)
[41/23745e] Submitted process > bamstats (6)
[f7/4d6fba] Submitted process > progressive_stats (5)
[3c/3b184c] Submitted process > makeReport (5)
[c4/8521ab] Submitted process > wf_dorado:align_and_qsFilter (6)
[c6/8eae6f] Submitted process > bamstats (7)
[ef/1d7e33] Submitted process > progressive_stats (6)
[25/912cc1] Submitted process > makeReport (6)
[0d/9d2d4d] Submitted process > wf_dorado:align_and_qsFilter (7)
[c5/1c85b1] Submitted process > bamstats (8)
[8a/57f908] Submitted process > progressive_stats (7)
[87/adb4b3] Submitted process > makeReport (7)
[39/6baab6] Submitted process > progressive_stats (8)
[c5/4b6a79] Submitted process > makeReport (8)
[e2/4aad3a] Submitted process > wf_dorado:align_and_qsFilter (8)
[36/09e425] Submitted process > bamstats (9)
[87/1781dc] Submitted process > progressive_stats (9)
[41/29af94] Submitted process > makeReport (9)
[05/361861] Submitted process > wf_dorado:align_and_qsFilter (9)
[04/41a592] Submitted process > bamstats (10)
[07/defd89] Submitted process > progressive_stats (10)
[9d/27529e] Submitted process > makeReport (10)
[b1/774cb8] Submitted process > wf_dorado:align_and_qsFilter (10)
[3d/ba97ad] Submitted process > bamstats (11)
[51/49300a] Submitted process > progressive_stats (11)
[dd/faf421] Submitted process > makeReport (11)
[cf/1fe7c2] Submitted process > wf_dorado:align_and_qsFilter (11)
[db/8cc2d3] Submitted process > bamstats (12)
[82/5dbf96] Submitted process > progressive_stats (12)
[71/90c6e7] Submitted process > makeReport (12)
[e4/681fe7] Submitted process > wf_dorado:align_and_qsFilter (12)
[1a/3902fd] Submitted process > bamstats (13)
[9a/b584cb] Submitted process > progressive_stats (13)
[38/d87386] Submitted process > makeReport (13)
[66/ffc5a2] Submitted process > wf_dorado:align_and_qsFilter (13)
ERROR ~ Error executing process > 'wf_dorado:align_and_qsFilter (12)'
Caused by:
Process `wf_dorado:align_and_qsFilter (12)` terminated with an error exit status (1)
Command executed:
samtools view -H --no-PG 11.ubam > reads.header
samtools reset -x tp,cm,s1,s2,NM,MD,AS,SA,ms,nn,ts,cg,cs,dv,de,rl --no-PG 11.ubam -o - | samtools fastq -T 1 -@ 1 - | minimap2 -y -t 8 -a -x lr:hq --cap-kalloc 100m --cap-sw-mem 50m ref.mmi - | workflow-glue reheader_samstream reads.header --insert $'@PG\tID:reset\tPN:samtools\tCL:samtools reset -x tp,cm,s1,s2,NM,MD,AS,SA,ms,nn,ts,cg,cs,dv,de,rl' --insert $'@PG\tID:fastq\tPN:samtools\tCL:samtools fastq -T 1' | samtools sort -@ 3 | samtools view -e '[qs] >= 10' --output 11.pass.cram --unoutput 11.fail.cram -O CRAM --reference GRCh38-no_alt.fasta -
Command exit status:
1
Command output:
(empty)
Command error:
[18:42:24 - workflow_glue] Bootstrapping CLI.
[18:42:24 - workflow_glue] Starting entrypoint.
[M::main::10.231*1.00] loaded/built the index for 195 target sequence(s)
[M::mm_mapopt_update::13.285*1.00] mid_occ = 136
[M::mm_idx_stat] kmer size: 19; skip: 19; is_hpc: 0; #seq: 195
[M::mm_idx_stat::15.478*1.00] distinct minimizers: 214664461 (92.23% are singletons); average occurrences: 1.372; average spacing: 10.529; total length: 3099922541
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 100131 reads
Inconsistency detected by ld.so: ../elf/dl-tls.c: 481: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!
[W::sam_hdr_create] Ignored @SQ SN:chrUn_KI : bad or missing LN tag
[E::sam_hrecs_update_hashes] Header includes @SQ line "chrUn_KI" with no LN: tag
[E::sam_hrecs_update_hashes] Header includes @SQ line "chrUn_KI" with no LN: tag
samtools sort: failed to change sort order header to 'SO:coordinate'
[main_samview] fail to read the header from "-".
Work dir:
/scratch/epi2me/instances/wf-basecalling_01JZB34XJJ25H5VNFMJNDH1CYY/work/e4/681fe719914bee86d8da498c2c975f
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
-- Check '/scratch/epi2me/instances/wf-basecalling_01JZB34XJJ25H5VNFMJNDH1CYY/nextflow.log' file for details
WARN: Killing running tasks (2)
Application activity log entry
Were you able to successfully run the latest version of the workflow with the demo data?
yes
Other demo data information
Operating System
Ubuntu 22.04
Other Linux
No response
Workflow Version
v1.5.2
Workflow Execution
EPI2ME Desktop (Local)
Other workflow execution
No response
EPI2ME Version
No response
CLI command run
No response
Workflow Execution - CLI Execution Profile
None
What happened?
I am re-basecalling a mix of old and new POD5s with the latest models and ran into an error partway through the alignment part of the workflow.
A quick web search of the first error message (
Inconsistency detected by ld.so: ../elf/dl-tls.c: 481: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!) comes up with this race condition bug in glibc: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1863162Could it be related to having a large number of files with small execution time per file? Our reads are from cell-free DNA so they are comparatively short. The old data was obtained before we optimised settings affecting the number of reads per file, so the workflow splits off into more than a thousand dorado basecaller calls that each take maybe a minute, with subsequent alignment steps being even faster to complete.
I have (manually) resumed the Nextflow run and it seems to be working so far. (As an aside, being able to resume failed runs in the GUI would be nice!)
Relevant log output
Application activity log entry
Were you able to successfully run the latest version of the workflow with the demo data?
yes
Other demo data information