Batching assemblies throughout workflow

[claudiacarter]

• Updated main.nf and SORF_EXTRA process to run through to completion with batching changes
• Updated container for bakta runs to include changes in create a batch pickle interface for the pseudo bulk mode bakta#5 (minus Dockerfile changes)
• Updated tests/features_test.py to take a file mapping reference GFF3 paths to matched test GFF3 paths

These changes are tested and validated on a small test set of assemblies.

[claudiacarter]

With buffer_size=4, 2 runs one with bundled output gff3s one unbundled:

bsub.py -q oversubscribed 4 batched_buffer_4 nextflow run /data/pam/team230/cc52/scratch/repositories/pannotator/main.nf --bakta_db /data/pam/software/bakta/v6.0_amrfinder_2026_01_21.1 --bakta_db_type full --indir assemblies --outdir buffer_4_results --buffer_size 4 --bundle_gff3 false -profile sanger_lsf
bsub.py -q oversubscribed 4 batched_buffer_4 nextflow run /data/pam/team230/cc52/scratch/repositories/pannotator/main.nf --bakta_db /data/pam/software/bakta/v6.0_amrfinder_2026_01_21.1 --bakta_db_type full --indir assemblies --outdir buffer_4_results --buffer_size 4 -profile sanger_lsf -resume

$ tree buffer_4_results/
buffer_4_results/
├── final_annotations
│   ├── batch_0.gff3.tar.gz
│   ├── batch_1.gff3.tar.gz
│   ├── DRR215911_contigs.gff3
│   ├── DRR217194_contigs.gff3
│   ├── DRR217197_contigs.gff3
│   ├── DRR235745_contigs.gff3
│   ├── DRR317403_contigs.gff3
│   ├── DRR317423_contigs.gff3
│   ├── DRR317550_contigs.gff3
│   └── DRR317576_contigs.gff3
└── pipeline_info
    ├── execution_report_2026-09-01_14-28-23.html
    ├── execution_report_2026-09-01_15-15-28.html
    ├── execution_timeline_2026-09-01_14-28-23.html
    ├── execution_timeline_2026-09-01_15-15-28.html
    ├── execution_trace_2026-09-01_14-28-23.txt
    ├── execution_trace_2026-09-01_15-15-28.txt
    ├── pipeline_dag_2026-09-01_14-28-23.svg
    └── pipeline_dag_2026-09-01_15-15-28.svg
$ tar -tf buffer_4_results/final_annotations/batch_0.gff3.tar.gz
 
DRR317423_contigs.gff3
DRR317576_contigs.gff3
DRR217197_contigs.gff3
DRR215911_contigs.gff3

With buffer_size=1:

bsub.py -q oversubscribed 4 batched_buffer_1 nextflow run /data/pam/team230/cc52/scratch/repositories/pannotator/main.nf --bakta_db /data/pam/software/bakta/v6.0_amrfinder_2026_01_21.1 --bakta_db_type full --indir assemblies --outdir buffer_1_results --buffer_size 1 --bundle_gff3 false -profile sanger_lsf -resume

$ tree buffer_1_results/
buffer_1_results/
├── final_annotations
│   ├── DRR215911_contigs.gff3
│   ├── DRR217194_contigs.gff3
│   ├── DRR217197_contigs.gff3
│   ├── DRR235745_contigs.gff3
│   ├── DRR317403_contigs.gff3
│   ├── DRR317423_contigs.gff3
│   ├── DRR317550_contigs.gff3
│   └── DRR317576_contigs.gff3
└── pipeline_info
    ├── execution_report_2026-09-01_14-23-10.html
    ├── execution_timeline_2026-09-01_14-23-10.html
    ├── execution_trace_2026-09-01_14-23-10.txt
    └── pipeline_dag_2026-09-01_14-23-10.svg

Tested with new feature_tests.py against results from a run before any changes on the Sanger farm installation of Pannotator (pannotator/v0.1.3-61680a55) as a reference:
buffer_size 1:

================================================================================
SUMMARY
================================================================================
Assembly                                 Status  
DRR215911_contigs.gff3                   OK      
DRR217194_contigs.gff3                   OK      
DRR217197_contigs.gff3                   OK      
DRR235745_contigs.gff3                   OK      
DRR317403_contigs.gff3                   OK      
DRR317423_contigs.gff3                   OK      
DRR317550_contigs.gff3                   OK      
DRR317576_contigs.gff3                   OK      

8/8 annotations identical.

buffer_size 4:

================================================================================
SUMMARY
================================================================================
Assembly                                 Status  
DRR215911_contigs.gff3                   OK      
DRR217194_contigs.gff3                   OK      
DRR217197_contigs.gff3                   OK      
DRR235745_contigs.gff3                   OK      
DRR317403_contigs.gff3                   OK      
DRR317423_contigs.gff3                   OK      
DRR317550_contigs.gff3                   OK      
DRR317576_contigs.gff3                   OK      

8/8 annotations identical.

Also checked with scratch=true:

bsub.py -q oversubscribed 4 batched_scratch nextflow run /data
/pam/team230/cc52/scratch/repositories/pannotator/main.nf --bakta_db /data/pam/software/bakta/v6.0_amrfinder_2026_01_21.1 --bakta_db_type full --indir ass
emblies --outdir buffer_8_scratch_results --buffer_size 8 --scratch true -profile sanger_lsf

$ ls -lah buffer_8_scratch_results/final_annotations/
total 16M
drwxr-sr-x 2 cc52 team230f 4.0K Sep  1 16:05 .
drwxr-sr-x 4 cc52 team230f 4.0K Sep  1 16:05 ..
-rw-r--r-- 1 cc52 team230f  16M Sep  1 16:05 batch_0.gff3.tar.gz
$ tar -tf buffer_8_scratch_results/final_annotations/batch_0.gff3.tar.gz 
DRR317423_contigs.gff3
DRR317576_contigs.gff3
DRR217197_contigs.gff3
DRR215911_contigs.gff3
DRR217194_contigs.gff3
DRR235745_contigs.gff3
DRR317550_contigs.gff3
DRR317403_contigs.gff3

[flass]

looks good!

[flass]

ideally we would not have to unbatch the pickle files here, and simply feed the batch pickle to Bakta, we avoid file count inflation. but that can be implemented later.
at least the scratch directive avoid retention of all those files.