perf(predict_t_rnas): chunk the input across tRNAscan-SE processes #289

[piotrkica]

Following #289 and predict_t_rnas performance.

Currently bakta passes cfg.threads to tRNAscan-SE and for metagenomes it is inefficient and reason behind #289 reported performance. Running on single core is much faster for this type of input.

As author in #289 proposed we can split the input into chunks and run one tRNAscan-SE process per chunk with --thread 0, so the parallelism comes from the processes. I explored some other possibilities, but this is the best approach currently. This is necessary to run on assemblies from e.g. 2GB FASTQ.gz efficiently.

"Why not run at --threads 1? Because at --cpu 0 cmsearch takes its serial path and never builds the thread pool or the work queue. At --cpu 1 it builds both to run a single worker, and that setup is paid on every invocation - two per contig. The difference is 0 to 8% for identical output" - AI summary, worth checking out but performance increase is real.

I adapted to current code style. Output is identical — chunks keep input order, are concatenated in order, and I compared the feature list against an unpatched run on many samples. Bakta tests pass. On a single-contig no_chunks = 1 so here logic is the same. For 2-3 contigs you could use hybrid approach e.g. 3 chunks each with 5 cores but i went with simpler approach and the difference is not that meaningful. With more contigs chunking wins.
Peak RSS is not meaningfully higher.

I measured the fix on multiple samples of different size and type.

sample	contigs	--thread 15 (current)	--thread 0	--thread 1	chunked, 15 cores (proposed)	vs current	vs --thread 0	efficiency* (prop)	cpu_efficiency** (prop)	peak_RSS (thr0 vs prop)
ERR4333936	95,607	4,384.0 s	1,785.3 s	-	137.2 s	31.95x	13.01x	87%	86%	2,957 MB -> 3,258 MB
ERR4341823	210,201	-	3,145.6 s	-	233.9 s	-	13.45x	90%	81%	3,831 MB -> 4,189 MB
ERR3607267	252,200	-	3,510.5 s	3,694.8 s	256.2 s	-	13.70x	91%	88%	4,138 MB -> 4,563 MB
E. coli	1	7.7 s	19.5 s	19.6 s	7.7 s	1.00x	2.53x	17%	16%	212 MB -> 379 MB

* efficiency = (--thread 0 / chunked) / 15 cores
** cpu_efficiency = mean cores busy / 15 cores allocated, from cgroups

Tests were done on human gut metagenome samples assembled with megahit and bakta v1.12.1 and full DB v6.0. Measured on m8a.4xlarge vm with 15 threads. Worked on bakta reduced to relevant part (--skip-* flags).

Added split/merge functions to fasta.py file, r_rna, nc_rna, nc_rna-regions will use the same function and approach - i measured similar performance benefit but thats for another PR.

Don't mind me while on vacation :D, enjoy your time off.

[piotrkica]

Five out of 6 predictors (pre-CDS) benefit from chunking approach (pilercr does not). I have already tested performance on those and verified the output is unchanged. Holding off with making another PR with this until this is merged.

Predictor input-chunking on ERR4333936 (95,607 contigs), baseline vs fixed, allocated cpu = 16

predictor	tool	baseline	baseline cpu_eff	fixed	fixed cpu_eff	speedup
tRNA	tRNAscan-SE	4,698.5s	7%	133.6s	86%	35.2×
tmRNA	aragorn	41.4s	6%	2.7s	91%	15.3×
rRNA	cmscan	604.9s	9%	62.6s	59%	9.7×
ncRNA	cmscan	5,024.9s	18%	980.5s	94%	5.1×
ncRNA regions	cmscan	971.3s	23%	209.7s	97%	4.6×
RNA block		11,341s		1,389s		8.2×

I can add more experiment data in next PR.
You can check already these changes on this branch:
https://github.com/piotrkica/bakta/tree/perf/predictors-input-chunks