Every block codec except lzma is samtools/htscodecs compiled to a single WebAssembly module. gzip goes through libdeflate and bzip2 through the emscripten bzip2 port. The module is inlined in the bundle and loads itself, so there is nothing to set up.
lzma blocks go through a second wasm module. htscodecs has no lzma codec
(htslib links liblzma for those blocks), so src/xz-decompress/ contains
xz-embedded as a separate 16 KB base64
module, instantiated from a data: URI. Only the streaming wrapper around it is
JS.
dataflow.md shows where both sit in a query.
All CRAM v3 and v3.1 codecs are supported. Golomb and Golomb-Rice, which only v2 files use, are not (see data-series codecs).
| ID | Method | Supported |
|---|---|---|
| 0 | raw | ✅ |
| 1 | gzip | ✅ |
| 2 | bzip2 | ✅ |
| 3 | lzma | ✅ |
| 4 | rans | ✅ |
| 5 | rans4x16 | ✅ all sub-variants (order-0/1, Pack, RLE, r32x16, striped, CAT, gzip-min) |
| 6 | arith | ✅ |
| 7 | fqzcomp | ✅ |
| 8 | tok3 | ✅ all sub-variants (tok3-rans, tok3-arith) |
| ID | Codec | Supported |
|---|---|---|
| 1 | External | ✅ |
| 2 | Golomb | ❌ CRAM v2-era, not emitted by modern tools |
| 3 | Huffman | ✅ |
| 4 | ByteArrayLength | ✅ |
| 5 | ByteArrayStop | ✅ |
| 6 | Beta | ✅ |
| 7 | SubExp | ✅ |
| 8 | Golomb-Rice | ❌ CRAM v2-era, not emitted by modern tools |
| 9 | Gamma | ✅ |
Between them, fqzcomp, tok3 and the adaptive arithmetic coder are thousands of lines of context-modeling C, and porting them by hand means keeping a second copy correct indefinitely. Compiling the real thing got all of it in one step, including the rANS 4x16 and tok3 sub-variants that tools emit.
Because cram-js runs the same C that samtools runs, a block decodes to the same
bytes that samtools view gets, which answers whether it decoded correctly. To
pick up an upstream htscodecs fix, rerun the update script.
Speed follows from compiling the real C too. Most of the time spent decoding a CRAM read goes to quality scores and read names, and compiled code now decodes both in place of an interpreted inner loop. gzip uses libdeflate.
Because the wasm is inlined, it behaves like any other JS dependency in node, browsers and workers. It needs no second request, no static asset copied into your build, and no MIME type or CSP rules.
Compiling the C is the easy half of the decision. The hard half is choosing where the JS/wasm boundary sits, because every crossing copies its input into the wasm heap and its output back out. Each of the four choices below names the alternative it was picked over:
-
The unit is one block, not one record. A block's two copies amortize over the thousands of records encoded in it. Crossing per record would multiply the copy count by roughly the record count, tens of thousands per slice, and would put a crossing inside the hot loop.
-
Everything above the block stays in JS, meaning the per-data-series codecs (external, huffman, beta, gamma, subexp, byteArray*) and the record decode itself. Those codecs are small bit readers over bytes that are already decompressed and already in the JS heap, so moving them into wasm gives no algorithmic win and costs a crossing per series per slice. Decoding whole slices in wasm would go further still, and would move the
fetchReferenceSequencecallback and the columnar output across the boundary too. -
Decoders only. The build leaves out the compressors and the SIMD-specialized htscodecs variants, because this library only reads. That is most of why the binary is 113 KB rather than the several hundred a full build would be.
-
One lazily-instantiated instance per JS context, for the life of the process. Instantiation takes ~5 ms and runs on the first compressed block, not at import, so a consumer that opens a file and never queries it skips the cost. It runs once however many CRAMs are opened. In a worker it runs during the pool's init handshake instead (
warmupWasm), so the first slice a user waits on does not also compile the module.
- The heap only grows (see Memory), so peak tracks the largest single block ever decompressed.
- The worker bundle contains a second copy of the decoder and the wasm, so
src/wasm/cram-worker-source.jsis 400 KB (96 KB gzipped) in the published package. A bundler splits it into a chunk that loads only when a pool starts; see workers.md. - No SIMD, no wasm threads, which caps single-block throughput; the pool gets its parallelism from slices instead (see Threading).
src/wasm/htscodecs.js |
128 KB (55 KB gzipped) |
| wasm binary inside it | 113 KB |
| instantiation | ~5 ms, once |
| wasm heap | 16 MB floor |
src/xz-decompress/wasm.ts |
16 KB, lzma only |
Only the lzma module is base64. src/xz-decompress/wasm.ts is a
data:application/wasm;base64, URI, but the htscodecs bundle is not.
Emscripten's SINGLE_FILE=1 writes the binary as a string of one character per
byte, unpacked by the b[c] = ~f >> 8 & f loop at the top of htscodecs.js, so
113 KB of wasm fits in a 128 KB file where base64 would have needed 151 KB. Keep
that in mind before trying to "fix" the encoding, or measuring the wrong thing.
The one-character-per-byte string is a property of emscripten's output, not a
better encoding that the lzma module lacks, and the lzma module stays base64 on
purpose. We measured converting it: the source drops from 16,810 to 13,730
bytes, but terser re-escapes the control bytes when it bundles, so the saving in
dist/ is 634 bytes, or 1.7 KB gzipped. The size ratio does not explain the
different encodings either, since one character per byte costs 1.09x here
against base64's 1.33x, the same as for htscodecs.
The encodings differ because the two files are different kinds of artifact:
htscodecs.jsis build output.pnpm build:wasmregenerates it,.prettierignoreskips it, and nobody opens it.wasm.tswas vendored once in e8e140c, has no generator, and goes through lint, format and typecheck. Emscripten's scheme writes control bytes into the source unescaped, 6,600 of them at this size. They would make git treat the file as binary, and a line-ending normalization could then corrupt it with no warning. The corruption would show up as aninstantiatefailure, with no generator to rebuild the file from. Base64 is inert ASCII, so it suits a hand-maintained blob.
Because the first instantiation is async, every decoder entry point in
src/htscodecs-wasm.ts is async too; once it has happened, calls resolve
against an instance that already exists.
The heap starts at 16 MB and grows on demand, up to a 2 GB cap. A wasm heap can only ever grow, since memory never goes back to the OS, so peak usage climbs to the largest single block you decompress and stays there. Ordinary CRAM slices never push it off the 16 MB floor.
Every allocation is freed before the decode returns, so the heap holds no allocations between calls.
On a large query, the decoded records on the JS heap outweigh the wasm heap anyway; see memory.md.
Decoding is single-threaded scalar code, with no wasm threads and no SIMD, so
one block decode uses one core. Slices are independent, though, so the pool in
workers.md adds the parallelism one level up by decoding them
concurrently, one wasm instance per worker. The pool transfers slices rather
than sharing them, so the library needs no SharedArrayBuffer and therefore no
cross-origin isolation.
To update htscodecs, run htscodecs-wasm/update-htscodecs.sh and then
htscodecs-wasm/build.sh, which has the emcc flags the numbers above come from.
The build writes the inlined bundle to src/wasm/htscodecs.js.