Important

This project is under the nercone-labs GitHub organization.

nercone-labs is a GitHub organization created by nercone to store source code that is in the planning stages, unstable, uncertain in its continuation, or unreliable, and this project is no exception.

rfcex-rs

Really Fast Compression and EXtract — a Rust library and CLI for compressing and extracting many archive formats, with CPU and GPU code paths.

Status: early scaffold. Library traits, format dispatcher, CPU codecs, and CLI shell are wired up; GPU kernels and several less-common formats are stubbed and tracked in the per-crate TODOs.

Supported formats

Format	Read	Write	Notes
zip	✓	✓	deflate / bzip2 / zstd / lzma / store, AES
tar	✓	✓	uncompressed POSIX tar
tar.gz	✓	✓	tar over gzip
tar.xz	✓	✓	tar over xz
tar.zst	✓	✓	tar over zstd
tar.bz2	✓	✓	tar over bzip2
7z	—	—	scaffolded; reader/writer land in 0.2
cab	—	—	scaffolded
rar	—	✗	extract-only by license; scaffolded
lzh	—	✗	extract-only; scaffolded
gz / bz2 / xz / zst / br / lz4 / sz	✓	✓	mature CPU paths via well-known crates.
lzfse	✓	✓	via lzfse (reference C library). Buffered, rfcex framing header.
lzo	✓	✓	via minilzo-rs — GPL-3.0+, opt-in feature lzo. Buffered.
zpaq	✓	✓	via zpaq_rs. Buffered (full archive in memory).
lzham	✓	✓	via lzham (CMake-built). Opt-in feature lzham. AArch64 hosts require running cargo xtask patch-vendor once after cargo fetch. Buffered.

Supported hardware

• CPU: x86 / amd64 / AArch64 / RISC-V 32+64. Pure-Rust code where possible; libdeflate / libbz2 / liblzma / libzstd / brotli via well-known crates. Block-level parallelism via rayon (pigz / pbzip2 style).
• GPU: CUDA (NVIDIA), Metal (Apple), ROCm/HIP (AMD). GPU support is layered behind opt-in features. The backend crates compile as no-op stubs unless their runtime feature is enabled and a vendor toolkit is available at build time.

Library design

rfcex-rs is a library first; the rfcex CLI is a thin wrapper. Three API tiers are offered:

Tier	Where	Example
High-level	rfcex::{compress_file, compress_dir, extract, list}	one-call helpers
Mid-level	rfcex::Archive builder	configurable streaming
Low-level	rfcex_codec, rfcex_container, rfcex_gpu_core	direct trait objects

A C ABI lives in the rfcex-ffi crate (cdylib / staticlib); the ffi/include/rfcex.h header is regenerated by cargo build -p rfcex-ffi.

Workspace layout

rfcex-rs/
├── crates/
│   ├── rfcex                       — facade
│   ├── rfcex-core                  — types, traits, errors
│   ├── rfcex-detect                — magic-bytes detection
│   ├── rfcex-io                    — mmap / counting readers
│   ├── rfcex-parallel              — block-level rayon scheduler
│   ├── rfcex-codec                 — dispatcher
│   ├── rfcex-codec-{deflate,bzip2,xz,zstd,brotli,lz4,snappy,lzfse,lzo,lzham,zpaq}
│   ├── rfcex-container             — zip / tar / 7z / cab / rar / lzh
│   ├── rfcex-gpu-core              — backend-agnostic GPU abstraction
│   ├── rfcex-gpu-{cuda,metal,rocm} — backends
│   ├── rfcex-gpu-kernels           — hand-written CUDA / MSL / HIP kernels
│   ├── rfcex-cli                   — CLI binary (`rfcex`)
│   └── rfcex-bench                 — criterion benches
├── ffi/                            — C ABI
└── xtask/                          — workspace task runner

Building

# Default build: CPU-only, common codecs.
cargo build --workspace --release

# Everything (requires CUDA toolkit / Metal toolchain installed):
cargo build --workspace --release --features "rfcex/full"

# Minimal — only zip + gzip:
cargo build -p rfcex --no-default-features --features "zip,gz"

# Opt into the codecs that require extra toolchains / different licenses:
cargo fetch
cargo xtask patch-vendor                  # one-time AArch64 fixup for lzham
CMAKE_POLICY_VERSION_MINIMUM=3.5 \
  cargo build --release --features "rfcex/lzo,rfcex/lzham"

Build prerequisites for opt-in codecs

Feature flag	Extra requirements
rfcex/lzo	LZO/minilzo is GPL-3.0+; binaries linking it inherit the GPL.
rfcex/lzham	CMake + a working C/C++ toolchain. On AArch64 hosts run cargo xtask patch-vendor once to fix two x86-only inline-asm fragments in upstream lzham_codec. CMake ≥ 4 needs CMAKE_POLICY_VERSION_MINIMUM=3.5 because lzham_codec's CMakeLists.txt predates the 3.5 floor.

Quick CLI tour

rfcex c out.tar.zst src/                # create
rfcex x archive.7z out/                 # extract
rfcex l archive.zip                     # list entries
rfcex t archive.tar.gz                  # integrity test
rfcex a archive.zip new_file.txt        # append to existing archive (zip + tar)
rfcex i archive.zip                     # aggregate archive metadata
rfcex bench corpus/silesia.tar          # compare backends
rfcex info                              # detected hardware

Licence

Dual-licensed under MIT and Apache-2.0; see LICENSE-MIT and LICENSE-APACHE. Third-party notices are collected in LICENSE-THIRD-PARTY.