Rawdinal

Rawdinal is an experimental Rust library for decoding selected Sigma X3F files and Apple ProRAW DNG files. It provides a safe Rust API and a C-compatible static library. JPEG XL and lossy JPEG decoding use pinned Rust codec crates; X3F and lossless JPEG decoding are implemented in this repository.

Input / API	Output
X3F: X3f::decode / rawdinal_sensor_v1_decode	Three uncalibrated u16 physical sensor planes, in bottom/middle/top order, at their full encoded dimensions
Apple ProRAW DNG: Dng::decode / rawdinal_raw_v1_decode	Interleaved camera-native f32 samples after linearization, black subtraction and white normalization
Original sd Quattro X3F: experimental::render / rawdinal_decode	Experimental reconstructed, white-balanced linear-sRGB output

The sensor and ProRAW APIs are intended for applications that provide their own raw-processing pipeline. The experimental X3F renderer is a separate path; Rawdinal does not produce a finished display image from ProRAW.

Relationship to X3F Tools

Rawdinal builds and runs without X3F Tools. Its decoders are implemented in Rust in this repository; the library neither links to X3F Tools nor launches its executables. The default Cargo tests and CMake build also require no X3F Tools installation.

X3F Tools is used separately to generate the external X3F comparison fixtures described below. Once generated, those fixtures can be compared without the tool installed. The current comparison harness accepts only the pinned Kalpanika decoder revision and records its internal plane mapping. Agreement checks decompressed samples and CAMF bytes; it does not establish an independently derived format specification or validate calibration and reconstruction.

There is also implementation heritage to distinguish from build dependencies. The experimental renderer follows the X3F Tools matrix/gain and bad-pixel conventions, including the sd Quattro autofocus-pixel coordinates. THIRD_PARTY_NOTICES retains the license for X3F Tools portions. Rawdinal is therefore not presented as a clean-room implementation. Its sensor API exposes uncalibrated physical planes without applying that rendering convention.

Sigma X3F

Rawdinal decodes TRUE-compressed sensor layers, decompresses CAMF calibration metadata, and exposes the embedded JPEG preview and available TIFF/EXIF. The Rust API can also parse CAMF entries into text, properties and matrices. Sensor decoding leaves calibration, reconstruction, white balance and color conversion to the caller.

Supported sensor layouts are:

Observed cameras	X3F versions	RAW type / TRUE format	CAMF type	Encoded layer dimensions (bottom, middle, top)
DP1/DP2/DP3 Merrill, SD1 Merrill	2.3, 3.0, 3.1	1 / 0x1e	4	4928x3264 on all layers
dp0/dp2/dp3 Quattro	4.1	1 / 0x23	5	2944x1836, 2944x1836, 6272x3672
sd Quattro	4.2	1 / 0x25	5	2944x1888, 2944x1888, 5888x3776
sd Quattro H	4.2	1 / 0x27	5	3328x2240, 3328x2240, 6656x4480

Support is restricted to these observed version/format/geometry combinations, rather than every mode or firmware of each camera. SensorFormat::SdQuattro covers both sd Quattro formats; callers must use each decoded plane's dimensions. The experimental linear-sRGB renderer is limited to the original sd Quattro.

The original 164-file validation corpus (59 Merrill, 100 dp3 Quattro, and 5 sd Quattro files) and 11 additional supported files from raw.pixls.us agree byte-for-byte with fixtures generated by pinned Kalpanika X3F Tools for decompressed CAMF and every physical-plane sample; this is not universal ground truth. Sensor planes preserve their full encoded dimensions; the dp Quattro top layer is wider than its nominal image width, and crop interpretation remains future calibration or adapter work.

The raw.pixls.us corpus also contains six unsupported legacy files: DP1, DP1s, SD9, SD10, SD14 and SD15. Tests check that decoding rejects them. In particular, legacy RAW type 3 / TRUE 0x1e is distinct from Merrill RAW type 1 / TRUE 0x1e. Rawdinal is not a general Foveon converter and does not provide calibrated rendering for Merrill, dp Quattro or sd Quattro H cameras.

Metadata-only access is available for the observed SD9, SD10 and SD14 layouts: X3F 2.1/2.2, RAW type 3 / format 0x06, and CAMF type 2. X3f::parse retains legacy RAW sections (IMAG on the SD9/SD10 originals, IMA2 on SD14); camf_bytes() decrypts their payload and calibration() parses its entries. Both sensor_format() and decode() reject RAW type 3. Legacy DP1/DP1s/SD15 CAMF type 4 access is still unsupported.

Parsing a container does not imply sensor-decoding support. To extract supported CAMF metadata without decoding pixels, use the existing example:

cargo run --locked --example dump_camf -- INPUT.x3f OUTPUT.camf

The output must not already exist. CAMF-2 extraction returns exactly the decrypted section payload, with no synthetic trailing padding. It is checked against three hash-identified originals; these checks do not establish calibration semantics. The rawdinal-inspect command still attempts sensor decoding and therefore fails on these legacy files after inspecting their metadata.

For the original sd Quattro, the experimental renderer offers bilinear or guided reconstruction and unbounded linear-sRGB output. It omits display tone mapping and sharpening. Optional clipping masks record contributing samples at their estimated encoded maxima; these are not measurements of physical sensor saturation.

Apple ProRAW DNG

The ProRAW decoder accepts Apple three-channel LinearRaw images in classic TIFF, in either byte order, with DNG version declarations from 1.0 through 1.7.1. Support is layout-based rather than a promise to decode every ProRAW mode or iPhone model.

Storage	Implemented sample domain
Uncompressed	Unsigned 8–16-bit samples, including packed rows
Lossless JPEG, compression 7	SOF3, unsigned 8–16-bit samples
Lossy JPEG, compression 34892	8-bit output from the JPEG decoder
JPEG XL, compression 52546	Integer codestreams up to 16 bits; TIFF storage is 8 bits for codestream precision up to 8, otherwise 16 bits

Tiled and strip storage, chunky and planar channels, and row/column interleave are supported. JPEG XL uses the full 8/16-bit output range of the Adobe SDK/libjxl integer route, rather than assuming codestream precision equals TIFF precision. Codestream orientation is undone during assembly; TIFF orientation remains unapplied. Animation, extra JPEG XL channels, floating-point codestreams, and XYB with an ICC output profile are rejected.

Normalization applies an optional linearization table, subtracts the active-area-relative repeated black pattern and horizontal/vertical deltas, then divides by WhiteLevel - maximum_channel_black. Missing tables use identity; missing black levels use zero. Missing white levels use the TIFF integer maximum. Normalization preserves negative and above-white values. Mandatory opcode processing can introduce the clipping or quantization prescribed by that operation.

Opcode lists execute before linearization, after normalization in active-image coordinates, and at the final LinearRaw stage. Implemented operations are TrimBounds, MapTable, MapPolynomial, GainMap, DeltaPerRow/Column and ScalePerRow/Column (IDs 6–13). Unsupported optional operations are skipped and reported; unsupported mandatory operations fail. Stage 1 uses logical uint16 code units, including for 8-bit inputs; subsequent stages use normalized float values with no retained black pedestal. Nondefault sub-tile blocks, CFA/mosaic DNG, BigTIFF and unsupported predictors remain outside this decoder.

Dng::decode_with_options returns DngDecodedImage, containing the existing LinearRawImage descriptor plus scoped, owned TIFF/EXIF/GPS fields. Calibration matrices, white balance, custom illuminants, noise metadata, profile curves and opaque private fields remain accessible through their original IFDs and tag numbers. metadata.calibration() provides numeric access to the main white-balance and three-illuminant matrix fields. Private embedded offsets and external/extra profile references can still require the original file.

The extended result reports fractional crop values and valid_area. Pixels outside that region have unspecified processing contents and must be excluded from downstream processing. The legacy Rust Dng::decode keeps its original return type and rejects fractional crop or opcode trimming that its descriptor cannot express. In C, the original descriptor layout is unchanged; use the new metadata getters for exact geometry and valid bounds.

Optional decode flags retain assembled pre-opcode codes, sample-domain clipping evidence, and decoded semantic masks. These flags do not measure photosite saturation. Masks are separate 8-bit weight images with their own dimensions, names and canvas mapping. Dng::jpeg_preview borrows an embedded JPEG preview. ProfileGainTable::parse interprets gain-table versions 1 and 2; its explicit evaluator requires already-converted linear RIMM/ProPhoto RGB. White balance, color conversion, baseline exposure and the preferred Apple appearance remain host responsibilities.

All 21 available originals decode. An original's 36,578,304 assembled samples also match the independent TIFF/SOF3 reference and the separate normalization formula reference, with zero float32 ULP difference. Synthetic tests cover JPEG XL, lossy JPEG, reduced precision and a 48 MP full-image decode. Apple Core Image/Adobe SDK color-reference comparisons and qualification with actual 48 MP/JPEG XL originals remain open.

Build and test

Install Rust 1.85 or newer with Cargo. The C library build also requires CMake 3.21 or newer and a C11 compiler. Build definitions cover native Linux, macOS (Intel or Apple Silicon) and Windows (MSVC). Windows and macOS still need validation on native hosts; the checks performed so far are on Linux. CMake rejects cross-compilation and macOS universal builds.

Rust workspace

Run from the repository root:

cargo build --workspace --release --locked
cargo test --workspace --locked

The rawdinal package provides the Rust library and the X3F-only rawdinal-inspect command. rawdinal-ffi produces librawdinal_ffi.a on Linux/macOS or rawdinal_ffi.lib with Windows MSVC. Its public C header is include/rawdinal.h.

Linux

Install a C compiler, CMake and Make or Ninja, then run:

cargo fetch --locked
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure

macOS

Install the Xcode Command Line Tools (xcode-select --install), CMake, and a native Rust toolchain matching your Mac's architecture. Makefile.macos uses the Make supplied by the developer tools:

make -f Makefile.macos
make -f Makefile.macos test
make -f Makefile.macos install PREFIX="$HOME/.local"
make -f Makefile.macos clean

The default build directory is build/macos; override it with BUILD_DIR=path. The install prefix defaults to /usr/local. CMake selects aarch64-apple-darwin or x86_64-apple-darwin and forwards an explicitly configured CMAKE_OSX_DEPLOYMENT_TARGET to Cargo.

Windows

Install Visual Studio 2022 Build Tools with the Desktop development with C++ workload and a Windows SDK, CMake, and Rust's MSVC toolchain. Open a Visual Studio Developer Command Prompt for the same architecture as your Rust toolchain (for example, the x64 Native Tools prompt with x86_64-pc-windows-msvc). Run these commands from the repository root:

nmake /f Makefile.windows
nmake /f Makefile.windows test
nmake /f Makefile.windows install PREFIX="C:\rawdinal"
nmake /f Makefile.windows clean

Makefile.windows uses NMAKE, not GNU Make. It defaults to build\windows for builds and build\install for installation; override them with BUILD_DIR=path and PREFIX=path. CMake selects the corresponding MSVC Rust target for x64, x86 or ARM64; that target must be installed. MinGW is not supported by this build definition.

CMake integration and installation

Both makefiles build the release static library and C ABI test executables. Their test target also runs the Rust workspace tests. CMake invokes Cargo with --release --offline --locked; run cargo fetch --locked once while online to populate the dependency cache before using CMake or the makefiles. Build output lives under the chosen build directory, with Cargo artifacts in its cargo/ subdirectory. No system libjxl or libjpeg installation is required.

The CMake build exposes the rawdinal target, including platform-specific system link libraries, when included with add_subdirectory:

add_subdirectory(path/to/rawdinal)
target_link_libraries(your_application PRIVATE rawdinal)

Standalone builds install the static library, include/rawdinal.h and license files by default. Embedded builds can opt in with RAWDINAL_INSTALL=ON. For a direct CMake build, use cmake --install build --config Release --prefix /your/prefix; with a multi-configuration generator, also pass --config Release to the build and -C Release to CTest. The makefiles provide configure, build (the default), test, install and clean targets. clean removes generated build products while retaining the configured build tree and Cargo caches.

Validation

X3F

The ignored byte-for-byte decoder test compares unprocessed physical sensor layers, not finished TIFF renderings. Its fixtures are generated by a separately executed X3F Tools decoder; shared format assumptions can still produce matching results. The existing test names use independent in this execution sense, not as a claim of independent algorithmic provenance or universal ground truth. To generate new fixtures, fetch the immutable Kalpanika/x3f commit e8b48cac0de92c1f0e694ac87a9e0fc31d65cbe3 as a tarball or checkout, then compile the dumper separately from Rawdinal's build systems:

git clone https://github.com/Kalpanika/x3f.git x3f
git -C x3f checkout --detach e8b48cac0de92c1f0e694ac87a9e0fc31d65cbe3
test "$(git -C x3f rev-parse HEAD)" = e8b48cac0de92c1f0e694ac87a9e0fc31d65cbe3
test -z "$(git -C x3f status --porcelain)"
cc -std=c11 -Wall -Wextra -Werror -I x3f/src \
  -DX3F_REFERENCE_DECODER='"Kalpanika/x3f@e8b48cac0de92c1f0e694ac87a9e0fc31d65cbe3"' \
  -c tests/reference_dump.c -o reference_dump.o
cc reference_dump.o x3f/src/x3f_io.c x3f/src/x3f_printf.c -o reference_dump
mkdir reference
./reference_dump INPUT.x3f reference
X3F_SAMPLE=INPUT.x3f X3F_REFERENCE_DIR=reference \
  cargo test -p rawdinal --test decoding sample_matches_independent_reference_byte_for_byte -- --ignored --exact
X3F_SAMPLE_DIR=.opencode/x3f-samples X3F_REFERENCE_DIR=.opencode/x3f-references \
  cargo test -p rawdinal --test decoding corpus_matches_independent_references_byte_for_byte -- --ignored --exact

Cargo and CMake do not build the external X3F Tools source. The dumper creates reference.txt, calibration.bin, and one little-endian layer-N.u16le file for each physical layer; the test requires all of them. Output creation is exclusive: after a failed or partial dump, replace the output directory with a fresh empty directory before retrying.

For the 17-file raw.pixls.us Sigma corpus, generate references for the 11 supported files with the same dumper. Preserve the model directories and use the input filename without its extension as the reference directory: sd Quattro H/SDIM0061.X3F maps to sd Quattro H/SDIM0061/. The six unsupported files require only their original X3Fs. Sample files and comparison dumps are external fixtures, not bundled with the library; the downloaded manifest.json records source URLs, SHA-256 hashes and per-file licenses.

X3F_PIXLS_SAMPLE_DIR=.opencode/x3f-samples/raw-pixls-us \
X3F_PIXLS_REFERENCE_DIR=.opencode/x3f-references/raw-pixls-us \
  cargo test --release --locked -p rawdinal --test decoding \
  pixls_corpus_matches_independent_references_and_rejects_unsupported_formats -- --ignored --exact

The CAMF-2 test uses the SD9, SD10 and SD14 originals from the same corpus. It verifies their source hashes, decrypted payload hashes and parsed entry counts (48, 54 and 94), and checks that sensor decoding remains unsupported. The expected hashes are included in the test:

X3F_PIXLS_SAMPLE_DIR=.opencode/x3f-samples/raw-pixls-us \
  cargo test --release --locked -p rawdinal --test camf_legacy \
  legacy_camf2_matches_known_payload_hashes -- --ignored --exact

ProRAW

The default tests use synthetic fixtures. To exercise your own supported ProRAW files, run the ignored corpus test:

PRORAW_CORPUS=/path/to/dng-directory \
  cargo test --locked -p rawdinal --test proraw decodes_proraw_corpus -- --ignored --exact

For an independent sample comparison, tests/proraw_reference.py requires numpy==2.2.6, tifffile==2025.5.10 and imagecodecs==2025.3.30 in a separate Python environment. The output directory must not already exist:

python3 tests/proraw_reference.py INPUT.dng NEW_REFERENCE_DIRECTORY
PRORAW_SAMPLE=INPUT.dng PRORAW_REFERENCE_DIR=NEW_REFERENCE_DIRECTORY \
  cargo test --locked -p rawdinal \
  lossless_jpeg::tests::proraw_tiles_match_independent_samples_before_linearization -- --ignored --exact
PRORAW_SAMPLE=INPUT.dng PRORAW_REFERENCE_DIR=NEW_REFERENCE_DIRECTORY \
  cargo test --release --locked -p rawdinal --test proraw \
  assembled_normalized_image_matches_formula_reference -- --ignored --exact --nocapture
cargo test --release --locked -p rawdinal --test proraw_extended \
  decodes_synthetic_48_megapixel_single_strip -- --ignored --exact

Schema-2 references record and verify source, tile, manifest and full-stage SHA-256 hashes, dependency versions, selected IFD and geometry. TIFF parsing and SOF3 decoding use separate libraries; assembly and normalization use an independently written formula implementation, not an independent complete DNG engine. The generator currently accepts chunky tiled SOF3 inputs without opcodes. Neither comparison validates camera-to-working-space conversion or the Apple look. The 48 MP test allocates a large synthetic image and is separate from real-device qualification.

Python and its reference dependencies are not needed to build or use Rawdinal. The environment-variable examples above use POSIX shell syntax; in a Windows Command Prompt, set each variable with set "NAME=value" before running Cargo.

X3F reconstruction diagnostics

The default suite includes literal known-answer entropy, predictor and CAMF vectors, plus synthetic reconstruction checks. An additional test-only diagnostic compares bilinear and guided reconstruction against native lower-plane samples, before white balance or color conversion:

X3F_SAMPLE_DIR=.opencode/x3f-samples \
  cargo test --release --locked -p rawdinal \
  experimental::diagnostics::sd_quattro_corpus_reports_native_layer_residuals \
  -- --ignored --exact --nocapture

This requires the five original 20220305-of_intermedio_ritratto-001.x3f through -005.x3f files and raw-pixls-us/sd Quattro/sample3.X3F, relative to the sample directory. It needs no external decoder or comparison dumps. Output reports accepted/excluded sample counts and residual bias, RMS and 95th-percentile absolute error in native code units.

Additional causes rows distinguish CAMF bad-pixel locations (bit 1), the hard-coded AF grid (bit 2), direct estimated encoded-maximum clipping (bit 4), and clipped-donor influence at a repaired location (bit 8). They report marginal counts, the number with multiple causes, and a 16-bin exact-combination histogram. Marginal counts overlap; histogram bin 0 is clear, and bins 1–15 partition the flagged population. A defective source code is excluded from direct clipping observations; the repair flag means a donor used by the repair was clipped, rather than that every repair is clipped.

The native-full-plane scope counts each encoded plane, including dark shields. projected-common counts the native lower-plane coordinates eligible for evaluation, after propagating causes through the common interpolation and regression support used by both methods. These populations have different dimensions and must not be summed together. If every candidate coordinate is excluded, the diagnostic reports its counts with NA residual statistics.

The diagnostic uses the renderer's half-pixel alignment and an assumed 2×2 averaging footprint. Both methods are evaluated on the same pixels, excluding support that touches known defects or estimated encoded maxima. These are conditional data-consistency measurements, not calibrated reconstruction accuracy: camera noise, excluded regions and the assumed sampling model limit their interpretation. Synthetic tests also characterize suppression of lower-plane detail absent from a flat top-layer guide.

A private, test-only restoration model operates on explicit intermediate buffers. Its synthetic tests cover constant fields, weighting phases, highlight transitions, nonpositive signal handling, invalid geometry and nonfinite arithmetic. It is not a public reconstruction option or a complete camera renderer.

C API ownership

• rawdinal_preview returns JPEG storage borrowed from the X3F input buffer.
• rawdinal_sensor_v1_decode owns the X3F physical planes, copied EXIF and decompressed CAMF. Getters borrow those buffers until rawdinal_sensor_v1_free.
• rawdinal_raw_v1_decode owns ProRAW samples and scoped metadata. rawdinal_raw_v1_decode_extended adds optional codes, diagnostic flags and masks. The info, metadata, directory, tag and mask getters borrow handle-owned storage until rawdinal_raw_v1_free; input bytes are borrowed only during decoding.
• rawdinal_decode returns an experimental X3F image handle whose EXIF pointer remains valid until rawdinal_free. rawdinal_copy_rgba copies unbounded linear-sRGB RGBA into caller-owned storage. rawdinal_decode_with_clipping_v1 also retains the masks borrowed through rawdinal_get_clipping_v1 until rawdinal_free.

Use the matching free function for each handle. See the header for pointer validity, non-overlap and concurrency requirements.

Malformed X3F input

The X3F parser limits input to 512 MiB and 64 non-overlapping sections. Decoding accepts only the sensor geometries listed above. Before allocating sensor planes, it checks every plane's byte range and whether its entropy stream can hold the declared samples. Invalid Huffman codes, truncated differences and out-of-range predictors return errors.

CAMF extraction is capped at 32 MiB for decrypted type-2 payloads and decompressed type-4/5 payloads. Calibration parsing also limits expanded matrix values to 32 MiB in total, with at most 4,096 entries, 4,096 properties per entry, 65,536 properties across all entries and a 4 MiB aggregate property-string budget, including terminators. Matrix payloads and property strings must follow their descriptors and offset tables. These bounds cover individual decoding operations, not total process memory or concurrent callers.

The default test suite exercises truncated sections with intact directories, hostile offsets and lengths, metadata expansion limits, malformed JPEG/EXIF and deterministic byte mutations. Native-size fixtures reach RAW entropy validation for every supported format. C-interface regressions check that malformed inputs return an input error rather than a caught panic, clear image outputs and terminate error messages.

Validation is staged: X3f::parse checks the container; decode, calibration and metadata accessors validate the data they consume. preview borrows JPEG bytes without validating the whole JPEG, and camf_bytes extracts CAMF without interpreting its entries. CAMF-2 has no integrity check: shortened or altered payloads may extract successfully but fail entry parsing. The sensor C API likewise returns uninterpreted CAMF and treats unreadable optional EXIF as absent. Successful decoding cannot detect every byte alteration that still describes valid samples or metadata. The mutation tests are regression coverage, not an exhaustive fuzzing or security guarantee.

License

Copyright (C) 2026 Paolo SANTUCCI

Rawdinal is distributed under GPL-3.0-only. See LICENSE for the complete license text. Retained X3F Tools portions are licensed under the BSD 3-Clause License; see THIRD_PARTY_NOTICES.