AProjQ4: imatrix-guided Q4_K attention and GPU runtime optimizations - #952

Open

GiorgioOppo wants to merge 86 commits into

antirez:mainfrom

GiorgioOppo:aprojq4-dense-attention

Commits on Sep 9, 2026

Add Q4_K attention dispatch and direct GGUF requantization
GiorgioOppo
committed
1fcd662
Port automatic Q4 attention kernels across GPU backends
GiorgioOppo
committed
c9474f6
Document Q4 dispatch, SSD memory policy and validation
GiorgioOppo
committed
b8507c9
Restore GPU decode defaults and fix ROCm KV ring writes
GiorgioOppo
committed
82fc226
Account for fixed weights in Metal manual SSD cache budgets
GiorgioOppo
committed
2c9d48f
Add isolated CUDA Q4 grouped tok8 benchmark and validation
GiorgioOppo
committed
670e6b9
Restore GB10 F16 compressor and aligned Q8 HC decode
GiorgioOppo
committed
4a87d9c

Commits on Sep 10, 2026

Commits on Sep 11, 2026

Commits on Sep 12, 2026

Commits on Sep 13, 2026

Fix CUDA long-context smoke test linking

Matthley
authored and
GiorgioOppo
committed
f6f036c
Merge V4.1 Metal SSD and Q4 support into aprojq4-dense-attention
GiorgioOppo
committed
a1c2574
Fuse V4.1 shared Q8 projections and BF16 SwiGLU on Metal
GiorgioOppo
committed
9f7e4eb
Reuse dead attention workspace for V4.1 Q4 Q-B FP16 matmul
GiorgioOppo
committed
73aae30
Cache V4.1 zero-prefix attention masks and block maps
GiorgioOppo
committed
4ec58cc
Dispatch V4.1 MoE prefill through N8 and N16 Metal tiles
GiorgioOppo
committed
7e593df
Fuse V4.1 BF16 activation and hyper-connection epilogues
GiorgioOppo
committed
2da41a4
Stage sixteen KV rows per V4.1 indexed prefill iteration
GiorgioOppo
committed
8cda676
Batch decoder KV publication and skip empty prefill submissions
GiorgioOppo
committed
8ec5f79
rocm: add DeepSeek V4.1 support for gfx1151
kyuz0
committed
48915fb
docs: qualify 256K allocation with 64K ROCm context checks
kyuz0
committed
4f33140
Reuse packed Q2 weights and shared FP16 inputs in V4.1 Metal MoE
GiorgioOppo
committed
249f627
Skip redundant V4.1 candidate ranking and retain growing top-k scratch
GiorgioOppo
committed
99de83c
Reuse selected V4.1 KV rows during sequential Metal decode
GiorgioOppo
committed
ecf0b13
Extend V4.1 SSD prefill sweeps and release temporary Metal storage
GiorgioOppo
committed
c12d639
Isolate DeepSeek V4.1 ROCm allocation and graph paths
kyuz0
committed
fb1516f
Align bulk attention references and document isolation regressions
kyuz0
committed
f120040
docs: describe final V4.1 ROCm support and validation
kyuz0
committed
c89c085
Reuse transient FP16 weights for V4.1 Metal Q4 output-B prefill
GiorgioOppo
committed
482898b
Fuse V4.1 scalar Metal decode HC and BF16 epilogues
GiorgioOppo
committed
5d83b55
Extend V4.1 Metal MoE benchmarks across large prefill batches
GiorgioOppo
committed
a6c27dd
docs: clarify Strix Halo GTT limits for SSD cache
kyuz0
committed
3c346d0
rocm: clarify V4.1 SSD cache fitting diagnostics
kyuz0
committed
8cc0c92
rocm: avoid extra GPU query in cache diagnostic
kyuz0
committed
006b3c1
Merge antirez/main V4.1 CUDA support into aprojq4-dense-attention
GiorgioOppo
committed
96b4659
Batch V4.1 CUDA Q4 attention output with scalar reduction order
GiorgioOppo
committed
8d6f3cc
feat(rocm): add two-rank V4.1 inference over TCP, USB4STREAM and RoCE
kyuz0
committed
d0b8be6
docs: document V4.1 ROCm clustering setup and qualification
kyuz0
committed
05d6991
docs(rocm): expose cluster options and measured transport performance
kyuz0
committed
e35891f

Commits on Sep 14, 2026

Commits on Sep 15, 2026

rocm: accelerate V4.1 batched prefill on gfx1151
kyuz0
committed
449a38d
rocm: overlap V4.1 shared experts with peer exchange
kyuz0
committed
f24cd3a
docs: record qualified ROCm prefill results and commands
kyuz0
committed
09f12d4
rocm: partition V4.1 vocabulary rows between TP peers
kyuz0
committed
cc27cae
rocm: fuse V4.1 scalar TP wait and reduction
kyuz0
committed
85bd209
rocm: parallelize V4.1 decode attention across FP32 key tiles
kyuz0
committed
1e2e033
docs: record qualified ROCm decode results and reproduction
kyuz0
committed
c14f8cb

Commits on Sep 16, 2026

rocm: keep cluster transports to TCP and RoCE
kyuz0
committed
0ed0aa1
Merge main and preserve ROCm model-cache call compatibility
kyuz0
committed
40b1f63

Commits on Sep 17, 2026

Commits on Sep 19, 2026