[feature] Enable Dynamic Batching

[eplatero97]

Objective

This PR enables dynamic batching for QAIC continuous-batching QPCs by using batch_index
KV-cache slicing.

For example, compile(batch_size=[1, 2, 4], continuous_batching=True, full_batch_size=4)
builds one QPC whose retained KV cache is always sized for batch 4, while runtime execution can
drive live decode batches 1, 2, or 4.

Assumptions and limitations

Dynamic batching is implemented only for continuous-batching QPCs. A list-valued batch_size
requires continuous_batching=True and an explicit full_batch_size.

The supported cartesian product is dynamic batching with multi-spec speculative decoding. Both
specialization discriminators are carried by input_ids shape: batch varies on axis 0, and
speculative decode length varies on axis 1.

By "uniquely identify," we mean the QAIC runtime/compiler must be able to select one decode
specialization from the input tensor shapes:

Combination	Specialization discriminator	Status
Dynamic batching	input_ids axis 0 (batch)	Supported
Multi-spec speculative decoding	input_ids axis 1 (seq_len)	Supported
Dynamic batching x multi-spec speculative decoding	input_ids axes 0 and 1	Supported
CCL	comp_ctx_lengths length	Supported
Dynamic batching x CCL	split across input_ids and comp_ctx_lengths	Rejected
Multi-spec speculative decoding x CCL	split across input_ids and comp_ctx_lengths	Rejected

For example:

Dynamic batch x SpD

b=1, k=1 -> input_ids (1, 2)
b=1, k=3 -> input_ids (1, 4)
b=2, k=1 -> input_ids (2, 2)
b=2, k=3 -> input_ids (2, 4)

Every row has a unique input_ids shape, so input_ids alone identifies the spec.

CCL is different because it uses the length of a separate comp_ctx_lengths input:

Dynamic batch x CCL

b=1, ccl=1024 -> input_ids (1, 1), comp_ctx_lengths (1024,)
b=1, ccl=2048 -> input_ids (1, 1), comp_ctx_lengths (2048,)
b=2, ccl=1024 -> input_ids (2, 1), comp_ctx_lengths (1024,)
b=2, ccl=2048 -> input_ids (2, 1), comp_ctx_lengths (2048,)

input_ids is not unique, and comp_ctx_lengths is not unique.

The rejected products previously reached the QAIC compiler and failed with:

No input that uniquely identifies specialization

This PR rejects those combinations at compile time with clear ValueErrors instead. Plain CCL,
dynamic batching without CCL, scalar speculative decoding with CCL, and dynamic batching with
multi-spec speculative decoding remain supported as applicable.

[eplatero97]

Dynamic batching decode latency performance

Ran latency comparison performance between baseline and dynamic QPCs compiled using the same full_batch_size.

Test configuration

Item	Value
Model	JackFram/llama-68m
Context / prefill	ctx_len=2048, prefill_seq_len=128
Runtime	QAIC, 1 device, 16 cores, continuous batching
Timing boundary	QAICInferenceSession.run(inputs) only
Samples	50 warmups + 500 timed samples
Metric	decode latency p50 / p90 / p95

Experiment 1: full_batch_size=128

Baseline: one decode specialization per QPC, batch_size=B, full_batch_size=128
Dynamic: one QPC with decode batches [1,2,4,8,16,32,64,128], full_batch_size=128

Batch	Baseline p50 ms	Dynamic p50 ms	p50 delta
1	1321.9	1283.0	-2.9%
2	1385.7	1294.7	-6.6%
4	1301.9	1283.8	-1.4%
8	1336.4	1289.7	-3.5%
16	1360.5	1298.7	-4.5%
32	1375.1	1327.0	-3.5%
64	1439.5	1352.9	-6.0%
128	1412.0	1343.8	-4.8%

Experiment 2: full_batch_size=8, paired run

Baseline: one decode specialization per QPC, batch_size=B, full_batch_size=8
Dynamic: one QPC with decode batches [1,2,4,8], full_batch_size=8

Batch	Baseline p50 ms	Dynamic p50 ms	p50 delta
1	76.5	77.5	-0.5%
2	75.8	75.8	-1.5%
4	75.9	75.4	-1.4%
8	78.3	84.2	+8.5%

Interpretation

With matched full_batch_size=128, the dynamic QPC with 8 decode specializations is within a few percent of the single-specialization baselines across all tested batches. The measured p50 deltas range from -1.4% to -6.6%.

With matched full_batch_size=8, the dynamic QPC with 4 decode specializations is effectively at parity for batches 1, 2, and 4. Batch 8 shows the clearest dynamic overhead signal, with p50 latency increasing from 78.3 ms to 84.2 ms (+8.5%).

Limitations

• These results measure steady-state decode QAICInferenceSession.run() latency only.
• The tested model is JackFram/llama-68m, which is a small non-production model.
• They do not cover production serving latency, throughput, compile time, multi-device behavior, or other models/configs.

[eplatero97]

follow-up experiment:

Experiment 3: full_batch_size=128, paired batch/K run

Baseline: 8 QPCs, one per batch size. Each QPC contains five decode
specializations for K=[0,3,7,15,31].

Dynamic: one QPC with decode batches
[1,2,4,8,16,32,64,128] and K=[0,3,7,15,31].

The corresponding decode sequence lengths are K+1=[1,4,8,16,32].
All QPCs use full_batch_size=128.

Batch	Baseline p50 ms	Dynamic p50 ms	Paired p50 delta
1	1371.5	1408.5	+1.48%
2	1366.2	1434.8	+3.91%
4	1392.3	1449.0	+4.13%
8	1399.0	1451.5	+3.20%
16	1406.2	1456.7	+2.34%
32	1387.2	1462.5	+4.34%
64	1526.4	1576.6	+1.31%
128	1650.9	1675.1	+0.16%

The batch-level values summarize the five K values for each batch. Baseline
and dynamic p50 values are medians across the K-specific medians.
The paired delta is calculated per (batch, K) on the same QID,
then reduced by taking the median across K.

K	Decode sequence length	Paired p50 delta median across batches
0	1	+1.44%
3	4	+2.51%
7	8	+4.09%
15	16	+1.83%
31	32	+3.46%

Interpretation

With full_batch_size=128 matched, expanding from five decode
specializations in each baseline QPC to all 40 batch/K specializations in
the dynamic QPC produces <5% regressions.

NOTE: this experiment specifically paired the same QID for respective baseline and dynamic batch run to minimize SOC-level variance