Skip to content

Qwen3VL: default per-image resolution to a 2,560 vision-token budget#398

Open
spokvulcan wants to merge 2 commits into
ml-explore:mainfrom
spokvulcan:feat/qwen3vl-vision-token-budget
Open

Qwen3VL: default per-image resolution to a 2,560 vision-token budget#398
spokvulcan wants to merge 2 commits into
ml-explore:mainfrom
spokvulcan:feat/qwen3vl-vision-token-budget

Conversation

@spokvulcan

Copy link
Copy Markdown
Contributor

Fixes #396.

The qwen3_5 ViT runs global O(patches²) attention with no window structure, so an uncapped full-resolution image (several published configs ship a ~16 MP longest_edge that never clamps) can allocate a tens-of-GB attention score matrix. Measured on a 27B Qwen3.6 checkpoint: 47 GB peak / 121 s prefill for one full-screen screenshot; ~21 GB with this cap.

  • Default each image — and each video frame — to a 2,560 vision-token budget: min(config.maxPixels, 2560 * factor²), mirroring the Qwen2.5-VL clamp.
  • Honor an explicit processing.maxPixels override, so full resolution stays available as an opt-in.
  • Only what the tower sees is downscaled; stored image bytes are untouched.
  • The budget multiply is guarded with multipliedReportingOverflow so a pathological patch/merge size degrades to the config ceiling instead of trapping, and the image and video preprocess paths clamp through the same config.maxPixels / config.minPixels accessors so the two clamp sites cannot drift.

On the default value: if you'd rather keep upstream's current behavior as the default, I'm happy to reshape this as a pure opt-in knob — the part that matters for on-device use is that a per-image budget exists.

@spokvulcan

Copy link
Copy Markdown
Contributor Author

Lint fixed.

…udget

The qwen3_5 ViT runs global O(patches^2) attention with no window structure,
so an uncapped full-resolution image (PARO ships a ~16 MP longest_edge that
never clamps) can allocate a tens-of-GB attention score matrix. Default each
image -- and each video frame -- to a 2,560 vision-token budget
(min(config.maxPixels, 2560 * factor^2)), mirroring the Qwen2.5-VL clamp, and
honor an explicit processing.maxPixels override. Downscales only what the
tower sees; the stored bytes and image digest are untouched.

The default budget multiply (2560 * factor^2) is guarded with
multipliedReportingOverflow so a pathological patch/merge size degrades to
the config ceiling instead of trapping, and both the image and video
preprocess paths clamp through config.maxPixels / config.minPixels so the
two clamp sites cannot drift.

See tesseract ADR-0014.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@spokvulcan

Copy link
Copy Markdown
Contributor Author

Rebased onto current main. The mac_build_and_test failure here was the CudaBuild package-plugin validation issue that #404 fixed on main (-skipPackagePluginValidation) — it predates this PR's code, which never got compiled in that run. Verified locally on the rebased branch with the CI steps (build-for-testing + MLXLMTests): builds clean, all tests pass. The branch's swift-format-603 style commit was dropped during rebase since #404 already contains the identical change.

@spokvulcan spokvulcan force-pushed the feat/qwen3vl-vision-token-budget branch from 3ad5fbe to c424edd Compare July 9, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen3.5/Qwen3-VL: uncapped per-image resolution lets the ViT allocate tens-of-GB attention buffers

1 participant