Skip to content

fix(write_io): capture complete generate() I/O traces for LLM and VLM models - #1309

Open
talnufai wants to merge 4 commits into
quic:mainfrom
talnufai:write_io_files
Open

fix(write_io): capture complete generate() I/O traces for LLM and VLM models#1309
talnufai wants to merge 4 commits into
quic:mainfrom
talnufai:write_io_files

Conversation

@talnufai

@talnufai talnufai commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix several issues in the write_io_files capture flow when using:

model.generate(..., write_io=True)

This PR resolves QRANIUMSW-63509 by:

  • Capturing the correct chunk_inputs during chunked prefill instead of the full padded inputs
  • Preserving all prefill and decode iterations using indexed file names
  • Capturing missing vision encoder inputs and outputs in VLM paths
  • Preventing collisions between vision and language prefill captures
  • Generating a timestamped io_dir for each generate() invocation
  • Fixing Dual-QPC language ONNX path selection
  • Adding runtime and virtual environment artifacts to .gitignore

Changes

Correct prefill input capture

Use the same chunk_inputs passed to session.run() when writing captured I/O.

Preserve all iterations

Store each generation step in a unique file:

prefill_0
prefill_1
...
decode_1
decode_2
...

Also remove logic that disabled capture after the first decode step.

Capture vision encoder I/O

Add write_io_files support for VLM vision encoder execution paths.

Vision captures are stored separately from language captures:

vision_prefill_0
vision_prefill_1
...

This avoids overwrite/collision with language prefill outputs.

Timestamp output directories

Each generate() call now writes into a unique timestamped directory:

io_dir/<timestamp>/

preventing collisions across multiple runs.

Dual-QPC fix

Replace reliance on self.onnx_path[1] with the explicit language model ONNX path.

Example Output

Text-to-Text Model

~/.cache/qeff_models/Qwen2ForCausalLM/
└── Qwen2ForCausalLM-1657fb5c3133a3d4/
    └── io_dir/
        └── 20260907_033641/
            ├── aic_batch_io.json
            ├── prefill_0
            ├── decode_1
            ├── decode_2
            ├── ...
            └── decode_99

Vision-Language Model

~/.cache/qeff_models/LlavaForConditionalGeneration/
└── LlavaDecoderWrapper-84c9b297d91aeb74/
    └── io_dir/
        └── 20260907_033803/
            ├── aic_batch_io.json
            ├── vision_prefill_0
            ├── prefill_0
            ├── prefill_1
            ├── prefill_2
            ├── prefill_3
            ├── prefill_4
            ├── decode_1
            ├── decode_2
            ├── ...
            └── decode_127

This demonstrates that vision prefill, language prefill, and all decode iterations are now preserved instead of being overwritten.

Result

With write_io=True, all inputs and outputs used throughout generation are captured and preserved for both text and vision-language models, including:

  • Vision prefill
  • Language prefill
  • All decode iterations

Validation

Verified on:

  • Qwen2ForCausalLM
  • GPT2LMHeadModel
  • LlavaForConditionalGeneration

Confirmed:

  • Correct chunked input capture
  • Vision encoder capture
  • No I/O overwrites
  • No vision/lang file collisions
  • Unique output directories per generate() call

@quic-rishinr
quic-rishinr requested a review from vbaddi September 8, 2026 08:25
- Pass chunk_inputs instead of full padded tensor during prefill
- Use indexed subdirs (prefill_{i}, decode_{n}) to preserve all steps
- Remove self._write_io_dir = None that suppressed subsequent captures

Signed-off-by: talnufai <talnufai@qualcomm.com>
… collision

Two write_io_files gaps in the VLM inference paths:

1. modeling_auto.py (kv_offload_generate): vision_session.run() inputs and
   outputs were never captured. Added write_io_files call immediately after
   the vision encoder run using subdir "vision_prefill_0".

2. vlm_generation.py (run_prefill_multi_frame_specialization): each per-frame
   vision_session.run() was never captured. Added write_io_files call inside
   the frame loop using subdir f"vision_prefill_{i}".

Naming: vision encoder subdirs are prefixed "vision_prefill_" to avoid
collision with the lang chunked prefill loop which writes to "prefill_{i}".
Using the same "prefill_{i}" namespace for both sessions would cause the lang
chunk to overwrite the vision .raw files on disk while producing two JSON
entries pointing to the same paths, silently corrupting the captured trace.

Signed-off-by: talnufai <talnufai@qualcomm.com>
…onnx_path indexing

Add get_io_dir() helper that appends an EST timestamp to io_dir so each run
writes to its own subdirectory. Replace all _write_io_dir assignments with it.
Also fix _QEffAutoModelForImageTextToTextDualQPC using self.onnx_path[1] —
replace with explicit self.lang_model.onnx_path.

Signed-off-by: talnufai <talnufai@qualcomm.com>
Signed-off-by: talnufai <talnufai@qualcomm.com>
@talnufai

talnufai commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Hi @quic-rishinr, the Python formatting issue is fixed as you requested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant