Kvasir-VQA-x1

A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy

🚧 Work in Progress — this repository is under active development. The training and evaluation code is being finalised and will be released soon.

If you urgently need access or have questions, contact 📧 sushant@simula.no.

🧠 About

Kvasir-VQA-x1 is a multimodal dataset for medical visual question answering (MedVQA) in GI endoscopy. Building on the original Kvasir-VQA, it adds 159,549 new QA pairs with richer reasoning and complexity stratification (Levels 1–3), plus weakly-augmented image variants for robustness testing.

This repository provides the full, reproducible pipeline:

• Dataset generation — LLM-assisted QA merging + answer naturalisation (data/)
• Image augmentation — weak perturbations for the robustness track (augmentation/)
• Training — LoRA recipes for Qwen2.5-VL and MedGemma via MS-Swift (training/)
• Inference — batched generation over the test sets (training/infer.sh)
• Evaluation — NLG metrics (Table 6), LLM-as-a-judge adjudication (Table 7), and paper figures (Figs 1–3) (evaluation/)
• Reference results — the aggregate metric + adjudication JSONs behind the paper tables (results/)

🖼️ Quick usage (run a fine-tuned model)

pip install ms-swift==3.8.0 bitsandbytes qwen_vl_utils==0.0.11

import torch
from swift.llm import PtEngine, RequestConfig, InferRequest
from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16)
engine = PtEngine(adapters=["SimulaMet/Qwen2.5-VL-KvasirVQA-x1-ft"],
                  model_id_or_path="Qwen/Qwen2.5-VL-7B-Instruct",
                  quantization_config=bnb, attn_impl="sdpa", use_hf=True)
req = RequestConfig(max_tokens=512, temperature=0.3, top_k=20, top_p=0.7, repetition_penalty=1.05)
r = InferRequest(messages=[{"role": "user", "content": [
    {"type": "image", "image": "https://huggingface.co/datasets/SimulaMet/Kvasir-VQA-x1/resolve/main/images/clb0kvxvm90y4074yf50vf5nq.jpg"},
    {"type": "text", "text": "What is shown in the image?"}]}])
print(engine.infer([r], req)[0].choices[0].message.content)

See the Colab usage notebook.

📦 Models trained in this work

Model	Hugging Face
Qwen2.5-VL-KvasirVQA-x1-ft	SimulaMet/Qwen2.5-VL-KvasirVQA-x1-ft
Qwen2.5-VL-Transf-KvasirVQA-x1-ft	SimulaMet/Qwen2.5-VL-Transf-KvasirVQA-x1-ft
MedGemma-KvasirVQA-x1-ft	SimulaMet/MedGemma-KvasirVQA-x1-ft

🧾 Dataset structure

Each sample: img_id, complexity (1–3), question, answer, original (source atomic QA pairs), question_class (clinical categories). Full data on Hugging Face.

🧪 Evaluation tracks

• Track 1 — Normal: QA on the original images.
• Track 2 — Transformed: QA on weakly-augmented images (generated by the scripts here).

🔬 Reproducing the paper end to end

0. Setup

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# BLEURT:
pip install git+https://github.com/google-research/bleurt.git
cp .env.example .env   # then edit VLLM_BASE_URL etc.

Training/inference additionally need the MS-Swift stack (ms-swift==3.8.0, deepspeed) on 4–8 GPUs.

1. Build the dataset (optional — data is already on the Hub)

Point a vLLM server at Qwen3-30B-A3B, then:

vllm serve Qwen/Qwen3-30B-A3B --port 8000 --host 0.0.0.0 --max-model-len 4096
export VLLM_BASE_URL="http://localhost:8000/v1" OPENAI_API_KEY="EMPTY"

python data/generate_requests.py --out final_requests.jsonl
python data/call_lmm.py         --in final_requests.jsonl --out grouped_results.json
python data/push_to_hf.py       --in grouped_results.json   # add --push --repo <you>/... to upload

2. Prepare VLM-ready JSONLs

python data/prepare_vqa_format.py            # Track 1 (original images)  -> data/1_transform_*.jsonl
python augmentation/generate_augmented.py    # 10 weak variants per image -> data/image_weak_augmented/
python data/prepare_transformed_format.py    # Track 2 (augmented)        -> data/2_transform_*.jsonl

3. Fine-tune (LoRA, MS-Swift)

bash training/train_qwen25vl.sh     # Qwen2.5-VL-7B, Track 1
bash training/train_medgemma.sh     # MedGemma-4B,   Track 2

4. Inference

ADAPTER=SimulaMet/Qwen2.5-VL-KvasirVQA-x1-ft \
VAL=data/1_transform_to_vqa_format_test.jsonl \
RESULT=results/pred_qwen25vl_ft.jsonl \
bash training/infer.sh

5. Evaluation

# Table 6 — NLG metrics
python evaluation/compute_metrics.py --in "results/pred_*.jsonl" --out-dir results/scores
python evaluation/combine_scores.py  --in-dir results/scores

# Table 7 — LLM-as-a-judge (needs the Qwen3 endpoint from step 1)
python evaluation/llm_adjudicator.py --in "results/pred_*.jsonl" --out-dir results/eval

# Figures 1–3
python evaluation/make_figures.py \
  --model gemma3=results/eval/1_transform_to_vqa_format_test_pred_gemma3_eval.json \
  --model medgemma=results/eval/1_transform_to_vqa_format_test_pred_medgemma-4b_eval.json \
  --model medgemma-ft=results/eval/1_transform_to_vqa_format_test_pred_medgemma-3952_eval.json \
  --model Qwen2.5-VL-7B=results/eval/1_transform_to_vqa_format_test_pred_Qwen2.5-VL_eval.json \
  --model Qwen2.5-VL-7B-ft=results/eval/1_transform_to_vqa_format_test_pred_v1-4444_eval.json \
  --out-dir results/figures

The reference results/scores/total.json and results/eval/*_eval.json shipped here let you regenerate the tables/figures without re-running inference.

📁 Repository layout

data/          dataset generation + VLM-ready JSONL builders
augmentation/  weak image augmentation (Track 2)
training/      LoRA training + inference recipes (MS-Swift)
evaluation/    NLG metrics, LLM adjudicator, figures
results/       reference metric + adjudication JSONs (paper tables)
notebooks/     usage.ipynb (quickstart)

📜 License

Dataset and code: CC BY-NC 4.0.

📌 Citation

@incollection{Gautam2025Oct,
  author    = {Gautam, Sushant and Riegler, Michael and Halvorsen, P{\aa}l},
  title     = {Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy},
  booktitle = {Data Engineering in Medical Imaging},
  year      = {2025},
  publisher = {Springer, Cham},
  doi       = {10.1007/978-3-032-08009-7_6}
}

@article{Gautam2025Jun,
  author  = {Gautam, Sushant and Riegler, Michael A. and Halvorsen, P{\aa}l},
  title   = {{Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy}},
  journal = {arXiv},
  year    = {2025},
  month   = jun,
  eprint  = {2506.09958},
  doi     = {10.48550/arXiv.2506.09958}
}