Skip to content

Latest commit

 

History

History
196 lines (144 loc) · 5.69 KB

EXPERIMENTS.md

File metadata and controls

196 lines (144 loc) · 5.69 KB

CoLorAI Experiment Pipeline

This file describes the current 4-page CoLorAI submission pipeline. It prepares a rigorous CIFAR-10 DDPM study with baseline anchors, a rank tradeoff, a no-orthogonality ablation, and mandatory qualitative samples. It intentionally does not include higher-resolution, latent-diffusion, or second-dataset experiments for this deadline.

Hardware assumed: B200 or similar CUDA GPU for full runs. Local machines should only run smoke tests.


0. Environment Setup

conda env create -f environment.yml
conda activate nsa_diff
pip install -e .

Fast local validation:

conda run -n nsa_diff pytest tests/test_losses.py tests/test_cp_decompose.py tests/test_hooks.py -q

1. Local Smoke Test

Do not run the full pipeline locally. Use a tiny smoke run only after code changes:

conda run -n nsa_diff python scripts/train.py \
    --method nsa_diff \
    --rank_ratio 0.7 \
    --num_steps 1 \
    --batch_size 1 \
    --num_workers 0 \
    --use_wandb false \
    --output_dir outputs/smoke \
    --run_name smoke_nsa_diff

Check that the loss is finite and a checkpoint is written.


2. Paper Training Runs

The submitted operating point is rank_ratio=0.7, trained for 50K steps with batch size 256. Loss weights were selected by coarse tuning to keep the denoising, KD, NSA, conditional NSA, and orthogonality terms comparable after warmup:

alpha=1.0, alpha_s=1.0, beta=0.5, lambda=0.01, warmup_steps=1000

Launch the main paper jobs on the server:

RANK=0.7 NUM_STEPS=50000 BATCH_SIZE=256 bash scripts/run_paper_experiments.sh

This runs:

lowrank_kd_r0.7
standard_nsa_r0.7
nsa_diff_r0.7
nsa_diff_no_orth_r0.7
nsa_diff_r0.5
nsa_diff_r0.25

The main result table should compare Teacher, CP/init if reported, low-rank KD, standard NSA, NSA-Diff, and NSA-Diff without L_orth at rho=0.7. The rank table should report NSA-Diff at rho in {0.7, 0.5, 0.25}.


3. Generation And FID

Generate 50K samples for each completed checkpoint with the same DDPM scheduler and 1,000 denoising steps:

python scripts/evaluate.py \
    --checkpoint outputs/paper/nsa_diff_r0.7/checkpoint_50000.pt \
    --num_samples 50000 \
    --batch_size 256 \
    --scheduler ddpm \
    --num_steps 1000 \
    --output_dir eval_samples/nsa_diff_r0.7 \
    --benchmark

Prepare CIFAR-10 reference images:

python - <<'PY'
import os
import torchvision
from torchvision.utils import save_image
from torchvision.transforms.functional import to_tensor

ds = torchvision.datasets.CIFAR10(root="./data", train=True, download=True)
os.makedirs("eval_samples/cifar10_ref", exist_ok=True)
for i, (img, _) in enumerate(ds):
    save_image(to_tensor(img), f"eval_samples/cifar10_ref/{i:06d}.png")
print(f"Saved {len(ds)} reference images.")
PY

Compute FID:

python -m pytorch_fid eval_samples/nsa_diff_r0.7 eval_samples/cifar10_ref

Use the same command for teacher and all baselines.


4. Mandatory Qualitative Figure

The main paper must include generated samples. Use the fixed-seed grid script after checkpoints are available:

python scripts/make_sample_grid.py \
    --teacher \
    --cp_init "CP init=0.7" \
    --checkpoint "Low-rank KD=outputs/paper/lowrank_kd_r0.7/checkpoint_50000.pt" \
    --checkpoint "Standard NSA=outputs/paper/standard_nsa_r0.7/checkpoint_50000.pt" \
    --checkpoint "NSA-Diff=outputs/paper/nsa_diff_r0.7/checkpoint_50000.pt" \
    --num_samples 16 \
    --cols 8 \
    --num_steps 1000 \
    --seed 42 \
    --output latex_report/figures/qualitative_grid.png

Do not move this figure to supplementary material; it is part of the response to prior reviewer feedback.


5. Result Collection

After FID values are known, collect rows into a CSV/JSON file:

python scripts/collect_results.py \
    --teacher \
    --checkpoint lowrank_kd=outputs/paper/lowrank_kd_r0.7/checkpoint_50000.pt \
    --checkpoint standard_nsa=outputs/paper/standard_nsa_r0.7/checkpoint_50000.pt \
    --checkpoint nsa_diff=outputs/paper/nsa_diff_r0.7/checkpoint_50000.pt \
    --checkpoint nsa_diff_no_orth=outputs/paper/nsa_diff_no_orth_r0.7/checkpoint_50000.pt \
    --fid teacher=13.71 \
    --fid lowrank_kd=<VALUE> \
    --fid standard_nsa=<VALUE> \
    --fid nsa_diff=<VALUE> \
    --fid nsa_diff_no_orth=<VALUE> \
    --output results/colorai_main.csv

Add --benchmark only when single-step latency and peak memory are needed. Report it as single-denoising-step latency on the measured GPU, not as edge-device deployment evidence.


6. Paper Tables

Main baseline table:

Model rho Params Compression FID
Teacher - 35.75M 1x ?
CP init 0.7 ? ? ?
Low-rank KD 0.7 ? ? ?
Standard NSA 0.7 ? ? ?
NSA-Diff 0.7 ? ? ?
NSA-Diff no-orth 0.7 ? ? ?

Rank tradeoff table:

Model rho Params Compression FID
NSA-Diff 0.7 ? ? ?
NSA-Diff 0.5 ? ? ?
NSA-Diff 0.25 ? ? ?

Tiny sensitivity note:

Setting Purpose Include if completed
nsa_diff_no_orth Verifies the value of L_orth Main table
one alternate alpha_s Checks conditional loss sensitivity Short text note

Troubleshooting

  • OOM: reduce --batch_size or use gradient accumulation outside this script.
  • NaN loss: reduce --lr to 5e-5, keep --grad_clip_norm 1.0, and check CP factor cache consistency.
  • MPS/CPU: use only smoke tests; full FID generation is CUDA-targeted.
  • CIFAR-10 download fails: manually place the dataset under ./data/cifar-10-batches-py/.