This file describes the current 4-page CoLorAI submission pipeline. It prepares a rigorous CIFAR-10 DDPM study with baseline anchors, a rank tradeoff, a no-orthogonality ablation, and mandatory qualitative samples. It intentionally does not include higher-resolution, latent-diffusion, or second-dataset experiments for this deadline.
Hardware assumed: B200 or similar CUDA GPU for full runs. Local machines should only run smoke tests.
conda env create -f environment.yml
conda activate nsa_diff
pip install -e .Fast local validation:
conda run -n nsa_diff pytest tests/test_losses.py tests/test_cp_decompose.py tests/test_hooks.py -qDo not run the full pipeline locally. Use a tiny smoke run only after code changes:
conda run -n nsa_diff python scripts/train.py \
--method nsa_diff \
--rank_ratio 0.7 \
--num_steps 1 \
--batch_size 1 \
--num_workers 0 \
--use_wandb false \
--output_dir outputs/smoke \
--run_name smoke_nsa_diffCheck that the loss is finite and a checkpoint is written.
The submitted operating point is rank_ratio=0.7, trained for 50K steps with batch size 256. Loss weights were selected by coarse tuning to keep the denoising, KD, NSA, conditional NSA, and orthogonality terms comparable after warmup:
alpha=1.0, alpha_s=1.0, beta=0.5, lambda=0.01, warmup_steps=1000
Launch the main paper jobs on the server:
RANK=0.7 NUM_STEPS=50000 BATCH_SIZE=256 bash scripts/run_paper_experiments.shThis runs:
lowrank_kd_r0.7
standard_nsa_r0.7
nsa_diff_r0.7
nsa_diff_no_orth_r0.7
nsa_diff_r0.5
nsa_diff_r0.25
The main result table should compare Teacher, CP/init if reported, low-rank KD, standard NSA, NSA-Diff, and NSA-Diff without L_orth at rho=0.7. The rank table should report NSA-Diff at rho in {0.7, 0.5, 0.25}.
Generate 50K samples for each completed checkpoint with the same DDPM scheduler and 1,000 denoising steps:
python scripts/evaluate.py \
--checkpoint outputs/paper/nsa_diff_r0.7/checkpoint_50000.pt \
--num_samples 50000 \
--batch_size 256 \
--scheduler ddpm \
--num_steps 1000 \
--output_dir eval_samples/nsa_diff_r0.7 \
--benchmarkPrepare CIFAR-10 reference images:
python - <<'PY'
import os
import torchvision
from torchvision.utils import save_image
from torchvision.transforms.functional import to_tensor
ds = torchvision.datasets.CIFAR10(root="./data", train=True, download=True)
os.makedirs("eval_samples/cifar10_ref", exist_ok=True)
for i, (img, _) in enumerate(ds):
save_image(to_tensor(img), f"eval_samples/cifar10_ref/{i:06d}.png")
print(f"Saved {len(ds)} reference images.")
PYCompute FID:
python -m pytorch_fid eval_samples/nsa_diff_r0.7 eval_samples/cifar10_refUse the same command for teacher and all baselines.
The main paper must include generated samples. Use the fixed-seed grid script after checkpoints are available:
python scripts/make_sample_grid.py \
--teacher \
--cp_init "CP init=0.7" \
--checkpoint "Low-rank KD=outputs/paper/lowrank_kd_r0.7/checkpoint_50000.pt" \
--checkpoint "Standard NSA=outputs/paper/standard_nsa_r0.7/checkpoint_50000.pt" \
--checkpoint "NSA-Diff=outputs/paper/nsa_diff_r0.7/checkpoint_50000.pt" \
--num_samples 16 \
--cols 8 \
--num_steps 1000 \
--seed 42 \
--output latex_report/figures/qualitative_grid.pngDo not move this figure to supplementary material; it is part of the response to prior reviewer feedback.
After FID values are known, collect rows into a CSV/JSON file:
python scripts/collect_results.py \
--teacher \
--checkpoint lowrank_kd=outputs/paper/lowrank_kd_r0.7/checkpoint_50000.pt \
--checkpoint standard_nsa=outputs/paper/standard_nsa_r0.7/checkpoint_50000.pt \
--checkpoint nsa_diff=outputs/paper/nsa_diff_r0.7/checkpoint_50000.pt \
--checkpoint nsa_diff_no_orth=outputs/paper/nsa_diff_no_orth_r0.7/checkpoint_50000.pt \
--fid teacher=13.71 \
--fid lowrank_kd=<VALUE> \
--fid standard_nsa=<VALUE> \
--fid nsa_diff=<VALUE> \
--fid nsa_diff_no_orth=<VALUE> \
--output results/colorai_main.csvAdd --benchmark only when single-step latency and peak memory are needed. Report it as single-denoising-step latency on the measured GPU, not as edge-device deployment evidence.
Main baseline table:
| Model | rho | Params | Compression | FID |
|---|---|---|---|---|
| Teacher | - | 35.75M | 1x | ? |
| CP init | 0.7 | ? | ? | ? |
| Low-rank KD | 0.7 | ? | ? | ? |
| Standard NSA | 0.7 | ? | ? | ? |
| NSA-Diff | 0.7 | ? | ? | ? |
| NSA-Diff no-orth | 0.7 | ? | ? | ? |
Rank tradeoff table:
| Model | rho | Params | Compression | FID |
|---|---|---|---|---|
| NSA-Diff | 0.7 | ? | ? | ? |
| NSA-Diff | 0.5 | ? | ? | ? |
| NSA-Diff | 0.25 | ? | ? | ? |
Tiny sensitivity note:
| Setting | Purpose | Include if completed |
|---|---|---|
nsa_diff_no_orth |
Verifies the value of L_orth |
Main table |
one alternate alpha_s |
Checks conditional loss sensitivity | Short text note |
- OOM: reduce
--batch_sizeor use gradient accumulation outside this script. - NaN loss: reduce
--lrto5e-5, keep--grad_clip_norm 1.0, and check CP factor cache consistency. - MPS/CPU: use only smoke tests; full FID generation is CUDA-targeted.
- CIFAR-10 download fails: manually place the dataset under
./data/cifar-10-batches-py/.