Skip to content

[codex] Support packing students into GPU pods - #3456

Open
morganmcg1 wants to merge 7 commits into
mainfrom
codex/multi-student-gpu-pods
Open

[codex] Support packing students into GPU pods#3456
morganmcg1 wants to merge 7 commits into
mainfrom
codex/multi-student-gpu-pods

Conversation

@morganmcg1

@morganmcg1 morganmcg1 commented May 22, 2026

Copy link
Copy Markdown
Member

Summary

Adds an opt-in deployment topology for scarce-GPU targets where multiple logical SENPAI students share one GPU pod.

  • Adds students_per_gpu_pod with default 1, preserving the current one-student-per-pod behavior.
  • Renders singleton and grouped student deployments from one reusable student deployment template.
  • Adds a grouped student entrypoint that launches isolated student loops under /workspace/senpai-<student> with separate HOME directories.
  • Annotates student deployments with senpai/student-names so advisor pod-health helpers can find both normal and grouped student pods.
  • Adds shared process ownership helpers so watchdog, stale-WIP, pod-anomaly, and status tooling can scope train.py, GPU, and Claude activity to the right logical student checkout.
  • Updates log harvest/status helpers for grouped student homes and documents the packed-GPU knob.

Why

Some targets, such as short inference-optimization benchmarks, have one large GPU but still benefit from several logical students preparing launchers, analyzing logs, doing smoke checks, and taking turns on the same device. This keeps the PR-routing model intact while allowing a single GPU pod to host a small student fleet.

Validation

  • python3 -m py_compile k8s/launch.py k8s/launch_helpers.py
  • bash -n k8s/entrypoint-advisor.sh k8s/entrypoint-student.sh k8s/entrypoint-student-group.sh k8s/student-claude-watchdog.sh k8s/senpai-processes.sh plugins/senpai/scripts/senpai-gh.sh scripts/arm_senpai_cluster_cutoff.sh
  • sh -n k8s/senpai-processes.sh
  • git diff --check
  • Dry-run packed manifest: uv run --frozen --with simple-parsing python k8s/launch.py --tag pack-test --target_repo_url https://github.com/morganmcg1/tandemfoil2.git --names alice,bob,charlie --gpus_per_student 1 --students_per_gpu_pod 2 --dry_run
  • Dry-run default manifest: uv run --frozen --with simple-parsing python k8s/launch.py --tag solo-test --target_repo_url https://github.com/morganmcg1/tandemfoil2.git --names alice,bob --gpus_per_student 1 --students_per_gpu_pod 1 --dry_run
  • YAML parse of both rendered dry-run manifest streams after stripping dry-run section headers.

Notes

This is intentionally minimal: one global image remains in use, the advisor deployment is unchanged, grouped mode is opt-in, and singleton/grouped student deployments share one template. The advisor still owns research coordination and GPU scheduling through assignments and PR comments.

@github-actions

Copy link
Copy Markdown


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@morganmcg1
morganmcg1 marked this pull request as ready for review May 23, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants