Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
b7892a9
Separate live Senpai skills from operator guides
mmcguire Aug 14, 2026
8c012a9
Merge slimmer program prompt parent
mmcguire Aug 14, 2026
54f7c72
Merge restored list experiments wording
mmcguire Aug 14, 2026
459bda6
Merge branch 'codex/program-md-system-prompt' into codex/senpai-runti…
mmcguire Aug 14, 2026
aa761e1
Merge branch 'codex/program-md-system-prompt' into codex/senpai-runti…
mmcguire Aug 14, 2026
e0b21a5
Merge branch 'codex/program-md-system-prompt' into codex/senpai-runti…
mmcguire Aug 14, 2026
b6f5c2a
Merge branch 'codex/program-md-system-prompt' into codex/senpai-runti…
mmcguire Aug 14, 2026
ba9f0e8
Merge branch 'codex/program-md-system-prompt' into codex/senpai-runti…
mmcguire Aug 14, 2026
5ec2679
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
56b9a3e
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
acbe0b8
Keep file-agent skills project-scoped
mmcguire Aug 15, 2026
ac491de
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
e45ba2f
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
968fb4b
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
bbb1d6d
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
560333c
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
7d7090f
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
26918b9
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
58ef094
Merge remote-tracking branch 'origin/codex/program-md-system-prompt' …
mmcguire Aug 15, 2026
930fbaa
Merge main into Senpai runtime skill boundary
mmcguire Aug 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .agents/agents/search.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,6 @@ permission_mode: never_confirm
tools:
- terminal
- file_editor
skills:
- exa-search
- alphaxiv-paper-lookup
---

You are Senpai's external research agent. The delegated prompt begins with one
Expand Down
10 changes: 6 additions & 4 deletions .agents/skills/analyze-experiments/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: analyze-experiments
description: >
Analyzes and categorizes all ML experiment PRs in the senpai research track.
Operator-side analysis of historical ML experiment PRs in Senpai research tracks.
Use this skill whenever the user asks to: analyze experiments, categorize PRs, bucket
experiments, summarize what's been tried, understand experiment history, review merged
vs closed results, or asks "what experiments have we run / worked / failed".
Expand All @@ -13,9 +13,11 @@ description: >

# Analyze Experiments Skill

Fetches fresh experiment PR data via the list-experiments skill, categorizes each PR
using a team of parallel readers, and produces a 5-section report: full catalogue,
category breakdown with merge rates, merged-only wins, closed-only failures, key narratives.
This is a human/operator analysis guide, not a live advisor or student skill.
It fetches fresh experiment PR data via the list-experiments skill, categorizes
each PR using parallel readers, and produces a 5-section report: full catalogue,
category breakdown with merge rates, merged-only wins, closed-only failures,
and key narratives.

---

Expand Down
11 changes: 6 additions & 5 deletions .agents/skills/git-research-log/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
---
name: git-research-log
description: >
How to document and publish ML experiment results to GitHub as a pull request.
Use this skill whenever a student agent has finished running experiments and
needs to create or update a PR — even if the instruction is just "wrap up" or
"log your results" or "open a PR". Also use it mid-session to update an
in-progress PR as trials complete.
Developer reference for Senpai's retired direct-GitHub experiment logging
workflow. Use only when auditing or migrating legacy research tracks.
---

# Experiment PR Skill

This guide is not installed into live advisor or student runtimes. Current
students use the plugin's `submit-experiment-results` skill and typed result
tool instead.

The master agent creates an empty PR and hands it to you. Your job is to fill it out as you run trials and finalize it when you're done. The master reads these PRs to decide what to explore next — completeness and honest analysis matter more than polish.

## PR Title Format
Expand Down
6 changes: 4 additions & 2 deletions .agents/skills/list-experiments/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
---
name: list-experiments
description: Use this skill whenever you need to list all of the experiment ideas tried and in progress for this research programme. It outputs 3 files organized by usefulness — merged winners, a compact results table, and full details for deep dives. Use when generating new experimental ideas to check what has already been tried.
description: Operator-side inventory of experiment PRs for auditing a Senpai research track. It outputs merged winners, a compact results table, and full details for deep dives.
---

# List Experiments

Run the script below to fetch all experiment PRs from the advisor branch and organize them into 3 files.
This is a human/operator audit guide, not a live advisor or student skill. Run
the script below to fetch all experiment PRs from the advisor branch and
organize them into 3 files.

The `BASE_BRANCH` should be set to the advisor branch (e.g. `noam`). Check the `$ADVISOR_BRANCH` env var or the PR base branch.

Expand Down
21 changes: 11 additions & 10 deletions .agents/skills/plot-experiment-charts/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,28 @@

name: plot-experiment-charts
description: >
Generate a training curve comparison chart and embed it in a GitHub PR description.
Use this skill whenever a student has finished running experiments and is preparing
to submit their PR for advisor review. Triggers on: "plot training curves", "add chart
to PR", "visualize experiment", "training curve comparison", "plot-experiment-charts",
"add chart", "generate comparison chart". Run this before marking the PR ready for review.
Operator-side guide for generating a target-specific training curve chart
for a legacy experiment PR. Use when auditing or presenting historical runs.
---

# Plot Experiment Charts

You've just finished one or more training runs. Before submitting for review, generate a
comparison chart so the advisor can see the training dynamics at a glance — not just the
final numbers, but how the experiment got there. A bolded best-run line and a properly
scaled y-axis make the story immediately readable, even if some runs diverged.
This guide is not installed into live advisor or student runtimes. Target
repositories that require experiment charts should provide their own project
skill with the correct metrics and plotting code.

Generate a comparison chart so a reviewer can see the training dynamics at a
glance—not just the final numbers, but how the experiment got there. A bolded
best-run line and a properly scaled y-axis make the story immediately readable,
even if some runs diverged.

This skill takes about 30 seconds. It's worth it.

## What you need

- **Baseline W&B run ID**: in the PR body under `## Baseline`, look for the `W&B run: \`xxxxxxxx\`` line.
- **Your own run IDs**: the 8-character W&B IDs of the runs you just completed. Find them in the W&B run URLs or in the training output (the run ID is printed at launch).
- **W&B credentials**: `WANDB_ENTITY` and `WANDB_PROJECT` env vars (already set in the pod environment).
- **W&B credentials**: `WANDB_ENTITY` and `WANDB_PROJECT` in the current environment.

## Step 1 — Run the script

Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ When helping a user onboard a target repository, inspect an explicitly configure
- operational guardrails such as commands, budgets, allowed edits, protected artifacts, and result-reporting expectations; and
- optional research avenues, papers, models, and libraries that provide useful starting points without forcing a narrow solution path.

Favor high-level goals and guardrails that let research agents discover the details. Avoid micromanaging methods or over-prompting one idea unless that narrow focus is the user's explicit goal. The [`bootstrap-target`](plugins/senpai/skills/bootstrap-target/SKILL.md) guide and its template can turn the confirmed decisions into the target contract.
Favor high-level goals and guardrails that let research agents discover the details. Avoid micromanaging methods or over-prompting one idea unless that narrow focus is the user's explicit goal. The [`bootstrap-target`](.agents/skills/bootstrap-target/SKILL.md) guide and its template can turn the confirmed decisions into the target contract.

Reference examples:

Expand Down
19 changes: 11 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,27 +303,30 @@ Children share the parent workspace, so their process and conversation are isola

## Task guides

OpenHands receives these as progressively disclosed skills; their bodies are loaded only when the task calls for them.
Live advisors and students load Senpai-owned skills only from
`plugins/senpai/skills`; target repositories may also supply project skills.
Guides under `.agents/skills` are for human users and Senpai developers and
are not installed into autoresearch pods.

### Core research workflow
### Live autoresearch

| Guide | Purpose |
|---|---|
| [Bootstrap a target](plugins/senpai/skills/bootstrap-target/SKILL.md) | Build `program.md` from a new ML repository. |
| [Assign an experiment](plugins/senpai/skills/assign-experiment/SKILL.md) | Turn a hypothesis into a typed student branch and draft PR. |
| [Delegate subagents](plugins/senpai/skills/delegate-subagents/SKILL.md) | Launch and coordinate bounded parallel research, review, and implementation help. |
| [Submit experiment results](plugins/senpai/skills/submit-experiment-results/SKILL.md) | Commit the tested implementation and publish a structured, evidence-backed result. |
| [Review an experiment](plugins/senpai/skills/review-experiment/SKILL.md) | Merge a reproducible winner, close a useful negative, or request the missing evidence. |
| [Handle human Issues](plugins/senpai/skills/check-human-issues/SKILL.md) | Respond to authenticated human-to-agent messages delivered through GitHub Issues. |
| [Senpai status check](plugins/senpai/skills/senpai-status-check/SKILL.md) | Produce a bounded, read-only GitHub, W&B, and local-controller status report. |
| [Exa search](plugins/senpai/skills/exa-search/SKILL.md) | Search the current web or scholarly publications with mode-specific defaults. |
| [AlphaXiv paper lookup](plugins/senpai/skills/alphaxiv-paper-lookup/SKILL.md) | Get a structured overview before reading a primary paper deeply. |
| [W&B and Weave](plugins/senpai/skills/wandb-primary/SKILL.md) | Inspect runs, metrics, artifacts, evaluations, and agent traces. |

### Evidence and research
### Human and developer guides

| Guide | Purpose |
|---|---|
| [Senpai status check](.agents/skills/senpai-status-check/SKILL.md) | Produce a bounded, read-only GitHub, W&B, and local-controller status report. |
| [Exa search](.agents/skills/exa-search/SKILL.md) | Search the current web or scholarly publications with mode-specific defaults. |
| [AlphaXiv paper lookup](.agents/skills/alphaxiv-paper-lookup/SKILL.md) | Get a structured overview before reading a primary paper deeply. |
| [W&B and Weave](.agents/skills/wandb-primary/SKILL.md) | Inspect runs, metrics, artifacts, evaluations, and agent traces. |
| [Bootstrap a target](.agents/skills/bootstrap-target/SKILL.md) | Build `program.md` from a new ML repository. |
| [Experiment report](.agents/skills/experiment-report/SKILL.md) | Create the project-standard `nn_cfd` W&B comparison report; this guide is target-specific rather than part of the generic runtime. |
| [Training code style](literature_and_guidance/TRAINING-CODE-STYLE.md) | Structure expensive ML entrypoints so configuration, artifacts, validation, and failure boundaries stay explicit. |

Expand Down
8 changes: 6 additions & 2 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,10 @@ suffix on every inference, and current time is rendered for every controller
wake. Operators must start fresh role state to apply a changed identity,
program, or role charter.

File-based subagents are discovered from `.agents/agents`. Skill bodies are not
File-based subagents are discovered from `.agents/agents`. Live advisor and
student skills come only from `plugins/senpai/skills`; `.agents/skills` is for
human operators and Senpai developers and is not installed into pods. Target
repositories may still supply their own project skills. Skill bodies are not
concatenated into agent definitions. The OpenHands fork's `main` branch applies
each agent definition's `reasoning_effort` override after resolving its
inherited LLM or stored model profile.
Expand Down Expand Up @@ -687,7 +690,8 @@ Removed:

Retained intentionally:

- Agent skills and their model/effort metadata under `.agents`;
- runtime skills and their model/effort metadata in the Senpai plugin;
- human and developer guides under `.agents/skills`, outside pod context;
- OpenHands Browser, task tracker, Think, and the high-quality default
condenser for providers not using stored OpenAI Responses continuation or
Anthropic native compaction;
Expand Down
2 changes: 1 addition & 1 deletion k8s/entrypoint-advisor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ ls \
"$HOME/.agents/agents/general-purpose.md" \
"$HOME/.agents/agents/explore.md" \
"$HOME/.agents/agents/search.md" \
"$HOME/.agents/skills/wandb-primary/SKILL.md"
"$SENPAI_PLUGIN/skills/wandb-primary/SKILL.md"

# --- Hivemind is intentionally disabled pending its OpenHands rewrite. ---
# source "$WORKDIR/k8s/start-hivemind.sh"
Expand Down
2 changes: 1 addition & 1 deletion k8s/entrypoint-student.sh
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ ls \
"$HOME/.agents/agents/general-purpose.md" \
"$HOME/.agents/agents/explore.md" \
"$HOME/.agents/agents/search.md" \
"$HOME/.agents/skills/wandb-primary/SKILL.md"
"$SENPAI_PLUGIN/skills/wandb-primary/SKILL.md"

# --- Hivemind is intentionally disabled pending its OpenHands rewrite. ---
# source "$WORKDIR/k8s/start-hivemind.sh"
Expand Down
11 changes: 7 additions & 4 deletions plugins/senpai/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,10 @@ no MCP server. The Python runtime binds the authenticated role and adds the
canonical `ADVISOR:` or `STUDENT:` prefix to Senpai-authored GitHub comments;
tool payloads contain only the unprefixed message text.

Keep Senpai-owned workflow skills here rather than relying on a provider's user
skill directory. Never commit secret values. The plugin remains the source of
truth for reusable workflow guidance, while Python remains the source of truth
for verified state changes.
Keep every Senpai-owned skill used by a live advisor or student here rather
than relying on a provider's user skill directory. Target repositories may
supply project skills separately. Human onboarding and developer guides stay
under the runner's `.agents/skills` and are not installed into pods. Never
commit secret values. The plugin remains the source of truth for reusable
runtime guidance, while Python remains the source of truth for verified state
changes.
11 changes: 3 additions & 8 deletions plugins/senpai/scripts/agent-context.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,10 @@ install_senpai_agent_context() {
local runtime_root="$3"
local runtime_plugin="$runtime_root/plugin"

mkdir -p "$HOME/.agents/skills"
cp -a "$workdir/.agents/." "$HOME/.agents/"
for marker in "$HOME"/.agents/skills/*/.senpai-developer-only; do
[ -e "$marker" ] || continue
rm -rf -- "${marker%/.senpai-developer-only}"
done
mkdir -p "$HOME/.agents/agents"
cp -a "$workdir/.agents/agents/." "$HOME/.agents/agents/"
cp -a "$source_plugin" "$runtime_plugin"
"$SENPAI_PYTHON" -m senpai_agent.agent_markdown \
"$HOME/.agents" "$runtime_plugin"
cp -a "$runtime_plugin/skills/." "$HOME/.agents/skills/"
"$HOME/.agents/agents" "$runtime_plugin"
printf '%s\n' "$runtime_plugin"
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ official `exa_py` client, loads the nearest `.env` through `python-dotenv`, and
preserves an `EXA_API_KEY` already set in the environment.

```bash
python "$HOME/.agents/skills/exa-search/scripts/search_exa.py" \
python "$SENPAI_PLUGIN/skills/exa-search/scripts/search_exa.py" \
general-web \
"current OpenHands SDK file-based agent documentation"
```

```bash
python "$HOME/.agents/skills/exa-search/scripts/search_exa.py" \
python "$SENPAI_PLUGIN/skills/exa-search/scripts/search_exa.py" \
research-publications \
"uncertainty calibration for neural networks"
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,13 @@ This skill covers everything an agent needs to work with Weights & Biases:

| I need to... | Use |
|---|---|
| Query training runs, loss curves, hyperparameters | **W&B SDK** (`wandb.Api()`) — see `references/WANDB_SDK.md` |
| Query GenAI traces, calls, evaluations | **Weave SDK** (`weave.init()`, `client.get_calls()`) — see `references/WEAVE_SDK.md` |
| Query training runs, loss curves, hyperparameters | **W&B SDK** (`wandb.Api()`) — see `$SENPAI_PLUGIN/skills/wandb-primary/references/WANDB_SDK.md` |
| Query GenAI traces, calls, evaluations | **Weave SDK** (`weave.init()`, `client.get_calls()`) — see `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md` |
| Convert Weave wrapper types to plain Python | **`weave_helpers.unwrap()`** |
| Build a DataFrame from training runs | **`wandb_helpers.runs_to_dataframe()`** |
| Extract eval results for analysis | **`weave_helpers.eval_results_to_dicts()`** |
| Need low-level Weave filtering (CallsFilter, Query) | **Raw Weave SDK** (`weave.init()`, `client.get_calls()`) — see `references/WEAVE_SDK.md` |
| Judge curve shape (spikes, smoothness, slope, overfit) | **`training_diagnostics` + `curve_plots`** — use the workflow below, then load `references/TRAINING_DIAGNOSTICS.md` for the heuristics |
| Need low-level Weave filtering (CallsFilter, Query) | **Raw Weave SDK** (`weave.init()`, `client.get_calls()`) — see `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md` |
| Judge curve shape (spikes, smoothness, slope, overfit) | **`training_diagnostics` + `curve_plots`** — use the workflow below, then load `$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md` for the heuristics |

---

Expand All @@ -74,8 +74,9 @@ This skill covers everything an agent needs to work with Weights & Biases:
### Helper libraries

```python
import os
import sys
sys.path.insert(0, ".agents/skills/wandb-primary/scripts")
sys.path.insert(0, f"{os.environ['SENPAI_PLUGIN']}/skills/wandb-primary/scripts")

# Weave helpers (traces, evals, GenAI)
from weave_helpers import (
Expand Down Expand Up @@ -125,9 +126,9 @@ from curve_plots import (

Read these as needed — they contain full API surfaces and recipes:

- **`references/WEAVE_SDK.md`** — Weave SDK for GenAI traces (`client.get_calls()`, `CallsFilter`, `Query`, stats). Start here for Weave queries.
- **`references/WANDB_SDK.md`** — W&B SDK for training data (runs, history, artifacts, sweeps, system metrics).
- **`references/TRAINING_DIAGNOSTICS.md`** — reference heuristics for reading loss / LR / grad-norm / grad-histogram charts. Load this when you are actively interpreting training curves.
- **`$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md`** — Weave SDK for GenAI traces (`client.get_calls()`, `CallsFilter`, `Query`, stats). Start here for Weave queries.
- **`$SENPAI_PLUGIN/skills/wandb-primary/references/WANDB_SDK.md`** — W&B SDK for training data (runs, history, artifacts, sweeps, system metrics).
- **`$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md`** — reference heuristics for reading loss / LR / grad-norm / grad-histogram charts. Load this when you are actively interpreting training curves.

---

Expand Down Expand Up @@ -216,7 +217,7 @@ df = pd.DataFrame(rows)
print(df.describe())
```

For full W&B SDK reference (filters, history, artifacts, sweeps), read `references/WANDB_SDK.md`.
For full W&B SDK reference (filters, history, artifacts, sweeps), read `$SENPAI_PLUGIN/skills/wandb-primary/references/WANDB_SDK.md`.

### Weave — SDK

Expand All @@ -226,7 +227,7 @@ client = weave.init(f"{entity}/{project}") # positional string, NOT keyword arg
calls = client.get_calls(limit=10)
```

For raw SDK patterns (CallsFilter, Query, advanced filtering), read `references/WEAVE_SDK.md`.
For raw SDK patterns (CallsFilter, Query, advanced filtering), read `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md`.

---

Expand Down Expand Up @@ -307,7 +308,7 @@ For structured failure analysis on eval results:
3. **Axial coding** — write a second Scorer that classifies notes into a taxonomy
4. **Summarize** — count primary labels with `collections.Counter`

See `references/WEAVE_SDK.md` for the full SDK reference.
See `$SENPAI_PLUGIN/skills/wandb-primary/references/WEAVE_SDK.md` for the full SDK reference.

### W&B Reports

Expand Down Expand Up @@ -344,7 +345,7 @@ Keep the inline workflow short and load detail on demand:
1. Confirm `step_key` before doing any curve work. Never assume `_step`.
2. Compute features with the bundled helpers instead of hand-rolling spike or slope logic.
3. Render PNGs and inspect them visually.
4. Load `references/TRAINING_DIAGNOSTICS.md` while you interpret the results.
4. Load `$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md` while you interpret the results.
5. End with a verdict, evidence tied to step ranges, and concrete next actions.

### Required sequence
Expand All @@ -370,7 +371,7 @@ Next actions:
- <concrete hyperparameter, logging, or code change>
```

Load `references/TRAINING_DIAGNOSTICS.md` for the interpretation heuristics, especially when the numbers and the image disagree.
Load `$SENPAI_PLUGIN/skills/wandb-primary/references/TRAINING_DIAGNOSTICS.md` for the interpretation heuristics, especially when the numbers and the image disagree.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@
run data into pandas-friendly structures for analysis.

Usage (in sandbox):
import os
import sys
sys.path.insert(0, ".agents/skills/wandb-primary/scripts")
sys.path.insert(0, f"{os.environ['SENPAI_PLUGIN']}/skills/wandb-primary/scripts")
from wandb_helpers import (
runs_to_dataframe, # Convert runs to a clean pandas DataFrame
diagnose_run, # Quick diagnostic summary of a training run
Expand Down
Loading
Loading