diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 472f0ef..5cebd0f 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -17,8 +17,8 @@ { "name": "aviator", "source": "./aviator", - "description": "Verify and Runbooks workflow automation with Aviator MCP", - "version": "1.2.0" + "description": "Verify and Runbooks workflow automation via the aviator CLI", + "version": "1.3.0" } ] } diff --git a/README.md b/README.md index 2005124..b82383d 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ run `av pr --all` to create PRs for the entire stack. ### aviator -Connects Claude Code to Aviator via MCP for spec submission to [Runbooks](https://aviator.co/runbooks) and Verify, to automate development workflows. +Submits specs from Claude Code to Aviator — [Runbooks](https://aviator.co/runbooks) and Verify — through the `aviator` CLI, to automate development workflows. **Verify vs Runbooks.** Aviator has two ways to hand a spec off from your Claude session: @@ -81,10 +81,9 @@ Connects Claude Code to Aviator via MCP for spec submission to [Runbooks](https: **What this plugin does:** -- Connects to the Aviator MCP server for spec submission and runbook operations -- Submits Verify specs and creates runbooks from your current Claude session context -- Handles OAuth authentication automatically -- Provides access to Aviator's workflow automation tools +- Guides Claude through writing the intent, spec, and acceptance criteria from your current session context +- Submits Verify specs and creates runbooks via the `aviator` CLI +- Keeps acceptance criteria fresh as a connected PR evolves **Usage:** @@ -95,17 +94,12 @@ Connects Claude Code to Aviator via MCP for spec submission to [Runbooks](https: - An Aviator account at https://app.aviator.co - Repository connected to Aviator +- The `aviator` CLI installed (`go install github.com/aviator-co/aviator-cli/cmd/aviator@latest`) and configured with an API token (`AVIATOR_API_TOKEN` or `~/.config/aviator/config.yaml`) -**Self-hosted / On-prem:** The plugin connects to `https://app.aviator.co/mcp` by default. To point it at a self-hosted instance, set the `AVIATOR_MCP_URL` environment variable: +**Self-hosted / On-prem:** The CLI talks to `https://api.aviator.co` by default. To point it at a self-hosted instance, set `AVIATOR_API_HOST` (or `apiHost` in `~/.config/aviator/config.yaml`): ```bash -export AVIATOR_MCP_URL=https://aviator.your-company.com/mcp -``` - -Alternatively, you can configure the MCP server manually: - -```bash -claude mcp add --transport http aviator https://aviator.your-company.com/mcp +export AVIATOR_API_HOST=https://aviator.your-company.com ``` ## Installation diff --git a/aviator/.mcp.json b/aviator/.mcp.json deleted file mode 100644 index 72d6be7..0000000 --- a/aviator/.mcp.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "mcpServers": { - "aviator": { - "type": "http", - "url": "${AVIATOR_MCP_URL:-https://app.aviator.co/mcp}" - } - } -} diff --git a/aviator/commands/create-runbook.md b/aviator/commands/create-runbook.md index 7a858f5..6a67993 100644 --- a/aviator/commands/create-runbook.md +++ b/aviator/commands/create-runbook.md @@ -8,7 +8,7 @@ Create an Aviator Runbook from the current Claude Code session context. **Aviato > Writing the code yourself and just want Aviator to verify it against intent + acceptance criteria? Use `/verify-submit` instead — it captures intent and AC with no implementation steps. -**Load the `spec-submission` skill** (Skill tool → `aviator:spec-submission`) before you start — it carries the shared mechanics this flow relies on: how the message reads, the Acceptance Criteria review loop, the `specSubmit` call, and the PR directive. This command file only covers what's specific to a Runbook. +**Load the `spec-submission` skill** (Skill tool → `aviator:spec-submission`) before you start — it carries the shared mechanics this flow relies on: how the message reads, the Acceptance Criteria review loop, the `aviator` CLI submission, and the PR directive. This command file only covers what's specific to a Runbook. ## Arguments @@ -26,7 +26,7 @@ If a plan file exists from plan mode (check the plan file path mentioned in the Similarly, if a spec file already exists in the conversation — one the user wrote, one generated earlier, or one provided via `$ARGUMENTS` — use it as-is. Do not restructure, reformat, or rewrite an existing spec. When the spec comes from a file, preserve the original filename. -If no existing spec is available, generate one. Keep it **free-form** — there's no required structure or fixed set of sections. Write whatever best conveys the change to the agent that will implement it: the intent and the implementation approach or steps, shaped to the task rather than forced into headings. (The acceptance criteria are passed separately as the `acceptance_criteria` argument — you don't hand-embed them; the backend folds them into the spec the agent works from.) +If no existing spec is available, generate one. Keep it **free-form** — there's no required structure or fixed set of sections. Write whatever best conveys the change to the agent that will implement it: the intent and the implementation approach or steps, shaped to the task rather than forced into headings. (The acceptance criteria are passed separately via `--criteria`/`--criteria-file` — you don't hand-embed them; the backend folds them into the spec the agent works from.) ## Step 2: Review with the user @@ -40,14 +40,23 @@ A simple "yes" or "go ahead" is enough to submit. ## Step 3: Create Runbook -Submit via the `specSubmit` call described in the `spec-submission` skill. For a Runbook: +Submit with `aviator runbook`, following the CLI mechanics in the `spec-submission` skill (preflight, repo derivation, criteria-file guidance, result parsing). What's specific to a Runbook: -- `submission_type`: **`"runbook"`** — the agent writes the code from the spec. This is the default, but pass it explicitly. -- `acceptance_criteria`: the confirmed AC as a JSON array of strings (optional but recommended — they're folded into the spec the agent works from). -- `spec_files`: include only if a spec was generated; always a single file. -- `target_branch`: **optional** — the base branch the runbook builds on and checks out; the generated PR opens against it. Omit for the repo default (trunk). (Runbook mode generates its own PR, so there's no `working_branch` to connect here.) +- `--intent`: **required** — the confirmed intent: what the runbook should accomplish and why. Keep it short and human-friendly; the implementation detail travels in the spec, and the intent is stored and displayed on the session as-is. +- `--spec` (optional): the spec file — include only if one was generated or already existed; always a single file. +- `--criteria` / `--criteria-file` (optional but recommended): the confirmed AC. The backend folds them into the spec the agent works from. Prefer `--criteria-file` for more than 2–3. Make sure the spec itself carries no "Acceptance Criteria" section when you pass these — the backend rejects that combination rather than guess which list wins. +- `--target-branch` (optional): the base branch the runbook builds on and checks out; the generated PR opens against it. Omit for the repo default (trunk). (Runbook mode generates its own PR, so there's no working branch to connect here.) +- `--title` (optional but worth setting): a short deliberate title for the runbook. Left unset, the backend derives one from the intent — currently by truncation, which reads poorly for multi-sentence intents. -Then return the Runbook URL and set the PR directive, both per the `spec-submission` skill. The expected PR body shape: +```bash +aviator runbook \ + --repo acme/web \ + --intent "Migrate the settings page to the new design system" \ + --spec /path/to/spec.md \ + --criteria-file /path/to/criteria.txt +``` + +On success the command prints `✓ Runbook created: ` and a `Runbook #` line. Then return the Runbook URL and set the PR directive, both per the `spec-submission` skill. The expected PR body shape: ``` Runbook: diff --git a/aviator/commands/verify-submit.md b/aviator/commands/verify-submit.md index a2d4e5c..1cffb03 100644 --- a/aviator/commands/verify-submit.md +++ b/aviator/commands/verify-submit.md @@ -10,7 +10,7 @@ Submit a Verify spec to Aviator from the current Claude Code session. Aviator Ve - **Key decisions & architecture** — a free-form record of the decisions made and the shape of the change, written so a reviewer can understand the PR without reading every line. - **Acceptance Criteria** — the concrete, observable behaviors the change must satisfy, verified independently against the code. -**Load the `spec-submission` skill** (Skill tool → `aviator:spec-submission`) before you start — it carries the shared mechanics this flow relies on: how the message reads, the Acceptance Criteria review loop, the `specSubmit` call, and the PR directive. This command file only covers what's specific to Verify. +**Load the `spec-submission` skill** (Skill tool → `aviator:spec-submission`) before you start — it carries the shared mechanics this flow relies on: how the message reads, the Acceptance Criteria review loop, the `aviator` CLI submission, and the PR directive. This command file only covers what's specific to Verify. ## Arguments @@ -20,8 +20,8 @@ $ARGUMENTS - Optional additional context or instructions for the Verify submissi The code is the ground truth for a Verify submission. Before generating anything: -- Identify the **working branch** — the branch the in-flight work lives on (typically the current git branch). You'll pass this as `working_branch` so Verify tracks the PR opened from it. -- Identify the **repository** in `owner/repo` form (e.g. from `git remote get-url origin`) — you'll pass this as `repo_name`. +- Identify the **working branch** — the branch the in-flight work lives on (typically the current git branch). You'll pass this as `--working-branch` so Verify tracks the PR opened from it. +- Identify the **repository** in `owner/repo` form, following the repo-derivation procedure in the `spec-submission` skill (don't just read `origin`) — you'll pass this as `--repo`. - Read the **actual current changes** end-to-end (the diff against the base branch, and the modified files in full — not just the hunks). Understand what the code does: what behavior each change introduces, what invariants it preserves, what it exposes, what failure modes it handles, what it replaces. Everything below is drawn from what the code actually does, cross-checked against `$ARGUMENTS` and any spec/plan already in the session — never from imagination. @@ -47,7 +47,7 @@ Aim for the altitude of "what a thoughtful reviewer needs to not be surprised, a ### Assembling the spec file -Generate a single spec file (name it `spec.md`, or preserve the original filename if a spec already exists in the session — use it as-is, don't restructure it). The spec body is **intent + key decisions** — the acceptance criteria are **not** in the spec; they're passed as the `acceptance_criteria` argument at submit. Use these sections: +Generate a single spec file (name it `spec.md`, or preserve the original filename if a spec already exists in the session — use it as-is, don't restructure it). The spec body is **intent + key decisions** — the acceptance criteria are **not** in the spec; they're passed through the `--criteria`/`--criteria-file` flags at submit. Use these sections: ``` ## Intent @@ -61,29 +61,40 @@ Intent always belongs. Include Key Decisions & Architecture whenever the change ## Step 3: Review Acceptance Criteria with the user -Run the Acceptance Criteria review loop from the `spec-submission` skill — iterate until the user explicitly confirms. +Run the Acceptance Criteria review loop from the `spec-submission` skill — iterate until the user explicitly confirms. (Running non-interactively with no user available? Use that skill's non-interactive provision instead of stalling.) One thing specific to Verify: show the user the **intent** line and the **Acceptance Criteria** — not the full spec body (Key Decisions & Architecture is submitted as supporting context, not what the user confirms). If they want to see the rest, they'll ask. ## Step 4: Submit for Verify -Submit via the `specSubmit` call described in the `spec-submission` skill. For Verify: - -- `submission_type`: **`"verify"`** — this is the argument that makes it a Verify submission (intent + AC over human-authored code, no step generation). Do not omit it; the default is `"runbook"`. -- `acceptance_criteria`: **required** — the confirmed AC as a JSON array of strings. -- `working_branch`: **required** — the branch the in-flight work lives on (from Step 1), passed by name, so Verify tracks the PR you open from that branch. +Submit with `aviator verify`, following the CLI mechanics in the `spec-submission` skill (preflight, repo derivation, criteria-file guidance, result parsing). What's specific to Verify: + +- `--intent`: **required** — the confirmed intent. +- `--criteria` / `--criteria-file`: **required** — the confirmed AC (`aviator verify` seeds its structured criteria set from these). Prefer `--criteria-file` for more than 2–3. +- `--working-branch`: **required for this flow** — the branch the in-flight work lives on (from Step 1), passed by name, so Verify tracks the PR you open from that branch. (The CLI marks the flag optional; a Verify submission still needs it, since without it no PR ever binds to the session.) +- `--spec` (optional): the spec file (intent + key decisions) from Step 2. +- `--target-branch` (optional): the base branch to verify against; omit for the repo default. + +```bash +aviator verify \ + --repo acme/web \ + --intent "Gate the new banner behind the beta flag" \ + --criteria-file /path/to/criteria.txt \ + --working-branch feature/banner \ + --spec /path/to/spec.md +``` -Then return the Runbook URL and set the PR directive, both per the `spec-submission` skill. +On success the command prints `✓ Verify submission created: ` and a `Runbook #` line. Then return the Runbook URL and set the PR directive, both per the `spec-submission` skill. ## Step 5: Keep Acceptance Criteria fresh as the PR evolves Verify AC are a living contract, not a one-time snapshot. As you keep pushing commits to the connected PR, the code drifts from the AC the user originally signed off on — new behavior appears, scope shifts, an edge case gets handled differently. **Stale AC verify the wrong thing.** -So, after a meaningful push to the connected PR in this session (a new behavior, a changed contract, a dropped or added piece of scope — not a typo fix): +So, after a meaningful change to the work on this branch — pushed or still local — in this session (a new behavior, a changed contract, a dropped or added piece of scope — not a typo fix): -1. Re-read the current AC and the runbook's version: `getRunbook(url, fields=['acceptance_criteria'])` — note the returned `runbook_version` (an int). +1. Re-read the current AC and the runbook's version: `aviator results r/ --json` — note the `runbook_version` field in the output (an int). (`aviator show r/ --json` returns the full session; `results` is the lighter call.) 2. Compare the AC against the **current** diff. If the code now does something the AC don't cover, or an AC no longer matches what the code does, the AC are stale. -3. Refresh them: `editRunbook(runbook_url, expected_version=, payload={"acceptance_criteria": []})`. The payload is the COMPLETE new list — including unchanged items — and expresses add/update/remove/reorder in one atomic edit. If the edit fails with a stale-version error, someone else moved the runbook; re-read and retry. +3. Refresh them with `aviator edit r/ --expected-version --criteria-file ` (or repeated `--criteria` flags). The edit **replaces the entire criteria list**, so the file must hold the COMPLETE new list — including unchanged items, in order — expressing add/update/remove/reorder in one atomic edit. If it fails with a stale-version error (409), someone else moved the runbook; re-read the version and retry — a stale edit writes nothing. 4. Keep the same quality bar as Step 2 — observable outcomes, no implementation detail — and keep the user in the loop on non-trivial AC changes rather than silently rewriting their signed-off list. -Do not re-run `specSubmit` to refresh AC — that creates a new runbook. Use `editRunbook` to update the existing one. +Do not re-run `aviator verify` to refresh AC — that creates a new runbook. Use `aviator edit` to update the existing one. diff --git a/aviator/skills/spec-submission/SKILL.md b/aviator/skills/spec-submission/SKILL.md index 224b2b4..a34210c 100644 --- a/aviator/skills/spec-submission/SKILL.md +++ b/aviator/skills/spec-submission/SKILL.md @@ -1,15 +1,15 @@ --- name: spec-submission -description: Shared submission mechanics for Aviator spec commands — how to write the intent, run the acceptance-criteria review loop with the user, call the specSubmit MCP tool, set the Runbook-URL PR directive, and handle errors. Load when running /verify-submit or /create-runbook; the command supplies the flow-specific spec shape and submission_type, this skill supplies everything both flows do identically. +description: Shared submission mechanics for Aviator spec commands — how to write the intent, run the acceptance-criteria review loop with the user, submit via the aviator CLI, set the Runbook-URL PR directive, and handle errors. Load when running /verify-submit or /create-runbook; the command supplies the flow-specific spec shape and which CLI command to run, this skill supplies everything both flows do identically. --- # Spec submission — shared mechanics -This is the machinery shared by `/verify-submit` and `/create-runbook`. Your command file provides what differs between the two flows — the spec sections to write and the `submission_type` to send. Everything below is identical across both flows: how the intent reads, how you align the Acceptance Criteria with the user, how you call `specSubmit`, and what happens to any PR opened afterward. +This is the machinery shared by `/verify-submit` and `/create-runbook`. Your command file provides what differs between the two flows — the spec sections to write and which `aviator` CLI command to run (`aviator verify` vs `aviator runbook`). Everything below is identical across both flows: how the intent reads, how you align the Acceptance Criteria with the user, how you submit through the CLI, and what happens to any PR opened afterward. ## The intent -A short, human-friendly description of what this change accomplishes and why — written the way a person would describe it to a colleague filing a ticket. A few sentences at most. No markdown structure, no file paths, no code details. This is the `intent` argument to `specSubmit`. +A short, human-friendly description of what this change accomplishes and why — written the way a person would describe it to a colleague filing a ticket. A few sentences at most. No markdown structure, no file paths, no code details. This is the `--intent` flag on both commands. It's stored verbatim on the session and displayed in Aviator as the session's intent — the words you write here are the face of the submission, so hold the quality bar even when the rest of the flow carries a detailed spec. If the user provided `$ARGUMENTS`, lean on their words — echo their intent rather than rephrasing it technically. @@ -21,7 +21,7 @@ Bad (too technical — that belongs in the spec): ## Acceptance Criteria are the primary output -The Acceptance Criteria (AC) are the highest-value artifact of the submission — prioritize their quality over the length or polish of the spec body. Sharp AC with a thin spec beat a lush spec with generic AC. AC are submitted as their own `acceptance_criteria` argument (below), not embedded in the spec. +The Acceptance Criteria (AC) are the highest-value artifact of the submission — prioritize their quality over the length or polish of the spec body. Sharp AC with a thin spec beat a lush spec with generic AC. AC are submitted through their own `--criteria`/`--criteria-file` flags (below), not embedded in the spec. **Before writing or reviewing any AC, load the `acceptance-criteria` skill** (Skill tool → `aviator:acceptance-criteria`) and apply its rulebook in full — it defines what makes an AC valid, the two readers each AC must serve, the north-star test, which sources to draw from, and the anti-patterns to avoid. This is a blocking step. @@ -37,27 +37,51 @@ Before submitting, get the user aligned on the AC. (Your command says what else, - **Repeat until the user explicitly confirms.** A simple "yes" or "go ahead" is enough. Do not submit on silence or an implied yes. - **If invoked non-interactively** (no user available to confirm — e.g. an automated or orchestrated run), treat the generated AC as pre-confirmed and note in your output that the confirmation step was skipped. +## Preflight — the `aviator` CLI must be installed and configured + +Submission goes through the `aviator` CLI. Before submitting, confirm it's available: + +- **Check it's installed:** `command -v aviator`. If it's missing, tell the user to install it and stop — don't attempt a workaround: + + ```bash + go install github.com/aviator-co/aviator-cli/cmd/aviator@latest + ``` + +- **Check it's current:** run the `--help` of the command you're about to use and confirm the flags you need are listed — in particular, `aviator runbook --help` must show `--spec` and `--criteria-file`. If they're missing, the installed CLI predates spec/criteria support on runbooks — tell the user to upgrade and stop. (`aviator verify` gained these flags earlier, so their presence there does not imply support on `runbook`.) +- **Check it's configured:** the CLI needs an API token, via the `AVIATOR_API_TOKEN` environment variable or `~/.config/aviator/config.yaml` (with an optional `AVIATOR_API_HOST` / `apiHost` override for on-prem). If a submit fails with an auth/config error, point the user at these — don't try to work around missing credentials. + ## Locking and submitting **Only submit after the user has explicitly confirmed in the review step.** -**Pass the confirmed AC as the `acceptance_criteria` argument below — do not embed them in the spec markdown.** They're a first-class input; the spec carries intent and supporting context, not the AC. +**Pass the confirmed AC through the `--criteria`/`--criteria-file` flags below — do not embed them in the spec markdown.** They're a first-class input; the spec carries intent and supporting context, not the AC. + +Assemble the invocation your command specifies (`aviator verify` or `aviator runbook`) from these inputs, shared across both flows: + +- **Repo** (`--repo`): the canonical `owner/repo` the PR will target. Getting this wrong is silent — a wrong-but-well-formed name is accepted and binds the submission to a repo no PR will ever link back to — so derive it in two steps: + 1. **Pick the remote PRs are opened against.** `git remote -v`; with one remote, that's it. With several, don't assume `origin`, and don't rely on the working branch's upstream — a fresh branch hasn't been pushed yet and has none. Look at where the repo's existing PRs actually target (`gh pr list --limit 3` on the candidates) or what recent work branches track; a personal fork loses to the org repo. If the evidence genuinely splits across two *different* repos, ask the user; running non-interactively, pick the org repo and flag the choice in your output. + 2. **Canonicalize the pick through GitHub:** `gh api repos// --jq .full_name` and pass exactly the `full_name` returned. Renamed repos redirect silently, so two remote URLs can be one repo under an old and new name — and Aviator records the stale and current names as *different* repos, accepting the stale one without complaint. +- **Spec** (`--spec`): write the spec content to a file and pass its path. If the spec already came from a file on disk, pass that file directly; otherwise write it to a temp path (e.g. under the scratchpad) and pass that. Always a single file. +- **Criteria** (`--criteria` / `--criteria-file`): the exact AC the user signed off on. `--criteria` is repeatable, but for more than 2–3 criteria prefer `--criteria-file ` — write one criterion per line to a file (same temp location as the spec) — to avoid shell-quoting issues with special characters. The two flags are mutually exclusive; pick one. +- **Branch flags**: flow-specific — your command says which to pass (`--working-branch` for Verify, `--target-branch` where the work is stacked). See your command file. + +Run the command. On success it prints a confirmation to stdout — the first two lines are stable, and more detail lines (branches, criteria count, status) may follow: -Then call the `specSubmit` MCP tool from the Aviator server with: +``` +✓ Verify submission created: https://app.aviator.co/r/42 + Runbook #42 + Working branch: feature/banner + Target branch: main + Criteria: 4 +``` -- `repo_name`: the repository in `owner/repo` format (derive it from the git remote, e.g. `git remote get-url origin`). -- `submission_type`: **provided by your command** — `"verify"` for `/verify-submit`, `"runbook"` for `/create-runbook`. Pass it explicitly. -- `intent`: the confirmed intent (see "The intent" above). -- `acceptance_criteria`: the exact AC the user signed off on, as a JSON array of strings, e.g. `["First criterion","Second criterion"]`. **Required for Verify** (seeds the structured criteria set); optional for a Runbook (folded into the spec the agent works from). -- `spec_files`: `[{"filename": "", "content": "..."}]` — always a single file. -- `working_branch`: **Verify only** — an existing branch, passed by name; a PR opened from it auto-connects back to this submission (no need to push first — the link is by name). Runbook mode omits it and uses `target_branch`, since the runbook generates its own PR. -- `target_branch` (optional): the branch this work is built on top of — omit for the repo default (trunk); pass the parent branch when this work is stacked on another in-flight branch. +Parse the URL and the `Runbook #` number from that output. The URL's host is the Aviator app the backend is configured with — don't expect it to match `AVIATOR_API_HOST`. Treat the URL as the canonical **Runbook URL** for this session, and refer to the session as `r/` (e.g. `r/42`) — that's the ID form every follow-up command takes: `aviator show r/42`, `aviator results r/42`, `aviator edit r/42`. (They also accept a bare number or the full URL.) Hold both for the AC-freshness loop and any PR opened later in the same session. -The tool returns the runbook URL. Treat it as the canonical **Runbook URL** for this session — hold it for any PR opened later in the same session. +One timing note for runbook submissions: `aviator show` returns a 400 "Runbook hasn't been generated yet" until step generation completes (it can take a few minutes). Right after submitting, that's expected — not a failed submission; retry later rather than treating it as an error. ## Return the link and set the PR directive -Give the user the Runbook URL from the tool response and a brief summary of what was submitted. +Give the user the Runbook URL from the command's output and a brief summary of what was submitted. Then, **when opening a PR for this work later in the same session**, the PR body **MUST** begin with `Runbook: ` on its own line, followed by a blank line, then the rest of the description. This applies to `gh pr create`, `av pr`, or any equivalent. @@ -69,6 +93,6 @@ Then, **when opening a PR for this work later in the same session**, the PR body ## Error handling -- If authentication is required, Claude Code will automatically open a browser for OAuth login. +- If the command fails with an authentication or configuration error, the CLI is missing a valid API token — point the user at `AVIATOR_API_TOKEN` or `~/.config/aviator/config.yaml` (see Preflight). Don't retry blindly or work around it. - If the repository is not found in Aviator, suggest connecting it in the Aviator dashboard under GitHub settings. -- If the API returns an error about credits, inform the user they may need to add runbook credits in their Aviator dashboard. +- If the command reports an error about credits, inform the user they may need to add runbook credits in their Aviator dashboard.