Skip to content

research-step: tighten workflows and schemas, add autods-to-theorizer plan template #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
charliemcgrady wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

research-step: tighten workflows and schemas, add autods-to-theorizer plan template #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 155 additions & 2 deletions plugins/asta-preview/skills/research-step/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,160 @@ Installing `bd` and `jq`, running `bd init`, and verifying `scripts/summary-chec
| `mission.md` | Input. The research task. |
| `.beads/` | Source of truth for state. |
| `summary.md` | Derived view of the session, regenerated by **update-summary**. Beads is the source of truth; this file is just a digest for humans and for **brainstorm**. Frontmatter `beads_snapshot` records the state it was rendered from. |
| `background_knowledge.txt` | Optional. Long-form context referenced from issue metadata via `summary_path`. |
| `.asta/<task_type>/<slug>/` | One **work_dir** per task. The agent picks the kebab-case slug at task creation; the full path is stored at `metadata.research_step.work_dir`. Every file the task writes goes here, following the per-task-type filename conventions in `assets/schemas.yaml > file_conventions`. |

## Per-task work_dir convention

Each task issue declares one working directory:

```
metadata.research_step.work_dir = ".asta/<task_type>/<slug>/"
```

- `<task_type>` is the schema task type (e.g. `literature_review`).
- `<slug>` is a short kebab-case name the agent picks at create-time
(e.g. `provenance`, `theme-a-aar-geometry`, `cross-dataset`).
- Every file the task writes lives **under** this directory. Sibling
directories (`.asta/literature_review/theme-b-ela-bands/`) are independent.
- The filenames inside follow `assets/schemas.yaml > file_conventions`
(e.g. lit_review always writes `summary.md`; synthesis writes
`report.md`) so downstream consumers can find them without reading
paths out of metadata.

The Asta Flows UI renders this directory as a file browser on the task's
**Files** tab. No `_path` fields are required; the directory IS the manifest.

## Markdown convention: inline citations

Any markdown file a task writes inline-links to the artifacts that back its
statements. The Asta Flows panel auto-linkifies bd ids, AutoDS node ids, and
run-relative paths; explicit Markdown links handle the rest.

Use real markdown links for every:

- **Paper mention** → `[Author+YY](https://www.semanticscholar.org/paper/<corpus_id>)`.
Never write a bare `corpusId 12345` or `doi:10.1234/...` in prose.
- **Numeric claim or verdict** → link to the script/log/table that produced
it, e.g. `[r=+0.22, p<1e-30](replicate.py)` (relative to the task's own
work_dir) or `[replicated](../../replication/<slug>/per-finding-table.md#node_3_2)`
(relative path into another task's work_dir).
- **Cross-reference to another task's output** → relative path from this
task's work_dir into the upstream task's work_dir (e.g.
`../../literature_review/theme-a-aar-geometry/summary.md`).
- **Cross-reference to another bd task** → bare bd id; the renderer
auto-linkifies it.
- **AutoDS hypothesis / experiment** → `../../autods_run/<slug>/experiments.json`.

Per-template tweaks live in `workflows/plans/<name>.md`.

## Voice for titles and descriptions

Every issue in beads must have a `--title` and a non-empty `--description`.
The Asta Flows UI surfaces both as the primary content of each task.

**Imagine the reader.** A smart college graduate scrolling the Tasks page
who has never heard of this run, this domain, or these acronyms. They are
not stupid — they have a degree — but they have zero context. Your titles
and descriptions are how they figure out what this run is doing.

### Titles — imperative phrase, ≤ 6 words

Lead with a verb and name the **specific subject** of the step in plain
English — don't stop at a generic reframing of the task type. "Define the
terms" is too vague; "Define the snowline, shape, and timing terms" is
right. The reader should be able to guess *what the task is about*, not
just *what kind of task it is*.

Stack the titles top-to-bottom and they should narrate the run like a
paragraph that names the actual subject matter at each step:

> Frame the Alaska glacier shape-vs-snowline question. Define the snowline,
> shape, and timing terms. Import 51 AutoDS-flagged glacier patterns. Find
> source papers for both datasets. Search literature on observed-vs-predicted
> snowline. Pick the patterns literature can't explain. Re-run the surviving
> patterns on a different year. Propose theories linking the surviving
> patterns. Write the closing report on what survived.

Forbidden in titles — none of these belong anywhere in `--title`:

| Pattern | Why it's wrong | Fix |
|---|---|---|
| `scope: …`, `literature_review: …`, `synthesis: …` | The task_type already appears as a chip in the UI. Repeating it eats half the title. | Drop the prefix. |
| `(6 surprises)`, `(2 surprises)` | Run-internal counts that read like meta-notes. | Drop. The number can live in the description. |
| `— observed-vs-hypsometric ELA discrepancy` | Field-specialist jargon (AAR, ELA, hypsometric, Lmax, surprisal). A non-expert can't parse it. | Use everyday English. "Compare textbook glacier shape to what's observed." |
| `findings.md`, `experiments.json`, `task.json` | Filenames in the title. | Describe what's inside the file, not the filename. |
| `theme A`, `theme B`, `node_2_4` | Run-internal labels. | Name the *content*: "glacier-shape predictors of AAR" not "theme A". |

Before / after on the current run. The "after" column names the actual
subject at each step, not just the task type:

| Before (don't do) | After (do this) |
|---|---|
| `scope: research question, boundaries, and success criteria for Alaska glacier AAR/ELA geometry synthesis` | `Frame Alaska glacier shape-vs-snowline question` |
| `definitions: operational definitions of ELA, AAR, geometry features, response-time terms` | `Define snowline, shape, and timing terms` |
| `autods_run: import local export 9fc65db8 (51 MCTS nodes, 16 surprising) into canonical experiments.json` | `Import 51 AutoDS-flagged glacier patterns` |
| `literature_review: provenance of TableA1_data.csv (Otto+25) and alaska_elas.csv (Zeller+25)` | `Find source papers for both datasets` |
| `literature_review: theme A — observed-vs-hypsometric ELA discrepancy (6 surprises)` | `Search literature on observed-vs-predicted snowline` |
| `literature_review: theme B — geometric predictors of observed AAR and AAR-hypsometric complexity` | `Search literature on shape-based AAR predictors` |
| `literature_review: theme C — regional and volume-area scaling structure` | `Search literature on regional volume-area scaling` |
| `literature_review: theme D — off_glacier_flag subpopulation morphology` | `Search literature on off-glacier subpopulations` |
| `autods_literature_synthesis: aggregate provenance + 4 themes into surviving_findings list` | `Pick patterns literature can't explain` |
| `replication: cross-year Zeller+25 re-test of 6 surviving findings + Otto+25 Zenodo geometry source` | `Re-run surviving patterns on different year` |
| `theorize: cross-cutting mechanisms for the 6 surviving findings (fallback synthesis, remote CLI unreachable)` | `Propose theories linking surviving glacier patterns` |
| `autods_synthesis: findings.md — closing artifact for run 9fc65db8` | `Write closing report on what survived` |

### Descriptions — one short paragraph, ELI-college

Two to four sentences. One continuous paragraph (no Markdown headers,
bullets, or bold). The paragraph answers, for a curious non-expert reader:

1. **Why** is this task here? (What question is it answering for the run?)
2. **What** will it produce, in everyday language?

Forbidden in descriptions:

- **Raw schema or AutoDS labels.** Don't paste `node_2_0`, `node_3_15`,
`gwi_feq`, `Lmax`, `ela_hyps`, `CLASSIFICATION:`, `surviving_findings`
into the description prose. If you need to refer to "the six surprising
findings", say *"the six surprising findings"* — the per-finding details
belong in the output JSON and the markdown report, not in the
description. A reader hitting `node_2_0` in the panel has no idea what
it means.
- **Internal field/file names.** No `findings.md`, no `experiments.json`,
no `output.surviving_findings`. Describe the *thing*, not the schema.
- **Paper citations (Smith+25, Otto et al. 2025).** Citations live in the
literature output. The description shouldn't double as a bibliography.
- **Meta narration.** "In this task we will…", "Let me…", "I'll now…",
"As an AI…", "Here is what we did".
- **LLM filler.** "deep dive", "leverage", "unlock", "delve", scare
quotes, excessive em dashes.
- **bd ids.** Reference upstream work by what it *does* ("the literature
review on glacier shape"), not its bd id.

Before / after, same Theme-A task:

> ❌ *This per-theme literature review classifies the AutoDS surprises about
> the gap between observed end-of-summer ELA (Zeller+25 alaska_elas.csv) and
> its theoretical / hypsometric counterparts. Six of the run's 16 surprising
> nodes fall in this theme: 2_0 (slope vs lela - Zelal), 3_4 (ela - Zela vs
> gwi_feq), 3_15 (hypsometric ELA correlation by glacier size), 4_10 (lela -
> ela_hypsl by area quartile), 4_15 (slope vs ela - ela_hyps), and 3_12
> (Lmax vs ELA directly). Each node is tagged with a CLASSIFICATION line
> (textbook, construction-artifact, candidate-surprise, novel-extension,
> inconclusive) so the autods_literature_synthesis can compute
> surviving_findings.*

> ✅ *Six of the surprising patterns AutoDS flagged are about a mismatch
> between the snowline glaciologists observe in the field and the one you'd
> predict from each glacier's shape. This task checks each pattern against
> the standard glaciology textbooks: is it a known relationship, a side
> effect of how the dataset was built, or something genuinely new? The
> verdict for each pattern is the input to the next step, which compiles
> what survived as worth a closer look.*

The "after" version is shorter, uses everyday English, and never references
a node id or schema field. A college reader can follow it; an expert still
gets the same information.

## Workflows

Expand All @@ -35,7 +188,7 @@ Installing `bd` and `jq`, running `bd init`, and verifying `scripts/summary-chec
| **execute** | Run one ready task end-to-end. Hands off to **plan** when the closed task type unlocks new structure; otherwise to **update-summary**. | `workflows/execute.md` |
| **update-summary** | Regenerate `summary.md` from beads. Idempotent — no-op when `scripts/summary-check.sh` reports `status: fresh`. | `workflows/update-summary.md` |

Task-type schemas live in `assets/schemas.yaml`.
Task-type schemas live in `assets/schemas.yaml`. **Per-template overrides** live in `workflows/plans/<name>.md` — each template file holds its bootstrap rules, replan rules, and per-task-type execution prose. Templates are selected by `plan.md` reading `mission.md` prose and matching against the templates' intro sections (no frontmatter required). The default (no template matches) is the hypothesis-driven flow described across `workflows/{plan,execute,update-summary}.md`.

## Routing

Expand Down
Loading
Loading