diff --git a/plugins/asta-preview/skills/research-step/SKILL.md b/plugins/asta-preview/skills/research-step/SKILL.md index 0d2fcee..4c9b7a8 100644 --- a/plugins/asta-preview/skills/research-step/SKILL.md +++ b/plugins/asta-preview/skills/research-step/SKILL.md @@ -23,7 +23,160 @@ Installing `bd` and `jq`, running `bd init`, and verifying `scripts/summary-chec | `mission.md` | Input. The research task. | | `.beads/` | Source of truth for state. | | `summary.md` | Derived view of the session, regenerated by **update-summary**. Beads is the source of truth; this file is just a digest for humans and for **brainstorm**. Frontmatter `beads_snapshot` records the state it was rendered from. | -| `background_knowledge.txt` | Optional. Long-form context referenced from issue metadata via `summary_path`. | +| `.asta///` | One **work_dir** per task. The agent picks the kebab-case slug at task creation; the full path is stored at `metadata.research_step.work_dir`. Every file the task writes goes here, following the per-task-type filename conventions in `assets/schemas.yaml > file_conventions`. | + +## Per-task work_dir convention + +Each task issue declares one working directory: + +``` +metadata.research_step.work_dir = ".asta///" +``` + +- `` is the schema task type (e.g. `literature_review`). +- `` is a short kebab-case name the agent picks at create-time + (e.g. `provenance`, `theme-a-aar-geometry`, `cross-dataset`). +- Every file the task writes lives **under** this directory. Sibling + directories (`.asta/literature_review/theme-b-ela-bands/`) are independent. +- The filenames inside follow `assets/schemas.yaml > file_conventions` + (e.g. lit_review always writes `summary.md`; synthesis writes + `report.md`) so downstream consumers can find them without reading + paths out of metadata. + +The Asta Flows UI renders this directory as a file browser on the task's +**Files** tab. No `_path` fields are required; the directory IS the manifest. + +## Markdown convention: inline citations + +Any markdown file a task writes inline-links to the artifacts that back its +statements. The Asta Flows panel auto-linkifies bd ids, AutoDS node ids, and +run-relative paths; explicit Markdown links handle the rest. + +Use real markdown links for every: + +- **Paper mention** → `[Author+YY](https://www.semanticscholar.org/paper/)`. + Never write a bare `corpusId 12345` or `doi:10.1234/...` in prose. +- **Numeric claim or verdict** → link to the script/log/table that produced + it, e.g. `[r=+0.22, p<1e-30](replicate.py)` (relative to the task's own + work_dir) or `[replicated](../../replication//per-finding-table.md#node_3_2)` + (relative path into another task's work_dir). +- **Cross-reference to another task's output** → relative path from this + task's work_dir into the upstream task's work_dir (e.g. + `../../literature_review/theme-a-aar-geometry/summary.md`). +- **Cross-reference to another bd task** → bare bd id; the renderer + auto-linkifies it. +- **AutoDS hypothesis / experiment** → `../../autods_run//experiments.json`. + +Per-template tweaks live in `workflows/plans/.md`. + +## Voice for titles and descriptions + +Every issue in beads must have a `--title` and a non-empty `--description`. +The Asta Flows UI surfaces both as the primary content of each task. + +**Imagine the reader.** A smart college graduate scrolling the Tasks page +who has never heard of this run, this domain, or these acronyms. They are +not stupid — they have a degree — but they have zero context. Your titles +and descriptions are how they figure out what this run is doing. + +### Titles — imperative phrase, ≤ 6 words + +Lead with a verb and name the **specific subject** of the step in plain +English — don't stop at a generic reframing of the task type. "Define the +terms" is too vague; "Define the snowline, shape, and timing terms" is +right. The reader should be able to guess *what the task is about*, not +just *what kind of task it is*. + +Stack the titles top-to-bottom and they should narrate the run like a +paragraph that names the actual subject matter at each step: + +> Frame the Alaska glacier shape-vs-snowline question. Define the snowline, +> shape, and timing terms. Import 51 AutoDS-flagged glacier patterns. Find +> source papers for both datasets. Search literature on observed-vs-predicted +> snowline. Pick the patterns literature can't explain. Re-run the surviving +> patterns on a different year. Propose theories linking the surviving +> patterns. Write the closing report on what survived. + +Forbidden in titles — none of these belong anywhere in `--title`: + +| Pattern | Why it's wrong | Fix | +|---|---|---| +| `scope: …`, `literature_review: …`, `synthesis: …` | The task_type already appears as a chip in the UI. Repeating it eats half the title. | Drop the prefix. | +| `(6 surprises)`, `(2 surprises)` | Run-internal counts that read like meta-notes. | Drop. The number can live in the description. | +| `— observed-vs-hypsometric ELA discrepancy` | Field-specialist jargon (AAR, ELA, hypsometric, Lmax, surprisal). A non-expert can't parse it. | Use everyday English. "Compare textbook glacier shape to what's observed." | +| `findings.md`, `experiments.json`, `task.json` | Filenames in the title. | Describe what's inside the file, not the filename. | +| `theme A`, `theme B`, `node_2_4` | Run-internal labels. | Name the *content*: "glacier-shape predictors of AAR" not "theme A". | + +Before / after on the current run. The "after" column names the actual +subject at each step, not just the task type: + +| Before (don't do) | After (do this) | +|---|---| +| `scope: research question, boundaries, and success criteria for Alaska glacier AAR/ELA geometry synthesis` | `Frame Alaska glacier shape-vs-snowline question` | +| `definitions: operational definitions of ELA, AAR, geometry features, response-time terms` | `Define snowline, shape, and timing terms` | +| `autods_run: import local export 9fc65db8 (51 MCTS nodes, 16 surprising) into canonical experiments.json` | `Import 51 AutoDS-flagged glacier patterns` | +| `literature_review: provenance of TableA1_data.csv (Otto+25) and alaska_elas.csv (Zeller+25)` | `Find source papers for both datasets` | +| `literature_review: theme A — observed-vs-hypsometric ELA discrepancy (6 surprises)` | `Search literature on observed-vs-predicted snowline` | +| `literature_review: theme B — geometric predictors of observed AAR and AAR-hypsometric complexity` | `Search literature on shape-based AAR predictors` | +| `literature_review: theme C — regional and volume-area scaling structure` | `Search literature on regional volume-area scaling` | +| `literature_review: theme D — off_glacier_flag subpopulation morphology` | `Search literature on off-glacier subpopulations` | +| `autods_literature_synthesis: aggregate provenance + 4 themes into surviving_findings list` | `Pick patterns literature can't explain` | +| `replication: cross-year Zeller+25 re-test of 6 surviving findings + Otto+25 Zenodo geometry source` | `Re-run surviving patterns on different year` | +| `theorize: cross-cutting mechanisms for the 6 surviving findings (fallback synthesis, remote CLI unreachable)` | `Propose theories linking surviving glacier patterns` | +| `autods_synthesis: findings.md — closing artifact for run 9fc65db8` | `Write closing report on what survived` | + +### Descriptions — one short paragraph, ELI-college + +Two to four sentences. One continuous paragraph (no Markdown headers, +bullets, or bold). The paragraph answers, for a curious non-expert reader: + +1. **Why** is this task here? (What question is it answering for the run?) +2. **What** will it produce, in everyday language? + +Forbidden in descriptions: + +- **Raw schema or AutoDS labels.** Don't paste `node_2_0`, `node_3_15`, + `gwi_feq`, `Lmax`, `ela_hyps`, `CLASSIFICATION:`, `surviving_findings` + into the description prose. If you need to refer to "the six surprising + findings", say *"the six surprising findings"* — the per-finding details + belong in the output JSON and the markdown report, not in the + description. A reader hitting `node_2_0` in the panel has no idea what + it means. +- **Internal field/file names.** No `findings.md`, no `experiments.json`, + no `output.surviving_findings`. Describe the *thing*, not the schema. +- **Paper citations (Smith+25, Otto et al. 2025).** Citations live in the + literature output. The description shouldn't double as a bibliography. +- **Meta narration.** "In this task we will…", "Let me…", "I'll now…", + "As an AI…", "Here is what we did". +- **LLM filler.** "deep dive", "leverage", "unlock", "delve", scare + quotes, excessive em dashes. +- **bd ids.** Reference upstream work by what it *does* ("the literature + review on glacier shape"), not its bd id. + +Before / after, same Theme-A task: + +> ❌ *This per-theme literature review classifies the AutoDS surprises about +> the gap between observed end-of-summer ELA (Zeller+25 alaska_elas.csv) and +> its theoretical / hypsometric counterparts. Six of the run's 16 surprising +> nodes fall in this theme: 2_0 (slope vs lela - Zelal), 3_4 (ela - Zela vs +> gwi_feq), 3_15 (hypsometric ELA correlation by glacier size), 4_10 (lela - +> ela_hypsl by area quartile), 4_15 (slope vs ela - ela_hyps), and 3_12 +> (Lmax vs ELA directly). Each node is tagged with a CLASSIFICATION line +> (textbook, construction-artifact, candidate-surprise, novel-extension, +> inconclusive) so the autods_literature_synthesis can compute +> surviving_findings.* + +> ✅ *Six of the surprising patterns AutoDS flagged are about a mismatch +> between the snowline glaciologists observe in the field and the one you'd +> predict from each glacier's shape. This task checks each pattern against +> the standard glaciology textbooks: is it a known relationship, a side +> effect of how the dataset was built, or something genuinely new? The +> verdict for each pattern is the input to the next step, which compiles +> what survived as worth a closer look.* + +The "after" version is shorter, uses everyday English, and never references +a node id or schema field. A college reader can follow it; an expert still +gets the same information. ## Workflows @@ -35,7 +188,7 @@ Installing `bd` and `jq`, running `bd init`, and verifying `scripts/summary-chec | **execute** | Run one ready task end-to-end. Hands off to **plan** when the closed task type unlocks new structure; otherwise to **update-summary**. | `workflows/execute.md` | | **update-summary** | Regenerate `summary.md` from beads. Idempotent — no-op when `scripts/summary-check.sh` reports `status: fresh`. | `workflows/update-summary.md` | -Task-type schemas live in `assets/schemas.yaml`. +Task-type schemas live in `assets/schemas.yaml`. **Per-template overrides** live in `workflows/plans/.md` — each template file holds its bootstrap rules, replan rules, and per-task-type execution prose. Templates are selected by `plan.md` reading `mission.md` prose and matching against the templates' intro sections (no frontmatter required). The default (no template matches) is the hypothesis-driven flow described across `workflows/{plan,execute,update-summary}.md`. ## Routing diff --git a/plugins/asta-preview/skills/research-step/assets/schemas.yaml b/plugins/asta-preview/skills/research-step/assets/schemas.yaml index b840628..b98a4c1 100644 --- a/plugins/asta-preview/skills/research-step/assets/schemas.yaml +++ b/plugins/asta-preview/skills/research-step/assets/schemas.yaml @@ -1,8 +1,43 @@ # Output schemas for research-step task types. -# Each task issue stores its realized output at metadata.research_step.output, -# matching the shape under `output:` for its task_type. +# +# Each task issue stores its realized output under metadata.research_step: +# +# metadata.research_step: +# task_type: +# inputs: [bd_id, ...] # concrete upstream task bd ids +# work_dir: ".asta///" # the single directory this task wrote to. +# # is a short kebab-case name the agent +# # picks at create time. Asta Flows reads this +# # to render the Files tab. +# output: { ...per-type structured fields, schema below... } +# +# What's NOT in here anymore (schema_version 3 hard cut): +# - `artifacts: []` and `consumes: []` uniform fields (M0.5) — replaced by `work_dir`. +# - Per-type `_path` / `_dir` legacy fields (`summary_path`, `experiments_path`, +# `report_path`, etc.) — files live under `work_dir`; canonical filenames +# are listed in the per-task convention block below so downstream tasks +# can find them without metadata pointers. +# +# Inline-citation rule for markdown outputs: see SKILL.md §"Markdown convention". -schema_version: 1 +schema_version: 3 + +# ── Per-task-type file conventions inside `work_dir` ─────────────────────────── +# Agents write predictable filenames under their work_dir so downstream tasks +# can find them without consulting metadata paths. +file_conventions: + scope: ["scope.md"] + definitions: ["terms.md"] + literature_review: ["summary.md", "thread/"] # thread/ is the asta-literature interactive workspace + hypothesis: ["hypothesis.md"] + experiment_design: ["design.md"] + evidence_gathering: ["log.md", "data/", "figures/"] + analysis: ["analysis.md"] + synthesis: ["report.md"] + autods_run: ["experiments.json", "run-metadata.json", "dataset-summary.md"] + autods_literature_synthesis: ["per-hypothesis-table.md", "summary.md"] + replication: ["per-finding-table.md", "/"] # one subdir per surviving node + theorize: ["task.json", "theories/", "novelty/", "citations.bib", "index.md"] task_types: @@ -24,14 +59,28 @@ task_types: literature_review: inputs: [scope, definitions] output: - summary_path: string # relative path; long-form context - key_findings: [string] # 3-10 bullets readable without opening summary_path + key_findings: [string] # 3-10 bullets readable without opening summary.md. + # autods-to-theorizer convention: per-theme lit_reviews include a + # "CLASSIFICATION: = " line per + # AutoDS hypothesis covered by the theme. gaps: [string] # gaps that motivate hypotheses citations: - id: string title: string url: string relevance: string + # ── optional fields used by autods-to-theorizer; hypothesis-driven leaves absent ── + provenance: # populated when this lit_review is the provenance phase. + source_papers: + - corpus_id: string + doi: string + title: string + authors: [string] + year: number + dataset_lineage: string + theme: string # populated when this lit_review is a per-theme task + corpus_ids: [string] # harvested by per-theme tasks; theorize unions these into paper_store hypothesis: inputs: [scope, literature_review] @@ -50,31 +99,165 @@ task_types: independent: [string] dependent: [string] controls: [string] - artifacts_expected: [string] # paths the gathering step will produce + artifacts_expected: [string] # filenames the gathering step is expected to write (under its own work_dir) evidence_gathering: inputs: [experiment_design] output: - artifacts: - - path: string - kind: string # data | log | figure | code | other - description: string - log_path: string # what was actually run deviations: [string] # ways execution diverged from design analysis: inputs: [hypothesis, evidence_gathering] output: verdict: enum [supported, refuted, inconclusive] - confidence: number # 0.0 - 1.0 + confidence: number # 0.0 - 1.0; agent's credence in the verdict reasoning: string caveats: [string] synthesis: - inputs: [scope, analysis_*] # all analysis issues in the epic + # The universal closing task for every template. Writes report.md as the + # run's deliverable. Two flavors: + # + # - Hypothesis-driven flow. + # inputs: [scope, analysis_*] # every analysis task in the epic + # The agent answers scope.question and lists which hypotheses were + # supported / refuted (by bd id). + # + # - AutoDS / theorize-driven flow. + # inputs: [autods_run, autods_literature_synthesis, theorize, + # (optional) replication] + # The agent leads with the theories from the upstream `theorize` + # task, curates them in report.md, references the AutoDS patterns + # and literature grounding that seeded each theory, and notes + # replication status when replication ran. Both + # supporting_hypotheses and refuted_hypotheses may be empty (or + # carry AutoDS node ids as strings if the agent finds that + # useful) — theories are the real centerpiece and live in + # report.md. + # + # Either way, the run's headline is in `answer` and any follow-up + # threads live in `open_questions`. + inputs: [scope, analysis_*] # hypothesis-driven flow; + # autods flow uses [autods_run, autods_literature_synthesis, theorize, replication] + output: + answer: string # one-line "so what" — names a theory for theory-driven runs, + # or the answer to scope.question for hypothesis-driven runs + supporting_hypotheses: [bd_id] # hypothesis-flow: bd ids of supported hypotheses. + # autods-flow: leave empty (theories live in report.md prose). + refuted_hypotheses: [bd_id] # symmetric counterpart; empty for autods flows + open_questions: [string] # next directions; become discovered-from edges on re-plan + + # ─────────────────────────────────────────────────────────────────────────────── + # autods-to-theorizer template task types + # ─────────────────────────────────────────────────────────────────────────────── + + autods_run: + inputs: [scope, definitions] + # Three sourcing modes, selected by what mission.md describes (no frontmatter needed): + # A1: UUID-shaped run id → `asta autodiscovery experiments --format json` + # A2: local export dir → parse /output/mcts_nodes_all.json (the STRUCTURED file; + # the node_*.json files are autogen conversation logs, don't use those) + # B: fresh-run intent → `asta autodiscovery create/upload/metadata/submit`; block until SUCCEEDED. + # Confirm with user before submitting (uses credits, 30+ min). + # If mission.md is ambiguous between modes, ask the user before doing anything expensive. + # Field-name gotcha: autodiscovery's records use `surprising` (boolean) and `normalized_surprisal` + # (number, can be negative — it's a log-likelihood-ratio, not a 0-1 probability). Map these into + # is_surprising and surprise_score below. + output: + run_id: string # A1: from mission; A2: basename of path; B: new ID + mode: enum [imported, executed] + dataset_summary: + datasets: + - name: string + description: string + n_rows: number + columns: [string] + intent: string + domain: string + hypotheses: + - node_id: string # e.g., node_2_1 + statement: string + prior_analysis: string # truncate inline; keep full text on disk + is_surprising: boolean # ← autodiscovery's `surprising` field + surprise_score: number # ← autodiscovery's `normalized_surprisal` field + surprising_count: number + + autods_literature_synthesis: + inputs: [autods_run, "literature_review_*"] # 1 provenance + N theme output: - answer: string # answer to scope.question - supporting_hypotheses: [bd_id] - refuted_hypotheses: [bd_id] - open_questions: [string] # become discovered-from edges on re-plan - report_path: string # generated markdown report + themes: + - name: string + contributing_lit_review_ids: [bd_id] + key_findings: [string] + gaps: [string] + surviving_findings: [string] # AutoDS node_ids where hypothesis.is_surprising == true AND + # lit-review CLASSIFICATION ∈ {candidate-surprise, novel-extension}. + # Drives the post-lit gates in the diagram. + contradicted_findings: [string] # AutoDS node_ids explicitly refuted by lit + headline: string # one-line: "3 of 10 AutoDS findings remain still-surprising after lit" + + replication: + inputs: [autods_run, autods_literature_synthesis] + # Replication = re-test surviving findings on a DIFFERENT cut of the data (different + # year, different dataset, different join). Not "reproduction" in the same-data-same- + # analysis sense; matches the NSF/NASEM "replicability" definition. + # Skip if surviving_findings is empty (replan handles). + # Faithful-attempt requirement + workflow guidance: see autods-to-theorizer.md. + output: + candidate_datasets: + - name: string + source_url: string + rationale: string # why "similar" + fetch_status: enum [ok, partial, failed] + findings: + - node_id: string + verdict: enum [replicated, failed_to_replicate, partial, inconclusive, unreachable] + # `replicated`: same direction + comparable effect size on the new cut + # `failed_to_replicate`: clearly different (opposite sign or null effect) + # `partial`: directionally consistent but effect differs substantially + # `inconclusive`: test runs but is underpowered or ambiguous + # `unreachable`: all candidate datasets failed to fetch + confidence: number # 0.0-1.0; anchors: + # 0.9+ = clean evidence, well-specified test + # 0.7+ = solid but one caveat + # 0.5+ = noisy/marginal + # <0.5 = use 'inconclusive' instead of a definitive verdict + # Post-replication gate fires on (failed_to_replicate AND confidence > 0.7). + datasets_used: [string] + reasoning: string + deviations: [string] # fetch failures, fallbacks taken, cross-dataset infeasibility + + theorize: + inputs: [autods_run, autods_literature_synthesis, replication] + # replication is absent if the all-known gate fired. + # + # ⚠️ Context-window contract: `asta generate-theories` is a remote A2A service. Its only inputs are + # the --theory-query string, the --paper-store JSON, and papers its own PaperFinder discovers. + # It CANNOT read bd, local files, or IDs not inlined in the query. Inline everything; cite papers + # by author/year, not corpus_id. + # + # Paper-store gateway limitation: `inject_papers` only honors entries with `paper_markdown`; + # identifier-only stubs are silently dropped. Best to pass an empty paper store for now. + # + # No agent-side fallback. If `asta generate-theories` does not return a populated theories + # list (PaperFinder failure, network failure, non-zero exit, empty result), the task is NOT + # closed and theories are NOT composed by the agent. Set status=failed on the theorize task + # AND on the epic via `bd update --status=failed` (the `failed` status is registered by + # init.md as a custom status with category=done), then stop the run for human review. + # + # After the CLI completes, dump the task JSON and export with `asta artifacts --format md` + # rather than parsing the streamed CLI log. Both end up inside work_dir. + output: + theory_query: string + theories: + - name: string + statement: string # the theory's mechanistic claim (from artifact description) + novelty_rollup: enum [genuinely-new, derivable-but-unstated, already-established] + # best-of across novelty_claims[].label + novelty_claims: + - claim_name: string + label: enum [genuinely-new, derivable-but-unstated, already-established] + judgment_excerpt: string # first ~200 chars of the agent's novelty judgment + falsifiable_prediction: string # concatenation of novelty_claims[].claim_name + evidence_grounding: [string] # paper s2_sha or corpus_ids cited in supporting passages + diff --git a/plugins/asta-preview/skills/research-step/scripts/validate-output.sh b/plugins/asta-preview/skills/research-step/scripts/validate-output.sh index 0f5a84e..68318c0 100755 --- a/plugins/asta-preview/skills/research-step/scripts/validate-output.sh +++ b/plugins/asta-preview/skills/research-step/scripts/validate-output.sh @@ -6,9 +6,9 @@ # Verifies that the JSON file: # 1. parses # 2. carries the canonical metadata envelope -# ({research_step: {task_type, inputs, output_schema_version, output}}) +# ({research_step: {task_type, inputs, work_dir, output}}) # 3. has every required `output.` for the given per -# assets/schemas.yaml (schema_version: 1) +# assets/schemas.yaml (schema_version: 3) # # Exit codes: # 0 — valid @@ -16,6 +16,7 @@ # 3 — unknown task_type # 4 — missing required field # 5 — task_type mismatch with envelope +# 6 — bad work_dir format # # This is structural validation only. Quality validation (sound prediction, # sane confidence, valid citations) is out of scope per execute.md. @@ -34,19 +35,24 @@ if ! jq -e . "$file" > /dev/null 2>&1; then exit 2 fi -# Required output fields, mirroring assets/schemas.yaml (schema_version: 1). +# Required output fields, mirroring assets/schemas.yaml (schema_version: 3). +# Only structured-answer fields — paths and artifact lists are gone. case "$task_type" in - scope) required="question boundaries success_criteria" ;; - definitions) required="terms" ;; - literature_review) required="summary_path key_findings gaps citations" ;; - hypothesis) required="statement rationale falsifiable_prediction expected_evidence" ;; - experiment_design) required="method procedure variables artifacts_expected" ;; - evidence_gathering) required="artifacts log_path deviations" ;; - analysis) required="verdict confidence reasoning caveats" ;; - synthesis) required="answer supporting_hypotheses refuted_hypotheses open_questions report_path" ;; + scope) required="question boundaries success_criteria" ;; + definitions) required="terms" ;; + literature_review) required="key_findings gaps citations" ;; + hypothesis) required="statement rationale falsifiable_prediction expected_evidence" ;; + experiment_design) required="method procedure variables artifacts_expected" ;; + evidence_gathering) required="deviations" ;; + analysis) required="verdict confidence reasoning caveats" ;; + synthesis) required="answer supporting_hypotheses refuted_hypotheses open_questions" ;; + autods_run) required="run_id mode dataset_summary hypotheses surprising_count" ;; + autods_literature_synthesis) required="themes surviving_findings contradicted_findings headline" ;; + replication) required="candidate_datasets findings deviations" ;; + theorize) required="theory_query theories" ;; *) echo "validate-output: unknown task_type '$task_type'" >&2 - echo "validate-output: expected one of scope|definitions|literature_review|hypothesis|experiment_design|evidence_gathering|analysis|synthesis" >&2 + echo "validate-output: expected one of scope|definitions|literature_review|hypothesis|experiment_design|evidence_gathering|analysis|synthesis|autods_run|autods_literature_synthesis|replication|theorize" >&2 exit 3 ;; esac @@ -63,14 +69,33 @@ if [[ "$envelope_type" != "$task_type" ]]; then exit 5 fi -# Envelope shape sanity. -for key in inputs output_schema_version output; do +# Envelope shape sanity. work_dir is the new mandatory pointer to this task's files. +for key in inputs work_dir output; do if ! jq -e ".research_step | has(\"$key\")" "$file" >/dev/null; then echo "validate-output: $file missing .research_step.$key" >&2 exit 5 fi done +# work_dir format check: must be a run-root-relative path that starts with +# `.asta//` and ends with `/`. Catches drift from the convention. +work_dir=$(jq -r '.research_step.work_dir // empty' "$file") +if [[ -z "$work_dir" ]]; then + echo "validate-output: .research_step.work_dir is empty" >&2; exit 6 +fi +expected_prefix=".asta/${task_type}/" +case "$work_dir" in + "$expected_prefix"*) : ;; + *) echo "validate-output: work_dir '$work_dir' should start with '$expected_prefix'" >&2; exit 6 ;; +esac +case "$work_dir" in + */) : ;; + *) echo "validate-output: work_dir '$work_dir' must end with '/'" >&2; exit 6 ;; +esac +case "$work_dir" in + /*|*..*) echo "validate-output: work_dir '$work_dir' must be run-root-relative (no leading '/', no '..')" >&2; exit 6 ;; +esac + # Required output fields. for key in $required; do if ! jq -e ".research_step.output | has(\"$key\")" "$file" >/dev/null; then @@ -80,8 +105,7 @@ for key in $required; do done # Type spot-checks for the high-leverage cases. Not exhaustive — just the -# fields where a wrong type at this layer would silently break update-summary rendering -# or downstream tasks. +# fields where a wrong type at this layer would silently break downstream tasks. case "$task_type" in literature_review) jq -e '.research_step.output.key_findings | type == "array"' "$file" >/dev/null \ @@ -97,6 +121,27 @@ case "$task_type" in jq -e '.research_step.output.confidence | type == "number" and . >= 0 and . <= 1' "$file" >/dev/null \ || { echo "validate-output: output.confidence must be a number in [0, 1]" >&2; exit 4; } ;; + autods_run) + jq -e '.research_step.output.mode | IN("imported", "executed")' "$file" >/dev/null \ + || { echo "validate-output: output.mode must be one of imported|executed" >&2; exit 4; } + ;; + replication) + if [[ $(jq '.research_step.output.findings | length' "$file") -gt 0 ]]; then + jq -e '.research_step.output.findings[0].verdict | IN("replicated","failed_to_replicate","partial","inconclusive","unreachable")' "$file" >/dev/null \ + || { echo "validate-output: findings[].verdict must be one of replicated|failed_to_replicate|partial|inconclusive|unreachable" >&2; exit 4; } + fi + ;; + theorize) + # No fallback path: theories must be populated by the remote generate-theories agent. + # If the agent did not return theories, the task should be marked status=failed (and the + # epic too) and NOT closed; validate-output is not called on a failed task. + if [[ $(jq '.research_step.output.theories | length' "$file") -eq 0 ]]; then + echo "validate-output: output.theories must be non-empty; if generate-theories did not return any theories, set status=failed on this task and the epic instead of closing" >&2 + exit 4 + fi + jq -e '.research_step.output.theories[0].novelty_rollup | IN("genuinely-new","derivable-but-unstated","already-established")' "$file" >/dev/null \ + || { echo "validate-output: theories[].novelty_rollup must be one of genuinely-new|derivable-but-unstated|already-established" >&2; exit 4; } + ;; esac echo "ok" diff --git a/plugins/asta-preview/skills/research-step/workflows/execute.md b/plugins/asta-preview/skills/research-step/workflows/execute.md index 5fba9ea..096d1d2 100644 --- a/plugins/asta-preview/skills/research-step/workflows/execute.md +++ b/plugins/asta-preview/skills/research-step/workflows/execute.md @@ -9,32 +9,31 @@ Run one ready task end-to-end. Loads its schema, gathers its declared inputs, pr ## Steps -1. **Pick a task.** If a task ID was supplied, use it. Else `bd ready --json` and pick the oldest issue (tiebreak by `bd-id` ascending). Hypothesis tasks are normally auto-resolved at creation by **plan**, so they should not appear here. If one does, it means the gap text was too thin for plan to fill the output without inventing content — flag this to the user and ask whether to refine the source `literature_review` first. +1. **Pick a task.** If a task ID was supplied, use it. Else `bd ready --json` and pick the oldest issue (tiebreak by `bd-id` ascending). 2. **Claim it.** `bd update --status=in_progress`. -3. **Load the schema.** Read the task type with `bd show --json | jq -r '.[0].metadata.research_step.task_type'`. Open `assets/schemas.yaml` and find the matching entry under `task_types`. -4. **Gather inputs.** For every issue listed in this issue's `inputs` (`bd show --json | jq '.[0].metadata.research_step.inputs'`), read its output with `bd show --json | jq '.[0].metadata.research_step.output'`. Also load `mission.md` and any files referenced from input outputs via `_path` fields (e.g., `summary_path` from a `literature_review`). **This is the only context to use** — do not pull in unrelated repo state. -5. **Do the work.** Produce a JSON object matching the schema. For schema fields ending in `_path`, write the file to disk first and put the relative path in the JSON. -6. **Validate structurally.** Run `scripts/validate-output.sh `. It checks the envelope (`research_step.task_type`, `inputs`, `output_schema_version`, `output`) and every required `output.` for the task_type, plus type spot-checks for the high-leverage cases (e.g., `analysis.verdict` enum, `analysis.confidence` range). Exit 0 ⇒ valid. Any non-zero exit ⇒ fail loudly and **leave the issue `in_progress`** for retry. Do not close. -7. **Persist the output.** Materialize the metadata JSON via `scripts/write-meta.sh` (reads JSON from stdin, prints a temp file path), then `bd update --metadata @`. Preserve the existing `task_type`, `inputs`, and `output_schema_version`. -8. **Close.** `bd close `. -9. **Hand off to plan or update-summary.** Some closed task types unlock new graph structure; others don't. Decide based on the closed task's `task_type`: - - | Closed task_type | Hand off to | - |---|---| - | `literature_review`, `hypothesis`, `analysis`, `synthesis` | **plan** (with this issue as the source). `plan` then chains to **update-summary**. Note: `hypothesis` only reaches this branch in the rare case it was left open at creation; the normal path is plan→auto-resolve. | - | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | **update-summary** directly. | - - Either path ends with `summary.md` rebuilt. - -## Notes on output files - -Schema fields ending in `_path` are relative paths. Conventions: - -- `summary_path` (from `literature_review`) → `background_knowledge.txt` by convention, but any path works. -- `log_path` (from `evidence_gathering`) → typically under `logs/`. -- `report_path` (from `synthesis`) → typically `report.md`. - -Write the file before setting the output JSON. If the executor crashes between writing the file and closing the issue, the file is harmless orphan data — re-running `execute` on the same issue will overwrite it. +3. **Load the schema.** Read `metadata.research_step.task_type` from the issue. Open `assets/schemas.yaml` and find the matching entry under `task_types`. **If the task type isn't one of the 8 standard types** (`scope`, `definitions`, `literature_review`, `hypothesis`, `experiment_design`, `evidence_gathering`, `analysis`, `synthesis`), scan `workflows/plans/*.md` for the template that defines this task type and follow its `### ` section for execution prose (CLI commands, paths, parsing details, fallback handling). +4. **Resolve the work_dir.** Read `metadata.research_step.work_dir`. Every file this task writes goes under that directory. The directory should already exist from `plan`'s create step; `mkdir -p "$work_dir"` is a safe no-op. Filenames inside follow `assets/schemas.yaml > file_conventions` for the task type — write `summary.md` for `literature_review`, `findings.md` for `autods_synthesis`, etc. +5. **Gather inputs.** Read the `metadata.research_step.output` of every issue listed in this issue's `inputs`. For each upstream task, also walk its `work_dir` directly (e.g., open `/summary.md`) when the prose context matters. **This is the only context to use** — do not pull in unrelated repo state. +5a. **Steering pickup.** Also `bd show --json` and inspect `comments[]`. For every comment **authored by someone other than `claude`/`agent`** (per `author` regex `/^(claude\|agent)/i`) that was created **since the most recent agent-authored comment on this task** (or since the task was created, whichever is later), treat it as a steering note. The agent decides how to respond — acknowledge inline (`bd comment add "" --author claude`), adjust the plan, or note explicitly why it cannot apply. Do not silently ignore. +6. **Do the work.** Write the structured output as a JSON object matching the schema. Write supporting files (markdown reports, JSON dumps, scripts, figures) inside `work_dir` per the file convention. Markdown files follow the inline-citation rule in `SKILL.md §"Markdown convention"`. +7. **Validate structurally.** Run `scripts/validate-output.sh `. It checks the envelope (`research_step.task_type`, `inputs`, `work_dir`, `output`) and every required `output.` for the task_type, plus type spot-checks for the high-leverage cases (e.g., `analysis.verdict` enum, `analysis.confidence` range). Exit 0 ⇒ valid. Any non-zero exit ⇒ fail loudly and **leave the issue `in_progress`** for retry. Do not close. +8. **Persist the output.** Materialize the metadata JSON via `scripts/write-meta.sh` (reads JSON from stdin, prints a temp file path), then `bd update --metadata @`. Preserve the existing `task_type`, `inputs`, and `work_dir`. +9. **Write a one-line `close_reason`.** A single short sentence stating what + was produced, in the plain-English voice from + `SKILL.md §"Voice for titles and descriptions"` (same ELI-college reader, + no jargon up front, no internal labels). Used in summary digests and + `bd list` output; not rendered in the Asta Flows panel. Avoid empty + placeholders (`"completed"`, `"done"`, `""`) — they aren't useful to + humans scanning the log. +10. **Close.** `bd close --reason ""`. +11. **Hand off to plan or update-summary.** Some closed task types unlock new graph structure; others don't. Decide based on the closed task's `task_type`: + + | Closed task_type | Hand off to | + |---|---| + | `literature_review`, `hypothesis`, `analysis`, `synthesis`, `*_synthesis` | **plan** (with this issue as the source). `plan` then chains to **update-summary**. | + | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | **update-summary** directly. | + + Either path ends with `summary.md` rebuilt. ## Out of scope for this workflow diff --git a/plugins/asta-preview/skills/research-step/workflows/init.md b/plugins/asta-preview/skills/research-step/workflows/init.md index fd11be3..37b34ed 100644 --- a/plugins/asta-preview/skills/research-step/workflows/init.md +++ b/plugins/asta-preview/skills/research-step/workflows/init.md @@ -1,6 +1,6 @@ # Workflow: init -Bootstrap the environment for a research session: install `bd` and `jq`, run `bd init`, wire beads to the project's git remote for cross-machine sync, and verify the staleness check works. This is the only workflow that may install or configure tools; `plan`, `update-summary`, and `execute` assume the environment is ready. +Bootstrap the environment for a research session: install `bd` and `jq`, run `bd init`, and verify the staleness check works. This is the only workflow that may install or configure tools; `plan`, `update-summary`, and `execute` assume the environment is ready. After environment setup, hand off to **plan** to bootstrap the mission epic and initial frontier. @@ -8,12 +8,6 @@ After environment setup, hand off to **plan** to bootstrap the mission epic and None. `init` is idempotent — installing already-installed tools is a no-op, and `bd init` is safe to skip if `.beads/` already exists. -## Backend choice - -`init` always uses beads' embedded Dolt backend (`bd init`, no `--server`). This satisfies the single-machine workflow with zero infrastructure. Cross-machine transfer is handled via `bd dolt push`/`pull` against the project's git remote (under `refs/dolt/*`) — git is the only sync layer. - -Server mode (`bd init --server`) is out of scope: it requires running a Dolt sql-server, which violates the zero-infrastructure constraint. Simultaneous writers on multiple machines are therefore unsupported; the model is one writer at a time, with git as the transfer medium. - ## Steps 1. **Ensure `bd` is installed.** @@ -21,36 +15,22 @@ Server mode (`bd init --server`) is out of scope: it requires running a Dolt sql - Otherwise, install per the beads project's documented method for the current platform. Consult https://github.com/gastownhall/beads (cross-platform options at time of writing include `go install`, Homebrew, `winget`, the project's install script, and the Nix flake). If you are uncertain which method applies, fetch the install docs at run time rather than guessing. - Verify with `bd --version`. If it still fails, abort and ask the user to install manually. -2. **Initialize beads (embedded Dolt) and wire it to git.** +2. **Ensure beads is initialized in this directory.** - If `.beads/` does not exist, run `bd init`. - - If the working directory has a git remote `origin`, configure beads' Dolt sync against it so `bd dolt push`/`pull` work without further setup. Probe with `bd dolt remote list`; if nothing is configured, add a remote pointing at `origin` per beads docs. - - If no git remote exists, skip the Dolt-remote step and tell the user that cross-machine transfer will need a remote added later. Do not block — the single-machine flow works without it. - Verify with `bd list` (should succeed and return an empty list or existing issues). + - Register the `failed` custom status used by the autods-to-theorizer template's failure path: + `bd config set status.custom "failed:done"`. Idempotent — safe to re-run. + Confirm with `bd statuses` (should list `failed` under "Custom statuses"). -3. **Fresh-clone recovery.** If `bd init` produced an empty DB but `.beads/issues.jsonl` is present (the project had upstream beads state and was freshly cloned), **do not silently `bd import`**. - - Run `git ls-remote origin 'refs/dolt/*'`. If any Dolt refs exist, run `bd dolt pull` — this is the canonical recovery path and preserves Dolt history. - - If no Dolt refs exist on the remote, surface the situation to the user with three options: (a) `bd import .beads/issues.jsonl` (fast, but discards Dolt history and any state newer than the export), (b) configure a Dolt remote and `bd dolt push` from another machine that has the live DB, then retry, (c) abort. - - Pick one path only after explicit user confirmation. Never auto-import. - -4. **Verify the staleness check works.** +3. **Verify the staleness check works.** - Run `scripts/summary-check.sh`. It hashes the sorted IDs of currently-open issues and compares against `summary.md`'s frontmatter. Backend-agnostic — beads can use whichever storage it likes. - Requires `jq` on PATH; if missing, install it (`brew install jq`, `apt-get install jq`, etc.) and retry. - At init time `summary.md` does not yet exist, so the script will print `status: missing` and exit 1 — that's fine; **update-summary** will create the file later. `status: no-tools` (exit 3) means abort and ask the user. -5. **Hand off to plan.** Per the router's chaining rule, run the **plan** workflow next. It will detect that no epic exists yet and bootstrap one from `mission.md`. If `mission.md` is missing, **plan** will route the user back to **brainstorm**. - -## Cross-machine transfer - -To move a session to another machine: - -1. On machine A: `bd dolt push` (sends the Dolt data as a git ref to `origin`). -2. On machine B: clone the repo and run this `init` workflow. Step 3's fresh-clone recovery will see the Dolt refs on `origin` and `bd dolt pull` them automatically. - -Two machines writing at the same time is not supported in embedded mode; coordinate manually (push before handing off, pull before resuming). +4. **Hand off to plan.** Per the router's chaining rule, run the **plan** workflow next. It will detect that no epic exists yet and bootstrap one from `mission.md`. If `mission.md` is missing, **plan** will route the user back to **brainstorm**. ## Out of scope for this workflow - Creating issues or building the graph. That belongs to **plan**. - Writing `summary.md`. That belongs to **update-summary** (chained automatically after `plan`). - Re-running setup once a session is initialized. If `bd` or `jq` breaks later, fix it manually rather than re-running `init`. -- Server-mode beads (`bd init --server`) and any setup requiring a running Dolt sql-server. diff --git a/plugins/asta-preview/skills/research-step/workflows/plan.md b/plugins/asta-preview/skills/research-step/workflows/plan.md index c5ffb2d..85d0031 100644 --- a/plugins/asta-preview/skills/research-step/workflows/plan.md +++ b/plugins/asta-preview/skills/research-step/workflows/plan.md @@ -5,6 +5,8 @@ Create or extend the research graph. The single home for "design the next set of - **bootstrap** — no epic exists yet. Create the mission epic and the initial frontier (scope, definitions, literature_review) from `mission.md`. - **replan** — an epic exists. Add downstream tasks based on a recently-closed task's output, or on user direction. +**Per-template override.** Before bootstrap or replan, scan `workflows/plans/*.md` to see if any template's intro section matches `mission.md`'s described intent. If a template matches, **use its bootstrap + replan rules instead of the defaults below.** The default behavior (no template matches) is the hypothesis-driven flow documented here. + Always chains to **update-summary** afterward so `summary.md` reflects the new graph. ## Preconditions @@ -22,7 +24,8 @@ Every task issue carries: "research_step": { "task_type": "", "inputs": ["bd-xxxx", "bd-yyyy"], - "output_schema_version": 1, + "work_dir": ".asta///", + "output_schema_version": 3, "output": null } } @@ -30,6 +33,23 @@ Every task issue carries: The mission epic additionally carries `epic_root: true`. +**Choosing the slug.** When creating a task, pick a short kebab-case slug +that names the *specific instance* (not the task type — the type is +already in the path). Examples: + +| task_type | sensible slugs | +|---|---| +| `scope` | `aar-ela`, `glacier-mass-balance` | +| `definitions` | `aar-ela-terms` | +| `literature_review` | `provenance`, `theme-a-aar-geometry`, `theme-b-ela-bands` | +| `autods_run` | the autodiscovery run id (e.g. `9fc65db8`) | +| `replication` | `cross-dataset`, `cross-year`, `2026-05-otto-zeller` | +| `theorize` | `cross-cutting-mechanisms` | +| `autods_synthesis` | `findings` | + +After `bd create` returns the new id, `mkdir -p .asta///` +so executors can write into it immediately. + ## Mode selection 1. Run `scripts/epic-root.sh`. `status: none` → **bootstrap**. @@ -43,10 +63,17 @@ The mission epic additionally carries `epic_root: true`. bd create --type=epic --title="" --description="$(cat mission.md)" bd update --metadata '{"research_step":{"epic_root":true}}' ``` -3. **Create the initial frontier.** Three `task` issues with the metadata convention above: - - `scope: ` — `inputs: []` - - `definitions: ` — `inputs: []` - - `literature_review: ` — `inputs: [, ]` +3. **Create the initial frontier.** Three `task` issues with the metadata + convention above. Every `bd create` call must include both `--title` and + `--description`, written per `SKILL.md §"Voice for titles and descriptions"` + — imperative phrase ≤ 6 words for the title (e.g. "Frame the Alaska + glacier question", "Define the geometry terms we'll use"); one short + paragraph in plain ELI-college English for the description. + + Initial frontier: + - `scope` — `inputs: []` + - `definitions` — `inputs: []` + - `literature_review` — `inputs: [, ]` 4. **Add edges.** - `parent-child` from each frontier task to the epic - `blocks`: scope → definitions; scope → literature_review; definitions → literature_review @@ -54,39 +81,20 @@ The mission epic additionally carries `epic_root: true`. ## Replan mode -Read the source task's task_type and output: +Read the source task's `task_type` and `metadata.research_step.output`. Apply this table. Every newly-created task gets both `--title` and `--description` per the voice rules in `SKILL.md` — descriptions are the primary surface in the UI. -``` -bd show --json | jq '.[0].metadata.research_step.task_type' -bd show --json | jq '.[0].metadata.research_step.output' -``` -Apply this table: | Source task_type | Action | |---|---| -| `literature_review` | For each gap in `output.gaps`, create a `hypothesis` task with `inputs: [, ]`. Edges: `parent-child` to epic; `blocks` from the source. **Populate `metadata.research_step.output` at creation time** (see below) and close the issue immediately — the gap text already contains the statement, rationale, and prediction in prose, so there is no separate `execute` pass for hypotheses. | +| `literature_review` | For each gap in `output.gaps`, create a `hypothesis` task with `inputs: [, ]`. Edges: `parent-child` to epic; `blocks` from the source. | | `hypothesis` | Create the chain `experiment_design` → `evidence_gathering` → `analysis`, each `blocks` the next. `experiment_design` depends on the hypothesis (via `inputs`); `analysis` depends on both the hypothesis and the new `evidence_gathering`. All three get `parent-child` to the epic. | | `analysis` | If every `hypothesis` in the epic now has a closed `analysis`, create one `synthesis` task with `inputs` listing all analysis IDs and the scope ID. `parent-child` to epic; `blocks` from each analysis. Otherwise no-op. | -| `synthesis` | If `output.open_questions` is non-empty, **stop and ask the user** before creating new `hypothesis` tasks. If approved, create them with a `discovered-from` edge back to the synthesis (in addition to the usual edges). | +| `synthesis` (or any `*_synthesis` closing task) | If `output.open_questions` is non-empty, **stop and ask the user** before creating new `hypothesis` tasks; if approved, create them with `discovered-from` edges back. Otherwise no replan — the synthesis is the closing artifact. | | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | No replan. Report no-op and stop. | If invoked without a source task and the user has not specified what to plan, do not invent work — ask, or stop. -### Auto-resolving hypothesis tasks - -When creating a `hypothesis` from a literature_review gap: - -1. Derive the four output fields directly from the gap text and surrounding `literature_review` output (`bd show --json | jq '.[0].metadata.research_step.output'`): - - `statement` — `H_n: ` - - `rationale` — why this gap implies the claim - - `falsifiable_prediction` — what observation would refute it - - `expected_evidence` — list of concrete evidence types that would support it -2. Validate with `scripts/validate-output.sh hypothesis ` before persisting. -3. Persist with `scripts/write-meta.sh` + `bd update --metadata @`, then `bd close `. - -If a gap is too thin to fill these fields without inventing content, **do not auto-resolve** — leave the hypothesis open and surface it to the user. Genuine ambiguity is the one case where a separate `execute` pass is warranted. - ## After either mode Hand off to **update-summary** so `summary.md` reflects the new state. diff --git a/plugins/asta-preview/skills/research-step/workflows/plans/autods-to-theorizer.md b/plugins/asta-preview/skills/research-step/workflows/plans/autods-to-theorizer.md new file mode 100644 index 0000000..ba8ab72 --- /dev/null +++ b/plugins/asta-preview/skills/research-step/workflows/plans/autods-to-theorizer.md @@ -0,0 +1,351 @@ +# Template: autods-to-theorizer + +AutoDS run → literature-grounded findings → focused cross-dataset replication → cross-cutting theorize → contextualized synthesis (always-`findings.md` closing artifact). + +**When this template is used.** `plan.md` selects this template by reading `mission.md` prose and recognizing AutoDS-flavored intent — e.g., the mission mentions "AutoDS run …", references a UUID-shaped run id, points at a downloaded export directory, or describes a fresh autodiscovery configuration. No frontmatter is required on `mission.md`; the agent reads the prose to decide. + +**Where the spec lives.** The Mermaid diagram below is the control-flow spec (task types, ordering, gates). The output schema for every new task type — including non-obvious gotchas (file paths, field names, decision thresholds, failure handling) — lives in `assets/schemas.yaml`. The three pieces of guidance below the diagram (theorize failure semantics, theorize query composition, and findings.md tone) don't fit naturally in either source. + +```mermaid +flowchart TD + %% Replan rules: each task close triggers creation of its successor(s) in the diagram. + %% Standard edges (parent-child to epic, blocks from upstream) are implied. + %% All `asta ...` calls should prefix `unset ASTA_GATEWAY_URL &&` (see init.md). + + %% ════ Bootstrap (plan.md reads mission.md prose to pick this template) ════ + mission[/"mission.md"/]:::data --> scope[scope]:::task + scope --> defs[definitions]:::task + + %% ════ AutoDS — sourcing the run ════ + defs --> autods[autods_run]:::new + autods --> aMode{resolve source
from mission.md prose}:::decision + aMode -->|run id found| aImport[asta autodiscovery
experiments]:::action + aMode -->|local export
path found| aLocal[parse local export
mcts_nodes_all.json + metadata]:::action + aMode -->|fresh-run intent| aExec[asta autodiscovery
create / upload / submit;
block to SUCCEEDED]:::action + aMode -.->|ambiguous /
silent| aAsk[/"ask user:
which source?"/]:::usergate + aAsk -.-> aMode + + %% ════ Literature ════ + aImport --> prov[literature_review
provenance]:::task + aLocal --> prov + aExec --> prov + prov --> themeCluster[cluster hypotheses
into 3-6 themes]:::action + themeCluster --> themes[literature_review × N
per theme; parallel]:::task + themes --> litsyn[autods_literature_synthesis]:::new + + %% ════ Post-lit gates ════ + %% surviving_findings filter: see schemas.yaml autods_literature_synthesis.surviving_findings comment. + litsyn --> qSurv{all surprisals
filtered as known?}:::decision + litsyn --> qLitContra{any AutoDS finding
contradicted by lit?}:::decision + + qSurv -->|yes| askDS[/"ask user:
provide replication dataset
or skip?"/]:::usergate + askDS -->|provide| updMis[update mission.md:
replication_dataset_path]:::action + askDS -->|skip| syn + qSurv -->|no| repro + + qLitContra -->|yes| hrContra[/"human_review:
contradicted findings"/]:::usergate + qLitContra -->|no| repro + hrContra -->|blocks| repro + + updMis --> repro + + %% ════ Replication ════ + repro[replication]:::new + + %% ════ Post-replication gate ════ + %% Confidence anchors + threshold rationale: see schemas.yaml replication.findings[].confidence. + repro --> qFailed{any failed_to_replicate
with confidence > 0.7?}:::decision + qFailed -->|yes| hrFailed[/"human_review:
failed replications"/]:::usergate + qFailed -->|no| theo + hrFailed -->|blocks| theo + + %% ════ Theorize ════ + %% CLI flags, context-window contract, artifacts export: see schemas.yaml `theorize` + %% task-type comment. Query-composition template: see below. + %% No agent-side fallback. If generate-theories does not return theories, the task and + %% the epic are marked status=failed and the run stops for human review. + theo[theorize]:::new + theo --> qTheoOk{generate-theories
returned theories?}:::decision + qTheoOk -->|yes| syn + qTheoOk -->|no| theoFail[bd update theorize-id --status=failed
bd update epic-id --status=failed
STOP for human review]:::action + + %% ════ Synthesis (ALWAYS runs — closing artifact on every path) ════ + %% Universal closing task. Theory-centric for autods flows: leads with the + %% theories generated by the upstream `theorize` task, contextualized with + %% the surviving AutoDS patterns and replication evidence. Schema is the + %% generic `synthesis` (see schemas.yaml); tone + outline guidance below. + syn[synthesis]:::new + syn --> closeOk(("close epic")):::terminal + syn -.->|V2 only:
open_questions AND
discovery_loop| askDisc[/"ask user:
spawn sub-epics?"/]:::usergate + askDisc -.->|select| spawn[spawn_subepic × K]:::action + askDisc -.->|decline| closeOk + + classDef task fill:#e1f5ff,stroke:#0288d1,color:#0c1024 + classDef new fill:#fff4e1,stroke:#d97706,color:#0c1024 + classDef decision fill:#fff9c4,stroke:#f9a825,color:#0c1024 + classDef usergate fill:#ffccbc,stroke:#d84315,color:#0c1024 + classDef action fill:#f0f4c3,stroke:#9e9d24,color:#0c1024 + classDef terminal fill:#e0e0e0,stroke:#616161,color:#0c1024 + classDef data fill:#ecfccb,stroke:#3f6212,color:#0c1024 +``` + +## Task creation: titles and descriptions + +Every task this template spawns gets both a `--title` and `--description` on +`bd create`, written per `SKILL.md §"Voice for titles and descriptions"`. +Read that section before creating tasks; the rules apply unchanged. As a +sanity-check, here are the titles this template should produce on a typical +run — read them top-to-bottom and they should sound like a paragraph +narrating the workflow: + +Each title names the **specific subject** of the step, not just the task +type. "Define the geometry terms" is too generic; "Define snowline, shape, +and timing terms" tells the reader which terms. + +| task_type | example title (imperative, names the subject, ≤ 6 words) | +|---|---| +| `scope` | "Frame Alaska glacier shape-vs-snowline question" | +| `definitions` | "Define snowline, shape, and timing terms" | +| `autods_run` | "Import 51 AutoDS-flagged glacier patterns" | +| `literature_review` (provenance) | "Find source papers for both datasets" | +| `literature_review` (theme A) | "Search literature on observed-vs-predicted snowline" | +| `literature_review` (theme B) | "Search literature on shape-based AAR predictors" | +| `literature_review` (theme C) | "Search literature on regional volume-area scaling" | +| `literature_review` (theme D) | "Search literature on off-glacier subpopulations" | +| `autods_literature_synthesis` | "Pick patterns literature can't explain" | +| `replication` | "Re-run surviving patterns on different year" | +| `theorize` | "Propose theories linking surviving glacier patterns" | +| `synthesis` (closing) | "Write closing report on the theories" | + +Bad titles this template *will* drift toward if you let it (do not write +these — `SKILL.md` enumerates the full ban list): +`scope: research question, boundaries, and success criteria for ...`, +`literature_review: theme A — observed-vs-hypsometric ELA discrepancy (6 surprises)`, +`synthesis: findings.md — closing artifact for run 9fc65db8`. + +## Replication: faithful-attempt requirement + +`replication` is the load-bearing empirical step — don't short-circuit it. +"Replication" here = re-test the surviving finding on a *different* cut of +the data (different year, different dataset, different join), per the NSF / +NASEM "replicability" definition. Not same-data-same-analysis reproduction. + +Before marking any candidate dataset `failed` or any finding `inconclusive` / +`unreachable`, walk this list and record the URL you tried for each: + +1. The source paper's data-availability statement (this is the first move, + not the last — Copernicus, Cambridge, AGU, EGU, Nature DASes almost always + name a stable repository DOI). +2. Zenodo, USGS ScienceBase, PANGAEA, ETH Research Collection, NSIDC. +3. Author / lab GitHub (committed data or release assets; check LFS). +4. Publisher supplementary materials. + +`fetch_status: failed` requires URL + one-line failure reason in `rationale` +or `deviations[]`. "I didn't try" is not "I tried and failed." + +Each per-finding `.asta/replication///` must contain a +runnable `replicate.py`, its input data, figures, a `log.md` of captured +numbers, and an `assessment.md` whose claims cite the script. A directory +with only `assessment.md` is not a valid replication — leave the task +`in_progress` and come back. + +**Anti-patterns that block close:** prose like "would test", "could be done", +"deferred to a follow-up"; using literature-consistency as the verdict +basis (the literature is the prior, not the verdict); marking all candidates +`failed` without attempted URLs; `confidence < 0.5` across all findings. + +## Theorize: failure semantics + +There is **no agent-side fallback**. The theorize task is satisfied +only by a successful run of the remote `asta generate-theories` +agent that returns a populated `theories` list. If the agent fails +(PaperFinder error, network failure, non-zero exit, empty +`theories`), the executor must: + +1. **Not** close the theorize task. +2. **Not** synthesize theories from the in-context evidence. +3. Mark the theorize task as failed: + `bd update --status=failed --notes ""`. +4. Propagate the failure to the epic: + `bd update --status=failed --notes "theorize did not return theories; see "`. +5. Stop the run. Do not proceed to `synthesis`. Surface the failure + to the user and wait for human review. + +The `failed` status is registered as a custom status with category +`done` by `workflows/init.md` (`bd config set status.custom "failed:done"`). +It is terminal: a `failed` task is not retried automatically. A +human operator can re-open it with `bd update --status=in_progress`, +fix the upstream condition, and re-run. + +## Theorize: query composition + +The remote `asta generate-theories` agent cannot read bd metadata, local files, or anything it didn't receive in its inputs. The `--theory-query` string IS its context window. Pack it carefully; inline information; cite supporting papers by author/year inline, not by corpus_id. Suggested shape: + +> Find cross-cutting theories that could explain these empirical findings about \, in the context of the literature gaps below. +> +> EMPIRICAL FINDINGS: \. +> +> LITERATURE GAPS: \. Note any methodological tensions explicitly (these are theory-bait). +> +> \ +> +> Surface theories that: (a) \; (b) \; (c) \. Use the supplied paper store. + +## Inline citations (template-specific targets) + +SKILL.md covers the baseline patterns. Template-specific link targets: + +All paths below are relative to the task's own `work_dir`. `../..///` reaches a sibling task's work_dir. + +- `autods_literature_synthesis` per-hypothesis table: each row's classification → the relevant theme summary at `../../literature_review/theme-/summary.md`; each node_id → `../../autods_run//experiments.json`. +- `theorize` per-theory markdown: supporting passages → `novelty/.md` and `extractions/.md` (same work_dir). +- closing `synthesis` `report.md`: cite the upstream theory markdowns + (`../../theorize//theories/.md`) as the source of each theory + claim, the per-finding pages (`../../replication///assessment.md`) + as replication evidence, and the theme summaries + (`../../literature_review/theme-/summary.md`) for literature grounding. + Don't cite the raw datasets unless the claim is about that data directly. A + link-rich `report.md` makes a trailing "Files in this run" appendix + redundant; drop it. + +## Synthesis: inputs and spawn + +When `plan.md` spawns the closing `synthesis` task at the end of an autods +flow, its `metadata.research_step.inputs` lists every upstream task the +agent will read while writing `report.md`: + +``` +inputs: [ + , + , + , # omit if replication was skipped + , # always included — theorize seeds the theories report.md leads with +] +``` + +`theorize` is the load-bearing upstream and a hard precondition for +`synthesis`: the closing `synthesis` task is only spawned after +`theorize` is **closed** with a populated `theories` list. If +`theorize` is `status=failed` (see "Theorize: failure semantics" +above), `synthesis` is not spawned, the epic is also `status=failed`, +and the run stops for human review. + +The agent then writes `report.md` around the theories `theorize` +produced, with replication status and literature grounding as +supporting evidence under each theory. + +## Synthesis: tone and shape for `report.md` + +The closing `synthesis` task uses the generic `synthesis` schema (see +`schemas.yaml`). For autods-to-theorizer runs the deliverable is +**theory-centric**: the headline answer (in `output.answer`) names a theory, +and `report.md` is structured around the theories the upstream `theorize` +task generated. The replication funnel (how many AutoDS patterns survived +literature, how many replicated) is supporting evidence under each theory, +not the lead. + +### Who you're writing for + +A **college-educated reader who has taken the 101 course in the broad +subject area but is not a specialist**. They've heard of the field's key +ideas but don't carry its jargon. They want to follow the run end-to-end +and understand what the theories say and why they're interesting. + +Build context before delivering theories. The first time a domain term +appears, gloss it in a parenthetical or a quick sentence ("AAR — the +fraction of the glacier above its snowline, used as a steady-state proxy"). +After the gloss, you can use the term normally. Lead each section with what +a 101-level reader needs to know to follow the next paragraph; specialist +detail (effect sizes, p-values, exact citations) can sit one paragraph +deeper for readers who want it. + +Walk the reader through the flow: + +1. What question motivated the run, in everyday English. +2. What patterns AutoDS surfaced (without listing all 51 — just the shape + of what came out). +3. How literature recalibrated which of those patterns were genuinely + surprising. +4. Which patterns held up on the secondary dataset. +5. The theories the run produced — these are the deliverable. Explain + each theory's mechanism in plain language before showing its grounding. +6. Where this leaves us — next directions. + +Stay **neutral and factual** about results. Avoid hype, avoid hedge: + +- "did not replicate when we reran the underlying statistics" rather than + "AutoDS got it wrong" +- "the result depends on which baseline assumption is used" rather than + "contaminated" +- present findings that didn't replicate as honest scientific results, not + as workflow failures + +Keep all the numbers, statistical specifics, and caveats — just don't +editorialize. + +### Don't talk about the reader + +The reader is established here, for *your* writing decisions. Once you +know who you're writing for, write at that level — don't put the level +into the report. Telegraphing the level reads as patronizing even when +the underlying explanation is good. + +| Bad | Why | Fix | +|---|---|---| +| `# What the AutoDS run found about Alaska glaciers — and what to do with it` | Titles the *process*, not the *finding*. Reads as a meta-frame. | State the actual finding plainly: `# Alaska glacier snowlines diverge from textbook predictions where ice is slow and debris-covered`. | +| `## Background — for a reader who has heard of glaciers` | "For a reader who has heard of X" telegraphs the level mid-page and reads as condescending. | `## Background`. | +| `**In plain English.** ...` | Labels the explanation as plain instead of just being plain. | Drop the label; lead with the sentence. | +| `## Caveats — if you're new to glaciology` / `## For the non-specialist` | Same patronizing pattern in different wrapping. | Drop the qualifier from the header. | + +**Title rule.** The H1 of `report.md` states the *finding*, not the +*run*. It should be a single approachable sentence a casual reader can +quote as the takeaway — same content as `output.answer`, optionally +sharpened for headline use. Avoid `# What X found about Y`, +`# Findings from `, `# Closing report on `. The Headline +section that immediately follows the H1 carries the 1-2 sentence +elaboration; don't duplicate. + +**Section-label rule.** Section headers should name the thing in the +section, full stop. `## Background`, `## What AutoDS surfaced`, +`## Theories`, `## Caveats`, `## Next directions`. No "for the +casual reader" qualifiers, no level indicators, no audience asides. + +**Inline rule.** Don't preface explanations with `**In plain English.**`, +`**Put simply,**`, `**For the non-expert,**`, or similar. If the +sentence is plain, the label is redundant; if the sentence is not, the +label doesn't fix it. + +`report.md` lives at `/report.md` (e.g. +`.asta/synthesis/findings/report.md`). It is the run's closing artifact; +downstream consumers (Asta Flows panel, future tooling) read it directly. + +**Suggested outline** (compress what isn't load-bearing for the specific +run): + +1. **Headline** — one or two sentences naming the top-line theory or + answer; the same content as `output.answer`. +2. **Background** — two or three sentences setting up the domain question + in 101-level terms before any specialist content. +3. **What AutoDS surfaced** — short prose summary of the run's shape (how + many hypotheses, broad themes), not a per-node list. +4. **Theories** — one subsection per theory from the upstream `theorize` + task. Each subsection in this order: (a) one-sentence plain-English + statement of the theory, (b) the mechanism sketch a 101-level reader + could follow, (c) the supporting AutoDS patterns with their + replication status, (d) supporting paper citations, (e) the novelty + class. +5. **Replication evidence** — table or bullets summarizing which AutoDS + patterns survived literature grounding and which replicated on the + secondary dataset. Reference the per-finding `assessment.md` files. +6. **Caveats** — methodological tensions, unreachable patterns, + and any human-review interventions that unstuck the run + (e.g., re-opening a failed `theorize` after fixing the + remote-service issue). +7. **Next directions** — what the next AutoDS run could profitably explore; + these populate `output.open_questions`. + +## Out of scope (V1) + +- Design-and-test phase (auto-exp-designer-driven loop) — deferred. +- Canonical-data reproduction (rerun AutoDS test on AutoDS source data as a same-data correctness audit, distinct from the `replication` step's different-data semantics) — V1 is cross-dataset replication only. +- Discovery loop (spawning sub-epics from `open_questions`) — scaffolding only. diff --git a/plugins/asta-preview/skills/research-step/workflows/update-summary.md b/plugins/asta-preview/skills/research-step/workflows/update-summary.md index a79f6ff..28dbce5 100644 --- a/plugins/asta-preview/skills/research-step/workflows/update-summary.md +++ b/plugins/asta-preview/skills/research-step/workflows/update-summary.md @@ -44,7 +44,7 @@ Regenerate `summary.md` from beads. Idempotent and safe to run anytime. This is ## Related Work - + ## Hypotheses " plus current verdict from its analysis if closed> diff --git a/plugins/asta/skills/research-step/SKILL.md b/plugins/asta/skills/research-step/SKILL.md index 0d2fcee..e27f3eb 100644 --- a/plugins/asta/skills/research-step/SKILL.md +++ b/plugins/asta/skills/research-step/SKILL.md @@ -23,7 +23,160 @@ Installing `bd` and `jq`, running `bd init`, and verifying `scripts/summary-chec | `mission.md` | Input. The research task. | | `.beads/` | Source of truth for state. | | `summary.md` | Derived view of the session, regenerated by **update-summary**. Beads is the source of truth; this file is just a digest for humans and for **brainstorm**. Frontmatter `beads_snapshot` records the state it was rendered from. | -| `background_knowledge.txt` | Optional. Long-form context referenced from issue metadata via `summary_path`. | +| `.asta///` | One **work_dir** per task. The agent picks the kebab-case slug at task creation; the full path is stored at `metadata.research_step.work_dir`. Every file the task writes goes here, following the per-task-type filename conventions in `assets/schemas.yaml > file_conventions`. | + +## Per-task work_dir convention + +Each task issue declares one working directory: + +``` +metadata.research_step.work_dir = ".asta///" +``` + +- `` is the schema task type (e.g. `literature_review`). +- `` is a short kebab-case name the agent picks at create-time + (e.g. `provenance`, `theme-a-aar-geometry`, `cross-dataset`). +- Every file the task writes lives **under** this directory. Sibling + directories (`.asta/literature_review/theme-b-ela-bands/`) are independent. +- The filenames inside follow `assets/schemas.yaml > file_conventions` + (e.g. lit_review always writes `summary.md`; synthesis writes + `report.md`) so downstream consumers can find them without reading + paths out of metadata. + +The Asta Flows UI renders this directory as a file browser on the task's +**Files** tab. No `_path` fields are required; the directory IS the manifest. + +## Markdown convention: inline citations + +Any markdown file a task writes inline-links to the artifacts that back its +statements. The Asta Flows panel auto-linkifies bd ids, AutoDS node ids, and +run-relative paths; explicit Markdown links handle the rest. + +Use real markdown links for every: + +- **Paper mention** → `[Author+YY](https://www.semanticscholar.org/paper/)`. + Never write a bare `corpusId 12345` or `doi:10.1234/...` in prose. +- **Numeric claim or verdict** → link to the script/log/table that produced + it, e.g. `[r=+0.22, p<1e-30](reproduce.py)` (relative to the task's own + work_dir) or `[reproduced](../../data_reproduction//per-finding-table.md#node_3_2)` + (relative path into another task's work_dir). +- **Cross-reference to another task's output** → relative path from this + task's work_dir into the upstream task's work_dir (e.g. + `../../literature_review/theme-a-aar-geometry/summary.md`). +- **Cross-reference to another bd task** → bare bd id; the renderer + auto-linkifies it. +- **AutoDS hypothesis / experiment** → `../../autods_run//experiments.json`. + +Per-template tweaks live in `workflows/plans/.md`. + +## Voice for titles and descriptions + +Every issue in beads must have a `--title` and a non-empty `--description`. +The Asta Flows UI surfaces both as the primary content of each task. + +**Imagine the reader.** A smart college graduate scrolling the Tasks page +who has never heard of this run, this domain, or these acronyms. They are +not stupid — they have a degree — but they have zero context. Your titles +and descriptions are how they figure out what this run is doing. + +### Titles — imperative phrase, ≤ 6 words + +Lead with a verb and name the **specific subject** of the step in plain +English — don't stop at a generic reframing of the task type. "Define the +terms" is too vague; "Define the snowline, shape, and timing terms" is +right. The reader should be able to guess *what the task is about*, not +just *what kind of task it is*. + +Stack the titles top-to-bottom and they should narrate the run like a +paragraph that names the actual subject matter at each step: + +> Frame the Alaska glacier shape-vs-snowline question. Define the snowline, +> shape, and timing terms. Import 51 AutoDS-flagged glacier patterns. Find +> source papers for both datasets. Search literature on observed-vs-predicted +> snowline. Pick the patterns literature can't explain. Re-run the surviving +> patterns on a different year. Propose theories linking the surviving +> patterns. Write the closing report on what survived. + +Forbidden in titles — none of these belong anywhere in `--title`: + +| Pattern | Why it's wrong | Fix | +|---|---|---| +| `scope: …`, `literature_review: …`, `synthesis: …` | The task_type already appears as a chip in the UI. Repeating it eats half the title. | Drop the prefix. | +| `(6 surprises)`, `(2 surprises)` | Run-internal counts that read like meta-notes. | Drop. The number can live in the description. | +| `— observed-vs-hypsometric ELA discrepancy` | Field-specialist jargon (AAR, ELA, hypsometric, Lmax, surprisal). A non-expert can't parse it. | Use everyday English. "Compare textbook glacier shape to what's observed." | +| `findings.md`, `experiments.json`, `task.json` | Filenames in the title. | Describe what's inside the file, not the filename. | +| `theme A`, `theme B`, `node_2_4` | Run-internal labels. | Name the *content*: "glacier-shape predictors of AAR" not "theme A". | + +Before / after on the current run. The "after" column names the actual +subject at each step, not just the task type: + +| Before (don't do) | After (do this) | +|---|---| +| `scope: research question, boundaries, and success criteria for Alaska glacier AAR/ELA geometry synthesis` | `Frame Alaska glacier shape-vs-snowline question` | +| `definitions: operational definitions of ELA, AAR, geometry features, response-time terms` | `Define snowline, shape, and timing terms` | +| `autods_run: import local export 9fc65db8 (51 MCTS nodes, 16 surprising) into canonical experiments.json` | `Import 51 AutoDS-flagged glacier patterns` | +| `literature_review: provenance of TableA1_data.csv (Otto+25) and alaska_elas.csv (Zeller+25)` | `Find source papers for both datasets` | +| `literature_review: theme A — observed-vs-hypsometric ELA discrepancy (6 surprises)` | `Search literature on observed-vs-predicted snowline` | +| `literature_review: theme B — geometric predictors of observed AAR and AAR-hypsometric complexity` | `Search literature on shape-based AAR predictors` | +| `literature_review: theme C — regional and volume-area scaling structure` | `Search literature on regional volume-area scaling` | +| `literature_review: theme D — off_glacier_flag subpopulation morphology` | `Search literature on off-glacier subpopulations` | +| `autods_literature_synthesis: aggregate provenance + 4 themes into surviving_findings list` | `Pick patterns literature can't explain` | +| `data_reproduction: cross-year Zeller+25 replication of 6 surviving findings + Otto+25 Zenodo geometry source` | `Re-run surviving patterns on different year` | +| `theorize: cross-cutting mechanisms for the 6 surviving findings (fallback synthesis, remote CLI unreachable)` | `Propose theories linking surviving glacier patterns` | +| `autods_synthesis: findings.md — closing artifact for run 9fc65db8` | `Write closing report on what survived` | + +### Descriptions — one short paragraph, ELI-college + +Two to four sentences. One continuous paragraph (no Markdown headers, +bullets, or bold). The paragraph answers, for a curious non-expert reader: + +1. **Why** is this task here? (What question is it answering for the run?) +2. **What** will it produce, in everyday language? + +Forbidden in descriptions: + +- **Raw schema or AutoDS labels.** Don't paste `node_2_0`, `node_3_15`, + `gwi_feq`, `Lmax`, `ela_hyps`, `CLASSIFICATION:`, `surviving_findings` + into the description prose. If you need to refer to "the six surprising + findings", say *"the six surprising findings"* — the per-finding details + belong in the output JSON and the markdown report, not in the + description. A reader hitting `node_2_0` in the panel has no idea what + it means. +- **Internal field/file names.** No `findings.md`, no `experiments.json`, + no `output.surviving_findings`. Describe the *thing*, not the schema. +- **Paper citations (Smith+25, Otto et al. 2025).** Citations live in the + literature output. The description shouldn't double as a bibliography. +- **Meta narration.** "In this task we will…", "Let me…", "I'll now…", + "As an AI…", "Here is what we did". +- **LLM filler.** "deep dive", "leverage", "unlock", "delve", scare + quotes, excessive em dashes. +- **bd ids.** Reference upstream work by what it *does* ("the literature + review on glacier shape"), not its bd id. + +Before / after, same Theme-A task: + +> ❌ *This per-theme literature review classifies the AutoDS surprises about +> the gap between observed end-of-summer ELA (Zeller+25 alaska_elas.csv) and +> its theoretical / hypsometric counterparts. Six of the run's 16 surprising +> nodes fall in this theme: 2_0 (slope vs lela - Zelal), 3_4 (ela - Zela vs +> gwi_feq), 3_15 (hypsometric ELA correlation by glacier size), 4_10 (lela - +> ela_hypsl by area quartile), 4_15 (slope vs ela - ela_hyps), and 3_12 +> (Lmax vs ELA directly). Each node is tagged with a CLASSIFICATION line +> (textbook, construction-artifact, candidate-surprise, novel-extension, +> inconclusive) so the autods_literature_synthesis can compute +> surviving_findings.* + +> ✅ *Six of the surprising patterns AutoDS flagged are about a mismatch +> between the snowline glaciologists observe in the field and the one you'd +> predict from each glacier's shape. This task checks each pattern against +> the standard glaciology textbooks: is it a known relationship, a side +> effect of how the dataset was built, or something genuinely new? The +> verdict for each pattern is the input to the next step, which compiles +> what survived as worth a closer look.* + +The "after" version is shorter, uses everyday English, and never references +a node id or schema field. A college reader can follow it; an expert still +gets the same information. ## Workflows @@ -35,7 +188,7 @@ Installing `bd` and `jq`, running `bd init`, and verifying `scripts/summary-chec | **execute** | Run one ready task end-to-end. Hands off to **plan** when the closed task type unlocks new structure; otherwise to **update-summary**. | `workflows/execute.md` | | **update-summary** | Regenerate `summary.md` from beads. Idempotent — no-op when `scripts/summary-check.sh` reports `status: fresh`. | `workflows/update-summary.md` | -Task-type schemas live in `assets/schemas.yaml`. +Task-type schemas live in `assets/schemas.yaml`. **Per-template overrides** live in `workflows/plans/.md` — each template file holds its bootstrap rules, replan rules, and per-task-type execution prose. Templates are selected by `plan.md` reading `mission.md` prose and matching against the templates' intro sections (no frontmatter required). The default (no template matches) is the hypothesis-driven flow described across `workflows/{plan,execute,update-summary}.md`. ## Routing diff --git a/plugins/asta/skills/research-step/assets/schemas.yaml b/plugins/asta/skills/research-step/assets/schemas.yaml index b840628..431cb10 100644 --- a/plugins/asta/skills/research-step/assets/schemas.yaml +++ b/plugins/asta/skills/research-step/assets/schemas.yaml @@ -1,8 +1,43 @@ # Output schemas for research-step task types. -# Each task issue stores its realized output at metadata.research_step.output, -# matching the shape under `output:` for its task_type. +# +# Each task issue stores its realized output under metadata.research_step: +# +# metadata.research_step: +# task_type: +# inputs: [bd_id, ...] # concrete upstream task bd ids +# work_dir: ".asta///" # the single directory this task wrote to. +# # is a short kebab-case name the agent +# # picks at create time. Asta Flows reads this +# # to render the Files tab. +# output: { ...per-type structured fields, schema below... } +# +# What's NOT in here anymore (schema_version 3 hard cut): +# - `artifacts: []` and `consumes: []` uniform fields (M0.5) — replaced by `work_dir`. +# - Per-type `_path` / `_dir` legacy fields (`summary_path`, `experiments_path`, +# `report_path`, etc.) — files live under `work_dir`; canonical filenames +# are listed in the per-task convention block below so downstream tasks +# can find them without metadata pointers. +# +# Inline-citation rule for markdown outputs: see SKILL.md §"Markdown convention". -schema_version: 1 +schema_version: 3 + +# ── Per-task-type file conventions inside `work_dir` ─────────────────────────── +# Agents write predictable filenames under their work_dir so downstream tasks +# can find them without consulting metadata paths. +file_conventions: + scope: ["scope.md"] + definitions: ["terms.md"] + literature_review: ["summary.md", "thread/"] # thread/ is the asta-literature interactive workspace + hypothesis: ["hypothesis.md"] + experiment_design: ["design.md"] + evidence_gathering: ["log.md", "data/", "figures/"] + analysis: ["analysis.md"] + synthesis: ["report.md"] + autods_run: ["experiments.json", "run-metadata.json", "dataset-summary.md"] + autods_literature_synthesis: ["per-hypothesis-table.md", "summary.md"] + data_reproduction: ["per-finding-table.md", "/"] # one subdir per surviving node + theorize: ["task.json", "theories/", "novelty/", "citations.bib", "index.md"] task_types: @@ -24,14 +59,28 @@ task_types: literature_review: inputs: [scope, definitions] output: - summary_path: string # relative path; long-form context - key_findings: [string] # 3-10 bullets readable without opening summary_path + key_findings: [string] # 3-10 bullets readable without opening summary.md. + # autods-to-theorizer convention: per-theme lit_reviews include a + # "CLASSIFICATION: = " line per + # AutoDS hypothesis covered by the theme. gaps: [string] # gaps that motivate hypotheses citations: - id: string title: string url: string relevance: string + # ── optional fields used by autods-to-theorizer; hypothesis-driven leaves absent ── + provenance: # populated when this lit_review is the provenance phase. + source_papers: + - corpus_id: string + doi: string + title: string + authors: [string] + year: number + dataset_lineage: string + theme: string # populated when this lit_review is a per-theme task + corpus_ids: [string] # harvested by per-theme tasks; theorize unions these into paper_store hypothesis: inputs: [scope, literature_review] @@ -50,31 +99,161 @@ task_types: independent: [string] dependent: [string] controls: [string] - artifacts_expected: [string] # paths the gathering step will produce + artifacts_expected: [string] # filenames the gathering step is expected to write (under its own work_dir) evidence_gathering: inputs: [experiment_design] output: - artifacts: - - path: string - kind: string # data | log | figure | code | other - description: string - log_path: string # what was actually run deviations: [string] # ways execution diverged from design analysis: inputs: [hypothesis, evidence_gathering] output: verdict: enum [supported, refuted, inconclusive] - confidence: number # 0.0 - 1.0 + confidence: number # 0.0 - 1.0; agent's credence in the verdict reasoning: string caveats: [string] synthesis: - inputs: [scope, analysis_*] # all analysis issues in the epic + # The universal closing task for every template. Writes report.md as the + # run's deliverable. Two flavors: + # + # - Hypothesis-driven flow. + # inputs: [scope, analysis_*] # every analysis task in the epic + # The agent answers scope.question and lists which hypotheses were + # supported / refuted (by bd id). + # + # - AutoDS / theorize-driven flow. + # inputs: [autods_run, autods_literature_synthesis, theorize, + # (optional) data_reproduction] + # The agent leads with the theories from the upstream `theorize` + # task, curates them in report.md, references the AutoDS patterns + # and literature grounding that seeded each theory, and notes + # reproduction status when data_reproduction ran. Both + # supporting_hypotheses and refuted_hypotheses may be empty (or + # carry AutoDS node ids as strings if the agent finds that + # useful) — theories are the real centerpiece and live in + # report.md. + # + # Either way, the run's headline is in `answer` and any follow-up + # threads live in `open_questions`. + inputs: [scope, analysis_*] # hypothesis-driven flow; + # autods flow uses [autods_run, autods_literature_synthesis, theorize, data_reproduction] + output: + answer: string # one-line "so what" — names a theory for theory-driven runs, + # or the answer to scope.question for hypothesis-driven runs + supporting_hypotheses: [bd_id] # hypothesis-flow: bd ids of supported hypotheses. + # autods-flow: leave empty (theories live in report.md prose). + refuted_hypotheses: [bd_id] # symmetric counterpart; empty for autods flows + open_questions: [string] # next directions; become discovered-from edges on re-plan + + # ─────────────────────────────────────────────────────────────────────────────── + # autods-to-theorizer template task types + # ─────────────────────────────────────────────────────────────────────────────── + + autods_run: + inputs: [scope, definitions] + # Three sourcing modes, selected by what mission.md describes (no frontmatter needed): + # A1: UUID-shaped run id → `asta autodiscovery experiments --format json` + # A2: local export dir → parse /output/mcts_nodes_all.json (the STRUCTURED file; + # the node_*.json files are autogen conversation logs, don't use those) + # B: fresh-run intent → `asta autodiscovery create/upload/metadata/submit`; block until SUCCEEDED. + # Confirm with user before submitting (uses credits, 30+ min). + # If mission.md is ambiguous between modes, ask the user before doing anything expensive. + # Field-name gotcha: autodiscovery's records use `surprising` (boolean) and `normalized_surprisal` + # (number, can be negative — it's a log-likelihood-ratio, not a 0-1 probability). Map these into + # is_surprising and surprise_score below. + output: + run_id: string # A1: from mission; A2: basename of path; B: new ID + mode: enum [imported, executed] + dataset_summary: + datasets: + - name: string + description: string + n_rows: number + columns: [string] + intent: string + domain: string + hypotheses: + - node_id: string # e.g., node_2_1 + statement: string + prior_analysis: string # truncate inline; keep full text on disk + is_surprising: boolean # ← autodiscovery's `surprising` field + surprise_score: number # ← autodiscovery's `normalized_surprisal` field + surprising_count: number + + autods_literature_synthesis: + inputs: [autods_run, "literature_review_*"] # 1 provenance + N theme output: - answer: string # answer to scope.question - supporting_hypotheses: [bd_id] - refuted_hypotheses: [bd_id] - open_questions: [string] # become discovered-from edges on re-plan - report_path: string # generated markdown report + themes: + - name: string + contributing_lit_review_ids: [bd_id] + key_findings: [string] + gaps: [string] + surviving_findings: [string] # AutoDS node_ids where hypothesis.is_surprising == true AND + # lit-review CLASSIFICATION ∈ {candidate-surprise, novel-extension}. + # Drives the post-lit gates in the diagram. + contradicted_findings: [string] # AutoDS node_ids explicitly refuted by lit + headline: string # one-line: "3 of 10 AutoDS findings remain still-surprising after lit" + + data_reproduction: + inputs: [autods_run, autods_literature_synthesis] + # Skip if surviving_findings is empty (replan handles). + # Faithful-attempt requirement + workflow guidance: see autods-to-theorizer.md. + output: + candidate_datasets: + - name: string + source_url: string + rationale: string # why "similar" + fetch_status: enum [ok, partial, failed] + findings: + - node_id: string + verdict: enum [reproduced, failed_to_reproduce, partial, inconclusive, unreachable] + # `reproduced`: same direction + comparable effect size + # `failed_to_reproduce`: clearly different (opposite sign or null effect) + # `partial`: directionally consistent but effect differs substantially + # `inconclusive`: test runs but is underpowered or ambiguous + # `unreachable`: all candidate datasets failed to fetch + confidence: number # 0.0-1.0; anchors: + # 0.9+ = clean evidence, well-specified test + # 0.7+ = solid but one caveat + # 0.5+ = noisy/marginal + # <0.5 = use 'inconclusive' instead of a definitive verdict + # Post-repro gate fires on (failed_to_reproduce AND confidence > 0.7). + datasets_used: [string] + reasoning: string + deviations: [string] # fetch failures, fallbacks taken, cross-dataset infeasibility + + theorize: + inputs: [autods_run, autods_literature_synthesis, data_reproduction] + # data_reproduction is absent if the all-known gate fired. + # + # ⚠️ Context-window contract: `asta generate-theories` is a remote A2A service. Its only inputs are + # the --theory-query string, the --paper-store JSON, and papers its own PaperFinder discovers. + # It CANNOT read bd, local files, or IDs not inlined in the query. Inline everything; cite papers + # by author/year, not corpus_id. + # + # Paper-store gateway limitation: `inject_papers` only honors entries with `paper_markdown`; + # identifier-only stubs are silently dropped. Best to pass an empty paper store for now. + # + # Two-shot fallback on PaperFinder failure: retry with simplified query, then close with + # theories=[], fallback_used=true. + # + # After the CLI completes, dump the task JSON and export with `asta artifacts --format md` + # rather than parsing the streamed CLI log. Both end up inside work_dir. + output: + theory_query: string + theories: + - name: string + statement: string # the theory's mechanistic claim (from artifact description) + novelty_rollup: enum [genuinely-new, derivable-but-unstated, already-established] + # best-of across novelty_claims[].label + novelty_claims: + - claim_name: string + label: enum [genuinely-new, derivable-but-unstated, already-established] + judgment_excerpt: string # first ~200 chars of the agent's novelty judgment + falsifiable_prediction: string # concatenation of novelty_claims[].claim_name + evidence_grounding: [string] # paper s2_sha or corpus_ids cited in supporting passages + fallback_used: boolean + fallback_reason: string + diff --git a/plugins/asta/skills/research-step/scripts/validate-output.sh b/plugins/asta/skills/research-step/scripts/validate-output.sh index 0f5a84e..36a1ca4 100755 --- a/plugins/asta/skills/research-step/scripts/validate-output.sh +++ b/plugins/asta/skills/research-step/scripts/validate-output.sh @@ -6,9 +6,9 @@ # Verifies that the JSON file: # 1. parses # 2. carries the canonical metadata envelope -# ({research_step: {task_type, inputs, output_schema_version, output}}) +# ({research_step: {task_type, inputs, work_dir, output}}) # 3. has every required `output.` for the given per -# assets/schemas.yaml (schema_version: 1) +# assets/schemas.yaml (schema_version: 3) # # Exit codes: # 0 — valid @@ -16,6 +16,7 @@ # 3 — unknown task_type # 4 — missing required field # 5 — task_type mismatch with envelope +# 6 — bad work_dir format # # This is structural validation only. Quality validation (sound prediction, # sane confidence, valid citations) is out of scope per execute.md. @@ -34,19 +35,24 @@ if ! jq -e . "$file" > /dev/null 2>&1; then exit 2 fi -# Required output fields, mirroring assets/schemas.yaml (schema_version: 1). +# Required output fields, mirroring assets/schemas.yaml (schema_version: 3). +# Only structured-answer fields — paths and artifact lists are gone. case "$task_type" in - scope) required="question boundaries success_criteria" ;; - definitions) required="terms" ;; - literature_review) required="summary_path key_findings gaps citations" ;; - hypothesis) required="statement rationale falsifiable_prediction expected_evidence" ;; - experiment_design) required="method procedure variables artifacts_expected" ;; - evidence_gathering) required="artifacts log_path deviations" ;; - analysis) required="verdict confidence reasoning caveats" ;; - synthesis) required="answer supporting_hypotheses refuted_hypotheses open_questions report_path" ;; + scope) required="question boundaries success_criteria" ;; + definitions) required="terms" ;; + literature_review) required="key_findings gaps citations" ;; + hypothesis) required="statement rationale falsifiable_prediction expected_evidence" ;; + experiment_design) required="method procedure variables artifacts_expected" ;; + evidence_gathering) required="deviations" ;; + analysis) required="verdict confidence reasoning caveats" ;; + synthesis) required="answer supporting_hypotheses refuted_hypotheses open_questions" ;; + autods_run) required="run_id mode dataset_summary hypotheses surprising_count" ;; + autods_literature_synthesis) required="themes surviving_findings contradicted_findings headline" ;; + data_reproduction) required="candidate_datasets findings deviations" ;; + theorize) required="theory_query theories fallback_used" ;; *) echo "validate-output: unknown task_type '$task_type'" >&2 - echo "validate-output: expected one of scope|definitions|literature_review|hypothesis|experiment_design|evidence_gathering|analysis|synthesis" >&2 + echo "validate-output: expected one of scope|definitions|literature_review|hypothesis|experiment_design|evidence_gathering|analysis|synthesis|autods_run|autods_literature_synthesis|data_reproduction|theorize" >&2 exit 3 ;; esac @@ -63,14 +69,33 @@ if [[ "$envelope_type" != "$task_type" ]]; then exit 5 fi -# Envelope shape sanity. -for key in inputs output_schema_version output; do +# Envelope shape sanity. work_dir is the new mandatory pointer to this task's files. +for key in inputs work_dir output; do if ! jq -e ".research_step | has(\"$key\")" "$file" >/dev/null; then echo "validate-output: $file missing .research_step.$key" >&2 exit 5 fi done +# work_dir format check: must be a run-root-relative path that starts with +# `.asta//` and ends with `/`. Catches drift from the convention. +work_dir=$(jq -r '.research_step.work_dir // empty' "$file") +if [[ -z "$work_dir" ]]; then + echo "validate-output: .research_step.work_dir is empty" >&2; exit 6 +fi +expected_prefix=".asta/${task_type}/" +case "$work_dir" in + "$expected_prefix"*) : ;; + *) echo "validate-output: work_dir '$work_dir' should start with '$expected_prefix'" >&2; exit 6 ;; +esac +case "$work_dir" in + */) : ;; + *) echo "validate-output: work_dir '$work_dir' must end with '/'" >&2; exit 6 ;; +esac +case "$work_dir" in + /*|*..*) echo "validate-output: work_dir '$work_dir' must be run-root-relative (no leading '/', no '..')" >&2; exit 6 ;; +esac + # Required output fields. for key in $required; do if ! jq -e ".research_step.output | has(\"$key\")" "$file" >/dev/null; then @@ -80,8 +105,7 @@ for key in $required; do done # Type spot-checks for the high-leverage cases. Not exhaustive — just the -# fields where a wrong type at this layer would silently break update-summary rendering -# or downstream tasks. +# fields where a wrong type at this layer would silently break downstream tasks. case "$task_type" in literature_review) jq -e '.research_step.output.key_findings | type == "array"' "$file" >/dev/null \ @@ -97,6 +121,24 @@ case "$task_type" in jq -e '.research_step.output.confidence | type == "number" and . >= 0 and . <= 1' "$file" >/dev/null \ || { echo "validate-output: output.confidence must be a number in [0, 1]" >&2; exit 4; } ;; + autods_run) + jq -e '.research_step.output.mode | IN("imported", "executed")' "$file" >/dev/null \ + || { echo "validate-output: output.mode must be one of imported|executed" >&2; exit 4; } + ;; + data_reproduction) + if [[ $(jq '.research_step.output.findings | length' "$file") -gt 0 ]]; then + jq -e '.research_step.output.findings[0].verdict | IN("reproduced","failed_to_reproduce","partial","inconclusive","unreachable")' "$file" >/dev/null \ + || { echo "validate-output: findings[].verdict must be one of reproduced|failed_to_reproduce|partial|inconclusive|unreachable" >&2; exit 4; } + fi + ;; + theorize) + jq -e '.research_step.output.fallback_used | type == "boolean"' "$file" >/dev/null \ + || { echo "validate-output: output.fallback_used must be boolean" >&2; exit 4; } + if [[ $(jq '.research_step.output.theories | length' "$file") -gt 0 ]]; then + jq -e '.research_step.output.theories[0].novelty_rollup | IN("genuinely-new","derivable-but-unstated","already-established")' "$file" >/dev/null \ + || { echo "validate-output: theories[].novelty_rollup must be one of genuinely-new|derivable-but-unstated|already-established" >&2; exit 4; } + fi + ;; esac echo "ok" diff --git a/plugins/asta/skills/research-step/workflows/execute.md b/plugins/asta/skills/research-step/workflows/execute.md index 5fba9ea..096d1d2 100644 --- a/plugins/asta/skills/research-step/workflows/execute.md +++ b/plugins/asta/skills/research-step/workflows/execute.md @@ -9,32 +9,31 @@ Run one ready task end-to-end. Loads its schema, gathers its declared inputs, pr ## Steps -1. **Pick a task.** If a task ID was supplied, use it. Else `bd ready --json` and pick the oldest issue (tiebreak by `bd-id` ascending). Hypothesis tasks are normally auto-resolved at creation by **plan**, so they should not appear here. If one does, it means the gap text was too thin for plan to fill the output without inventing content — flag this to the user and ask whether to refine the source `literature_review` first. +1. **Pick a task.** If a task ID was supplied, use it. Else `bd ready --json` and pick the oldest issue (tiebreak by `bd-id` ascending). 2. **Claim it.** `bd update --status=in_progress`. -3. **Load the schema.** Read the task type with `bd show --json | jq -r '.[0].metadata.research_step.task_type'`. Open `assets/schemas.yaml` and find the matching entry under `task_types`. -4. **Gather inputs.** For every issue listed in this issue's `inputs` (`bd show --json | jq '.[0].metadata.research_step.inputs'`), read its output with `bd show --json | jq '.[0].metadata.research_step.output'`. Also load `mission.md` and any files referenced from input outputs via `_path` fields (e.g., `summary_path` from a `literature_review`). **This is the only context to use** — do not pull in unrelated repo state. -5. **Do the work.** Produce a JSON object matching the schema. For schema fields ending in `_path`, write the file to disk first and put the relative path in the JSON. -6. **Validate structurally.** Run `scripts/validate-output.sh `. It checks the envelope (`research_step.task_type`, `inputs`, `output_schema_version`, `output`) and every required `output.` for the task_type, plus type spot-checks for the high-leverage cases (e.g., `analysis.verdict` enum, `analysis.confidence` range). Exit 0 ⇒ valid. Any non-zero exit ⇒ fail loudly and **leave the issue `in_progress`** for retry. Do not close. -7. **Persist the output.** Materialize the metadata JSON via `scripts/write-meta.sh` (reads JSON from stdin, prints a temp file path), then `bd update --metadata @`. Preserve the existing `task_type`, `inputs`, and `output_schema_version`. -8. **Close.** `bd close `. -9. **Hand off to plan or update-summary.** Some closed task types unlock new graph structure; others don't. Decide based on the closed task's `task_type`: - - | Closed task_type | Hand off to | - |---|---| - | `literature_review`, `hypothesis`, `analysis`, `synthesis` | **plan** (with this issue as the source). `plan` then chains to **update-summary**. Note: `hypothesis` only reaches this branch in the rare case it was left open at creation; the normal path is plan→auto-resolve. | - | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | **update-summary** directly. | - - Either path ends with `summary.md` rebuilt. - -## Notes on output files - -Schema fields ending in `_path` are relative paths. Conventions: - -- `summary_path` (from `literature_review`) → `background_knowledge.txt` by convention, but any path works. -- `log_path` (from `evidence_gathering`) → typically under `logs/`. -- `report_path` (from `synthesis`) → typically `report.md`. - -Write the file before setting the output JSON. If the executor crashes between writing the file and closing the issue, the file is harmless orphan data — re-running `execute` on the same issue will overwrite it. +3. **Load the schema.** Read `metadata.research_step.task_type` from the issue. Open `assets/schemas.yaml` and find the matching entry under `task_types`. **If the task type isn't one of the 8 standard types** (`scope`, `definitions`, `literature_review`, `hypothesis`, `experiment_design`, `evidence_gathering`, `analysis`, `synthesis`), scan `workflows/plans/*.md` for the template that defines this task type and follow its `### ` section for execution prose (CLI commands, paths, parsing details, fallback handling). +4. **Resolve the work_dir.** Read `metadata.research_step.work_dir`. Every file this task writes goes under that directory. The directory should already exist from `plan`'s create step; `mkdir -p "$work_dir"` is a safe no-op. Filenames inside follow `assets/schemas.yaml > file_conventions` for the task type — write `summary.md` for `literature_review`, `findings.md` for `autods_synthesis`, etc. +5. **Gather inputs.** Read the `metadata.research_step.output` of every issue listed in this issue's `inputs`. For each upstream task, also walk its `work_dir` directly (e.g., open `/summary.md`) when the prose context matters. **This is the only context to use** — do not pull in unrelated repo state. +5a. **Steering pickup.** Also `bd show --json` and inspect `comments[]`. For every comment **authored by someone other than `claude`/`agent`** (per `author` regex `/^(claude\|agent)/i`) that was created **since the most recent agent-authored comment on this task** (or since the task was created, whichever is later), treat it as a steering note. The agent decides how to respond — acknowledge inline (`bd comment add "" --author claude`), adjust the plan, or note explicitly why it cannot apply. Do not silently ignore. +6. **Do the work.** Write the structured output as a JSON object matching the schema. Write supporting files (markdown reports, JSON dumps, scripts, figures) inside `work_dir` per the file convention. Markdown files follow the inline-citation rule in `SKILL.md §"Markdown convention"`. +7. **Validate structurally.** Run `scripts/validate-output.sh `. It checks the envelope (`research_step.task_type`, `inputs`, `work_dir`, `output`) and every required `output.` for the task_type, plus type spot-checks for the high-leverage cases (e.g., `analysis.verdict` enum, `analysis.confidence` range). Exit 0 ⇒ valid. Any non-zero exit ⇒ fail loudly and **leave the issue `in_progress`** for retry. Do not close. +8. **Persist the output.** Materialize the metadata JSON via `scripts/write-meta.sh` (reads JSON from stdin, prints a temp file path), then `bd update --metadata @`. Preserve the existing `task_type`, `inputs`, and `work_dir`. +9. **Write a one-line `close_reason`.** A single short sentence stating what + was produced, in the plain-English voice from + `SKILL.md §"Voice for titles and descriptions"` (same ELI-college reader, + no jargon up front, no internal labels). Used in summary digests and + `bd list` output; not rendered in the Asta Flows panel. Avoid empty + placeholders (`"completed"`, `"done"`, `""`) — they aren't useful to + humans scanning the log. +10. **Close.** `bd close --reason ""`. +11. **Hand off to plan or update-summary.** Some closed task types unlock new graph structure; others don't. Decide based on the closed task's `task_type`: + + | Closed task_type | Hand off to | + |---|---| + | `literature_review`, `hypothesis`, `analysis`, `synthesis`, `*_synthesis` | **plan** (with this issue as the source). `plan` then chains to **update-summary**. | + | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | **update-summary** directly. | + + Either path ends with `summary.md` rebuilt. ## Out of scope for this workflow diff --git a/plugins/asta/skills/research-step/workflows/init.md b/plugins/asta/skills/research-step/workflows/init.md index fd11be3..627c497 100644 --- a/plugins/asta/skills/research-step/workflows/init.md +++ b/plugins/asta/skills/research-step/workflows/init.md @@ -1,6 +1,6 @@ # Workflow: init -Bootstrap the environment for a research session: install `bd` and `jq`, run `bd init`, wire beads to the project's git remote for cross-machine sync, and verify the staleness check works. This is the only workflow that may install or configure tools; `plan`, `update-summary`, and `execute` assume the environment is ready. +Bootstrap the environment for a research session: install `bd` and `jq`, run `bd init`, and verify the staleness check works. This is the only workflow that may install or configure tools; `plan`, `update-summary`, and `execute` assume the environment is ready. After environment setup, hand off to **plan** to bootstrap the mission epic and initial frontier. @@ -8,12 +8,6 @@ After environment setup, hand off to **plan** to bootstrap the mission epic and None. `init` is idempotent — installing already-installed tools is a no-op, and `bd init` is safe to skip if `.beads/` already exists. -## Backend choice - -`init` always uses beads' embedded Dolt backend (`bd init`, no `--server`). This satisfies the single-machine workflow with zero infrastructure. Cross-machine transfer is handled via `bd dolt push`/`pull` against the project's git remote (under `refs/dolt/*`) — git is the only sync layer. - -Server mode (`bd init --server`) is out of scope: it requires running a Dolt sql-server, which violates the zero-infrastructure constraint. Simultaneous writers on multiple machines are therefore unsupported; the model is one writer at a time, with git as the transfer medium. - ## Steps 1. **Ensure `bd` is installed.** @@ -21,36 +15,19 @@ Server mode (`bd init --server`) is out of scope: it requires running a Dolt sql - Otherwise, install per the beads project's documented method for the current platform. Consult https://github.com/gastownhall/beads (cross-platform options at time of writing include `go install`, Homebrew, `winget`, the project's install script, and the Nix flake). If you are uncertain which method applies, fetch the install docs at run time rather than guessing. - Verify with `bd --version`. If it still fails, abort and ask the user to install manually. -2. **Initialize beads (embedded Dolt) and wire it to git.** +2. **Ensure beads is initialized in this directory.** - If `.beads/` does not exist, run `bd init`. - - If the working directory has a git remote `origin`, configure beads' Dolt sync against it so `bd dolt push`/`pull` work without further setup. Probe with `bd dolt remote list`; if nothing is configured, add a remote pointing at `origin` per beads docs. - - If no git remote exists, skip the Dolt-remote step and tell the user that cross-machine transfer will need a remote added later. Do not block — the single-machine flow works without it. - Verify with `bd list` (should succeed and return an empty list or existing issues). -3. **Fresh-clone recovery.** If `bd init` produced an empty DB but `.beads/issues.jsonl` is present (the project had upstream beads state and was freshly cloned), **do not silently `bd import`**. - - Run `git ls-remote origin 'refs/dolt/*'`. If any Dolt refs exist, run `bd dolt pull` — this is the canonical recovery path and preserves Dolt history. - - If no Dolt refs exist on the remote, surface the situation to the user with three options: (a) `bd import .beads/issues.jsonl` (fast, but discards Dolt history and any state newer than the export), (b) configure a Dolt remote and `bd dolt push` from another machine that has the live DB, then retry, (c) abort. - - Pick one path only after explicit user confirmation. Never auto-import. - -4. **Verify the staleness check works.** +3. **Verify the staleness check works.** - Run `scripts/summary-check.sh`. It hashes the sorted IDs of currently-open issues and compares against `summary.md`'s frontmatter. Backend-agnostic — beads can use whichever storage it likes. - Requires `jq` on PATH; if missing, install it (`brew install jq`, `apt-get install jq`, etc.) and retry. - At init time `summary.md` does not yet exist, so the script will print `status: missing` and exit 1 — that's fine; **update-summary** will create the file later. `status: no-tools` (exit 3) means abort and ask the user. -5. **Hand off to plan.** Per the router's chaining rule, run the **plan** workflow next. It will detect that no epic exists yet and bootstrap one from `mission.md`. If `mission.md` is missing, **plan** will route the user back to **brainstorm**. - -## Cross-machine transfer - -To move a session to another machine: - -1. On machine A: `bd dolt push` (sends the Dolt data as a git ref to `origin`). -2. On machine B: clone the repo and run this `init` workflow. Step 3's fresh-clone recovery will see the Dolt refs on `origin` and `bd dolt pull` them automatically. - -Two machines writing at the same time is not supported in embedded mode; coordinate manually (push before handing off, pull before resuming). +4. **Hand off to plan.** Per the router's chaining rule, run the **plan** workflow next. It will detect that no epic exists yet and bootstrap one from `mission.md`. If `mission.md` is missing, **plan** will route the user back to **brainstorm**. ## Out of scope for this workflow - Creating issues or building the graph. That belongs to **plan**. - Writing `summary.md`. That belongs to **update-summary** (chained automatically after `plan`). - Re-running setup once a session is initialized. If `bd` or `jq` breaks later, fix it manually rather than re-running `init`. -- Server-mode beads (`bd init --server`) and any setup requiring a running Dolt sql-server. diff --git a/plugins/asta/skills/research-step/workflows/plan.md b/plugins/asta/skills/research-step/workflows/plan.md index c5ffb2d..b498509 100644 --- a/plugins/asta/skills/research-step/workflows/plan.md +++ b/plugins/asta/skills/research-step/workflows/plan.md @@ -5,6 +5,8 @@ Create or extend the research graph. The single home for "design the next set of - **bootstrap** — no epic exists yet. Create the mission epic and the initial frontier (scope, definitions, literature_review) from `mission.md`. - **replan** — an epic exists. Add downstream tasks based on a recently-closed task's output, or on user direction. +**Per-template override.** Before bootstrap or replan, scan `workflows/plans/*.md` to see if any template's intro section matches `mission.md`'s described intent. If a template matches, **use its bootstrap + replan rules instead of the defaults below.** The default behavior (no template matches) is the hypothesis-driven flow documented here. + Always chains to **update-summary** afterward so `summary.md` reflects the new graph. ## Preconditions @@ -22,7 +24,8 @@ Every task issue carries: "research_step": { "task_type": "", "inputs": ["bd-xxxx", "bd-yyyy"], - "output_schema_version": 1, + "work_dir": ".asta///", + "output_schema_version": 3, "output": null } } @@ -30,6 +33,23 @@ Every task issue carries: The mission epic additionally carries `epic_root: true`. +**Choosing the slug.** When creating a task, pick a short kebab-case slug +that names the *specific instance* (not the task type — the type is +already in the path). Examples: + +| task_type | sensible slugs | +|---|---| +| `scope` | `aar-ela`, `glacier-mass-balance` | +| `definitions` | `aar-ela-terms` | +| `literature_review` | `provenance`, `theme-a-aar-geometry`, `theme-b-ela-bands` | +| `autods_run` | the autodiscovery run id (e.g. `9fc65db8`) | +| `data_reproduction` | `cross-dataset`, `2026-05-otto-zeller` | +| `theorize` | `cross-cutting-mechanisms` | +| `autods_synthesis` | `findings` | + +After `bd create` returns the new id, `mkdir -p .asta///` +so executors can write into it immediately. + ## Mode selection 1. Run `scripts/epic-root.sh`. `status: none` → **bootstrap**. @@ -43,10 +63,17 @@ The mission epic additionally carries `epic_root: true`. bd create --type=epic --title="" --description="$(cat mission.md)" bd update --metadata '{"research_step":{"epic_root":true}}' ``` -3. **Create the initial frontier.** Three `task` issues with the metadata convention above: - - `scope: ` — `inputs: []` - - `definitions: ` — `inputs: []` - - `literature_review: ` — `inputs: [, ]` +3. **Create the initial frontier.** Three `task` issues with the metadata + convention above. Every `bd create` call must include both `--title` and + `--description`, written per `SKILL.md §"Voice for titles and descriptions"` + — imperative phrase ≤ 6 words for the title (e.g. "Frame the Alaska + glacier question", "Define the geometry terms we'll use"); one short + paragraph in plain ELI-college English for the description. + + Initial frontier: + - `scope` — `inputs: []` + - `definitions` — `inputs: []` + - `literature_review` — `inputs: [, ]` 4. **Add edges.** - `parent-child` from each frontier task to the epic - `blocks`: scope → definitions; scope → literature_review; definitions → literature_review @@ -54,39 +81,20 @@ The mission epic additionally carries `epic_root: true`. ## Replan mode -Read the source task's task_type and output: +Read the source task's `task_type` and `metadata.research_step.output`. Apply this table. Every newly-created task gets both `--title` and `--description` per the voice rules in `SKILL.md` — descriptions are the primary surface in the UI. -``` -bd show --json | jq '.[0].metadata.research_step.task_type' -bd show --json | jq '.[0].metadata.research_step.output' -``` -Apply this table: | Source task_type | Action | |---|---| -| `literature_review` | For each gap in `output.gaps`, create a `hypothesis` task with `inputs: [, ]`. Edges: `parent-child` to epic; `blocks` from the source. **Populate `metadata.research_step.output` at creation time** (see below) and close the issue immediately — the gap text already contains the statement, rationale, and prediction in prose, so there is no separate `execute` pass for hypotheses. | +| `literature_review` | For each gap in `output.gaps`, create a `hypothesis` task with `inputs: [, ]`. Edges: `parent-child` to epic; `blocks` from the source. | | `hypothesis` | Create the chain `experiment_design` → `evidence_gathering` → `analysis`, each `blocks` the next. `experiment_design` depends on the hypothesis (via `inputs`); `analysis` depends on both the hypothesis and the new `evidence_gathering`. All three get `parent-child` to the epic. | | `analysis` | If every `hypothesis` in the epic now has a closed `analysis`, create one `synthesis` task with `inputs` listing all analysis IDs and the scope ID. `parent-child` to epic; `blocks` from each analysis. Otherwise no-op. | -| `synthesis` | If `output.open_questions` is non-empty, **stop and ask the user** before creating new `hypothesis` tasks. If approved, create them with a `discovered-from` edge back to the synthesis (in addition to the usual edges). | +| `synthesis` (or any `*_synthesis` closing task) | If `output.open_questions` is non-empty, **stop and ask the user** before creating new `hypothesis` tasks; if approved, create them with `discovered-from` edges back. Otherwise no replan — the synthesis is the closing artifact. | | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | No replan. Report no-op and stop. | If invoked without a source task and the user has not specified what to plan, do not invent work — ask, or stop. -### Auto-resolving hypothesis tasks - -When creating a `hypothesis` from a literature_review gap: - -1. Derive the four output fields directly from the gap text and surrounding `literature_review` output (`bd show --json | jq '.[0].metadata.research_step.output'`): - - `statement` — `H_n: ` - - `rationale` — why this gap implies the claim - - `falsifiable_prediction` — what observation would refute it - - `expected_evidence` — list of concrete evidence types that would support it -2. Validate with `scripts/validate-output.sh hypothesis ` before persisting. -3. Persist with `scripts/write-meta.sh` + `bd update --metadata @`, then `bd close `. - -If a gap is too thin to fill these fields without inventing content, **do not auto-resolve** — leave the hypothesis open and surface it to the user. Genuine ambiguity is the one case where a separate `execute` pass is warranted. - ## After either mode Hand off to **update-summary** so `summary.md` reflects the new state. diff --git a/skills/research-step/workflows/execute.md b/skills/research-step/workflows/execute.md index 5fba9ea..ec3510b 100644 --- a/skills/research-step/workflows/execute.md +++ b/skills/research-step/workflows/execute.md @@ -9,10 +9,10 @@ Run one ready task end-to-end. Loads its schema, gathers its declared inputs, pr ## Steps -1. **Pick a task.** If a task ID was supplied, use it. Else `bd ready --json` and pick the oldest issue (tiebreak by `bd-id` ascending). Hypothesis tasks are normally auto-resolved at creation by **plan**, so they should not appear here. If one does, it means the gap text was too thin for plan to fill the output without inventing content — flag this to the user and ask whether to refine the source `literature_review` first. +1. **Pick a task.** If a task ID was supplied, use it. Else `bd ready --json` and pick the oldest issue (tiebreak by `bd-id` ascending). 2. **Claim it.** `bd update --status=in_progress`. -3. **Load the schema.** Read the task type with `bd show --json | jq -r '.[0].metadata.research_step.task_type'`. Open `assets/schemas.yaml` and find the matching entry under `task_types`. -4. **Gather inputs.** For every issue listed in this issue's `inputs` (`bd show --json | jq '.[0].metadata.research_step.inputs'`), read its output with `bd show --json | jq '.[0].metadata.research_step.output'`. Also load `mission.md` and any files referenced from input outputs via `_path` fields (e.g., `summary_path` from a `literature_review`). **This is the only context to use** — do not pull in unrelated repo state. +3. **Load the schema.** Read `metadata.research_step.task_type` from the issue. Open `assets/schemas.yaml` and find the matching entry under `task_types`. +4. **Gather inputs.** Read the `metadata.research_step.output` of every issue listed in this issue's `inputs`. Also load `mission.md` and any files referenced from input outputs via `_path` fields (e.g., `summary_path` from a `literature_review`). **This is the only context to use** — do not pull in unrelated repo state. 5. **Do the work.** Produce a JSON object matching the schema. For schema fields ending in `_path`, write the file to disk first and put the relative path in the JSON. 6. **Validate structurally.** Run `scripts/validate-output.sh `. It checks the envelope (`research_step.task_type`, `inputs`, `output_schema_version`, `output`) and every required `output.` for the task_type, plus type spot-checks for the high-leverage cases (e.g., `analysis.verdict` enum, `analysis.confidence` range). Exit 0 ⇒ valid. Any non-zero exit ⇒ fail loudly and **leave the issue `in_progress`** for retry. Do not close. 7. **Persist the output.** Materialize the metadata JSON via `scripts/write-meta.sh` (reads JSON from stdin, prints a temp file path), then `bd update --metadata @`. Preserve the existing `task_type`, `inputs`, and `output_schema_version`. @@ -21,7 +21,7 @@ Run one ready task end-to-end. Loads its schema, gathers its declared inputs, pr | Closed task_type | Hand off to | |---|---| - | `literature_review`, `hypothesis`, `analysis`, `synthesis` | **plan** (with this issue as the source). `plan` then chains to **update-summary**. Note: `hypothesis` only reaches this branch in the rare case it was left open at creation; the normal path is plan→auto-resolve. | + | `literature_review`, `hypothesis`, `analysis`, `synthesis` | **plan** (with this issue as the source). `plan` then chains to **update-summary**. | | `scope`, `definitions`, `experiment_design`, `evidence_gathering` | **update-summary** directly. | Either path ends with `summary.md` rebuilt. diff --git a/skills/research-step/workflows/init.md b/skills/research-step/workflows/init.md index fd11be3..627c497 100644 --- a/skills/research-step/workflows/init.md +++ b/skills/research-step/workflows/init.md @@ -1,6 +1,6 @@ # Workflow: init -Bootstrap the environment for a research session: install `bd` and `jq`, run `bd init`, wire beads to the project's git remote for cross-machine sync, and verify the staleness check works. This is the only workflow that may install or configure tools; `plan`, `update-summary`, and `execute` assume the environment is ready. +Bootstrap the environment for a research session: install `bd` and `jq`, run `bd init`, and verify the staleness check works. This is the only workflow that may install or configure tools; `plan`, `update-summary`, and `execute` assume the environment is ready. After environment setup, hand off to **plan** to bootstrap the mission epic and initial frontier. @@ -8,12 +8,6 @@ After environment setup, hand off to **plan** to bootstrap the mission epic and None. `init` is idempotent — installing already-installed tools is a no-op, and `bd init` is safe to skip if `.beads/` already exists. -## Backend choice - -`init` always uses beads' embedded Dolt backend (`bd init`, no `--server`). This satisfies the single-machine workflow with zero infrastructure. Cross-machine transfer is handled via `bd dolt push`/`pull` against the project's git remote (under `refs/dolt/*`) — git is the only sync layer. - -Server mode (`bd init --server`) is out of scope: it requires running a Dolt sql-server, which violates the zero-infrastructure constraint. Simultaneous writers on multiple machines are therefore unsupported; the model is one writer at a time, with git as the transfer medium. - ## Steps 1. **Ensure `bd` is installed.** @@ -21,36 +15,19 @@ Server mode (`bd init --server`) is out of scope: it requires running a Dolt sql - Otherwise, install per the beads project's documented method for the current platform. Consult https://github.com/gastownhall/beads (cross-platform options at time of writing include `go install`, Homebrew, `winget`, the project's install script, and the Nix flake). If you are uncertain which method applies, fetch the install docs at run time rather than guessing. - Verify with `bd --version`. If it still fails, abort and ask the user to install manually. -2. **Initialize beads (embedded Dolt) and wire it to git.** +2. **Ensure beads is initialized in this directory.** - If `.beads/` does not exist, run `bd init`. - - If the working directory has a git remote `origin`, configure beads' Dolt sync against it so `bd dolt push`/`pull` work without further setup. Probe with `bd dolt remote list`; if nothing is configured, add a remote pointing at `origin` per beads docs. - - If no git remote exists, skip the Dolt-remote step and tell the user that cross-machine transfer will need a remote added later. Do not block — the single-machine flow works without it. - Verify with `bd list` (should succeed and return an empty list or existing issues). -3. **Fresh-clone recovery.** If `bd init` produced an empty DB but `.beads/issues.jsonl` is present (the project had upstream beads state and was freshly cloned), **do not silently `bd import`**. - - Run `git ls-remote origin 'refs/dolt/*'`. If any Dolt refs exist, run `bd dolt pull` — this is the canonical recovery path and preserves Dolt history. - - If no Dolt refs exist on the remote, surface the situation to the user with three options: (a) `bd import .beads/issues.jsonl` (fast, but discards Dolt history and any state newer than the export), (b) configure a Dolt remote and `bd dolt push` from another machine that has the live DB, then retry, (c) abort. - - Pick one path only after explicit user confirmation. Never auto-import. - -4. **Verify the staleness check works.** +3. **Verify the staleness check works.** - Run `scripts/summary-check.sh`. It hashes the sorted IDs of currently-open issues and compares against `summary.md`'s frontmatter. Backend-agnostic — beads can use whichever storage it likes. - Requires `jq` on PATH; if missing, install it (`brew install jq`, `apt-get install jq`, etc.) and retry. - At init time `summary.md` does not yet exist, so the script will print `status: missing` and exit 1 — that's fine; **update-summary** will create the file later. `status: no-tools` (exit 3) means abort and ask the user. -5. **Hand off to plan.** Per the router's chaining rule, run the **plan** workflow next. It will detect that no epic exists yet and bootstrap one from `mission.md`. If `mission.md` is missing, **plan** will route the user back to **brainstorm**. - -## Cross-machine transfer - -To move a session to another machine: - -1. On machine A: `bd dolt push` (sends the Dolt data as a git ref to `origin`). -2. On machine B: clone the repo and run this `init` workflow. Step 3's fresh-clone recovery will see the Dolt refs on `origin` and `bd dolt pull` them automatically. - -Two machines writing at the same time is not supported in embedded mode; coordinate manually (push before handing off, pull before resuming). +4. **Hand off to plan.** Per the router's chaining rule, run the **plan** workflow next. It will detect that no epic exists yet and bootstrap one from `mission.md`. If `mission.md` is missing, **plan** will route the user back to **brainstorm**. ## Out of scope for this workflow - Creating issues or building the graph. That belongs to **plan**. - Writing `summary.md`. That belongs to **update-summary** (chained automatically after `plan`). - Re-running setup once a session is initialized. If `bd` or `jq` breaks later, fix it manually rather than re-running `init`. -- Server-mode beads (`bd init --server`) and any setup requiring a running Dolt sql-server. diff --git a/skills/research-step/workflows/plan.md b/skills/research-step/workflows/plan.md index c5ffb2d..4b8ccb8 100644 --- a/skills/research-step/workflows/plan.md +++ b/skills/research-step/workflows/plan.md @@ -54,18 +54,11 @@ The mission epic additionally carries `epic_root: true`. ## Replan mode -Read the source task's task_type and output: - -``` -bd show --json | jq '.[0].metadata.research_step.task_type' -bd show --json | jq '.[0].metadata.research_step.output' -``` - -Apply this table: +Read the source task's `task_type` and `metadata.research_step.output`. Apply this table: | Source task_type | Action | |---|---| -| `literature_review` | For each gap in `output.gaps`, create a `hypothesis` task with `inputs: [, ]`. Edges: `parent-child` to epic; `blocks` from the source. **Populate `metadata.research_step.output` at creation time** (see below) and close the issue immediately — the gap text already contains the statement, rationale, and prediction in prose, so there is no separate `execute` pass for hypotheses. | +| `literature_review` | For each gap in `output.gaps`, create a `hypothesis` task with `inputs: [, ]`. Edges: `parent-child` to epic; `blocks` from the source. | | `hypothesis` | Create the chain `experiment_design` → `evidence_gathering` → `analysis`, each `blocks` the next. `experiment_design` depends on the hypothesis (via `inputs`); `analysis` depends on both the hypothesis and the new `evidence_gathering`. All three get `parent-child` to the epic. | | `analysis` | If every `hypothesis` in the epic now has a closed `analysis`, create one `synthesis` task with `inputs` listing all analysis IDs and the scope ID. `parent-child` to epic; `blocks` from each analysis. Otherwise no-op. | | `synthesis` | If `output.open_questions` is non-empty, **stop and ask the user** before creating new `hypothesis` tasks. If approved, create them with a `discovered-from` edge back to the synthesis (in addition to the usual edges). | @@ -73,20 +66,6 @@ Apply this table: If invoked without a source task and the user has not specified what to plan, do not invent work — ask, or stop. -### Auto-resolving hypothesis tasks - -When creating a `hypothesis` from a literature_review gap: - -1. Derive the four output fields directly from the gap text and surrounding `literature_review` output (`bd show --json | jq '.[0].metadata.research_step.output'`): - - `statement` — `H_n: ` - - `rationale` — why this gap implies the claim - - `falsifiable_prediction` — what observation would refute it - - `expected_evidence` — list of concrete evidence types that would support it -2. Validate with `scripts/validate-output.sh hypothesis ` before persisting. -3. Persist with `scripts/write-meta.sh` + `bd update --metadata @`, then `bd close `. - -If a gap is too thin to fill these fields without inventing content, **do not auto-resolve** — leave the hypothesis open and surface it to the user. Genuine ambiguity is the one case where a separate `execute` pass is warranted. - ## After either mode Hand off to **update-summary** so `summary.md` reflects the new state.