Evidence-backed retrospective over delivered plans and their review findings

[fastner]

Deferred — to be reviewed by Sebastian before any planning or implementation.
Open decision: Should an evidence-backed retrospective live in Effective Flow, in the central skills repo (sebastian-software/skills.sebastian-software.com, candidate skill effective-delivery), or split across both? It is not decided yet. The split sketched below is the working hypothesis to confirm or reject before any plan is written.

Problem

Effective Flow produces everything a retrospective needs but never evaluates it across runs:

• Delivered plans are archived under <plan.dir>/archive/ with their acceptance criteria, affected files, and a ## Review findings summary (src/tools/build.md Phase 7, src/shared/plan-archival.md).
• Residual findings get stable R-XXXXXXX IDs in .effective-flow/review/review-report-*.md (src/shared/review-report-format.md:39). In remote mode they become forge issues with effective-flow-review-finding labels. Implemented findings get a ✅ backlink (src/shared/review-report-backlinks.md).
• Correction rounds are recorded only in the wisdom file, which build deletes at the end of a run (src/tools/build.md:424), so that learning is lost.

Existing checks work on single artifacts: plan-review can review one implemented plan retrospectively (src/tools/plan-review.md:135-137), and issue-post-merge-observation checks tracker state per merged PR. Nothing looks across a batch: whether acceptance criteria were met, where churn and rework concentrated, which finding classes recur, and whether earlier action items landed.

Proposal

The mechanism, independent of where it ends up:

1. Evidence inventory. For a selected set of archived plans, derive each delivery commit or PR from the commit that moved the plan into archive/. Collect commits, per-file churn (merges counted separately, never double-counted), linked R-ID findings with their state, and the previous retrospective. A missing input is recorded as narrowed scope, so "checked and clean" stays distinguishable from "never checked".
2. Analysis. Every claim cites a file and line, a commit, or an R-ID; worker reports are re-checked against the primary source before use.
3. Verdict and action items. Each plan gets accepted, accepted-with-open-items, or not accepted against its acceptance criteria. A human decision always overrides the machine verdict. Action items are only proposed, as copyable fix/refactor/build/docs prompts. The retrospective itself never edits code, plans, or ADRs.
4. Follow-through. Each action item from the previous retrospective is recorded as landed (with evidence) or no evidence found.

Possible split:

• Effective Flow would own: a retro tool (or a mode of an existing tool) that resolves the archived-plan set, including hidden mode where archiving leaves no Git linkage; a retro-evidence.mjs / retro-evidence-core.mjs pair emitting the Git and churn inventory as JSON; reading R-IDs from local reports and the configured tracker target; the artifact's location and one-language status line; and the next-steps rows (for example after merge-gate | merged).
• effective-delivery would own: the methodology: evidence standards, verdict judgment, turning recurring findings into process lessons, and action-item reconciliation. Its audit route already reconciles recorded work as done, stale, or blocked, but it has no retrospective procedure yet.

Acceptance criteria

• A decision is recorded on ownership (Effective Flow only, skills repo only, or split), naming the owning skill.
• If Effective Flow owns any part: the retrospective artifact format, its location (tracked or runtime state), and its status line are specified in both German and English.
• Every claim in a produced retrospective carries a checkable reference, and every missing input is listed as a narrowed scope.
• The run never modifies code, plans, ADRs, or tracker issues. Action items are proposals only.
• The evidence script is deterministic and has node:test coverage for merge handling, empty ranges, and hidden mode.

Scope

Only if Effective Flow owns part of this: src/tools/retro.md (new), src/scripts/retro-evidence.mjs and retro-evidence-core.mjs, build.mjs (TOOL_GROUPS, CONTEXT_BUDGET_LINES, RUNTIME_SCRIPT_FILES), src/shared/next-steps.md, docs/developer-guide/skill-ownership.md and .json, test/, user guide.

Constraints and notes

• Run the ownership check. Effective Flow keeps no second copy of the retrospective methodology, only a minimal fallback.
• A new exposed tool changes the router, and a new next-steps row changes a fragment that merge-gate loads. Either makes the merge-gate eval evidence stale, so it must be re-recorded before the next release.
• Evidence is thin today: plans record no baseline SHA or PR number, local review reports are untracked, and wisdom files are deleted. Size L; sensible only after Record a diff baseline before the first edit and hand the diff to validators and reviewers #469 and Admit review findings by an evidence-checked verdict instead of the reviewer's self-rated confidence #470.

Related

• Record a diff baseline before the first edit and hand the diff to validators and reviewers #469: a recorded diff baseline gives each plan an exact commit range.
• Admit review findings by an evidence-checked verdict instead of the reviewer's self-rated confidence #470: stored per-finding verdicts are the input for spotting recurring finding classes.
• Give acceptance criteria, concept capabilities and work packages stable IDs and show plan coverage in open-plans #471: stable acceptance-criterion IDs allow a verdict per criterion.
• Add a read-only next tool that reports where work stands and decide the future of open-plans #486: a next tool could suggest a retrospective once enough plans are archived.
• [R-0000054] Resolve retrospective plan reviews from the archive #186 (closed): made archived plans reachable for single-plan retrospective review in plan-review.