✨ workflowmodel: ingest → lifecycle models, /api/workflows/ingest, survey compile stats (RFC #17620 P3a) (#17652)

ingest.Lifecycle composes hive-default's compiled lifecycle with a
repository's pipelines: PR-verify pipelines on the CI / Review states with
citations, the post-merge chain as inferred columns after Merged,
provenance ingested-unreviewed. New pipeline.status fact (engine reports
unknown until post-merge runs are observed). ingest.ReadTree selects CI
files from a checkout. POST /api/workflows/ingest fetches a repo's default
branch read-only (or takes pasted files), compiles, optionally saves at the
next integer version. workflow-survey compiles every corpus repo's
lifecycle model: 317/317 compile, 0 refusals.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

Author: clubanderson

changelog.d/added-workflowmodel-ingest-lifecycle.md

@@ -0,0 +1 @@
+- Workflow models, RFC phase 3a ([#17620](https://github.com/hivecommons/hive/issues/17620)): an ingested repository now yields a model that compiles. `ingest.Lifecycle` composes the hive-default lifecycle (states, columns, builtin derivation, roles, gates, events) with the repository's pipelines — PR-verify pipelines bound to the CI / Review states with file citations, the post-merge chain (main CI, release, per-environment deploys, done) as new columns after Merged — every ingested element marked inferred and the model's provenance `ingested-unreviewed`. New fact `pipeline.status` names the post-merge run the chain waits on (the engine reports `unknown` until it observes post-merge runs, so nothing moves on it yet). New owner-gated `POST /api/workflows/ingest` takes `{repo}` (default branch fetched read-only through the hive's GitHub client, CI files selected by `ingest.ReadTree`) or pasted `{files}`, returns the model, compile result and diagnostics, and with `save: true` stores it at the next integer version. `cmd/workflow-survey` compiles every repository's lifecycle model and reports pass/refusal counts by code; over the 317-repository CNCF corpus 317/317 compile with 1,117 inferred-element warnings and no refusals.

dashboard/openapi.json

@@ -39356,6 +39356,79 @@
           }
         }
       }
+    },
+    "/api/workflows/ingest": {
+      "post": {
+        "tags": [
+          "Workflows"
+        ],
+        "summary": "Ingest a repository's CI definitions into a lifecycle workflow model",
+        "description": "Reads the repository's default-branch GitHub Actions workflows and Tekton resources (or pasted files) and returns the hive-default lifecycle with the repository's pipelines bound: PR-verify pipelines on the CI / Review states, the post-merge chain as new columns. Provenance ingested-unreviewed. With save=true the model is stored as the next integer version under name. Owner role required.",
+        "requestBody": {
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "type": "object",
+                "properties": {
+                  "repo": {
+                    "type": "string",
+                    "description": "owner/name to fetch through the forge client; exactly one of repo or files"
+                  },
+                  "files": {
+                    "type": "object",
+                    "additionalProperties": {
+                      "type": "string"
+                    },
+                    "description": "repo-relative path to file contents, for pasted or uploaded definitions"
+                  },
+                  "save": {
+                    "type": "boolean"
+                  },
+                  "name": {
+                    "type": "string",
+                    "description": "store name (default: slug of the repository)"
+                  }
+                }
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "Ingested (not saved)",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/WorkflowIngestResult"
+                }
+              }
+            }
+          },
+          "201": {
+            "description": "Ingested and saved",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/WorkflowIngestResult"
+                }
+              }
+            }
+          },
+          "400": {
+            "description": "Bad request"
+          },
+          "403": {
+            "description": "Owner role required"
+          },
+          "502": {
+            "description": "Repository fetch failed"
+          },
+          "503": {
+            "description": "No forge client or no workflow store"
+          }
+        }
+      }
     }
   },
   "components": {
@@ -39759,6 +39832,38 @@
         "type": "object",
         "description": "Compiled, derivation-ready form of a model (hash, states, columns, ordered rules). Never trusted from the wire: the spoke recompiles on save and on bind.",
         "additionalProperties": true
+      },
+      "WorkflowIngestResult": {
+        "type": "object",
+        "properties": {
+          "model": {
+            "$ref": "#/components/schemas/WorkflowModel"
+          },
+          "compiled": {
+            "type": "boolean"
+          },
+          "diagnostics": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/WorkflowDiagnostic"
+            }
+          },
+          "profile": {
+            "type": "object"
+          },
+          "ref": {
+            "type": "string"
+          },
+          "files": {
+            "type": "integer"
+          },
+          "pipelines": {
+            "type": "integer"
+          },
+          "entry": {
+            "$ref": "#/components/schemas/WorkflowEntry"
+          }
+        }
       }
     }
   }

src/cmd/hive/main.go

@@ -1087,6 +1087,7 @@ func (b *boot) wireBootClosures() {
 		return &dashboard.Dependencies{
 			Config:            b.cfg,
 			WorkflowStore:     wfstore.Open(wfstore.Root(b.cfg.Data.WorkflowsDir)),
+			WorkflowRepoFiles: workflowRepoFiles(func() *github.Client { return b.ghClient }),
 			AgentMgr:          b.agentMgr,
 			CruxRuntimeHealth: cruxRuntimeHealthFunc(b.cruxSupervision),
 			// Provider gateways (#5565 slice 3): concrete openrouter/watsonx/

src/cmd/hive/workflow_ingest.go

@@ -0,0 +1,37 @@
+package main
+
+import (
+	"context"
+	"errors"
+
+	"github.com/hivecommons/hive/pkg/github"
+	"github.com/hivecommons/hive/pkg/workflowmodel/ingest"
+)
+
+// errNoForgeClient is returned by the workflow ingest fetcher before a
+// GitHub client exists.
+var errNoForgeClient = errors.New("no GitHub client")
+
+// workflowRepoFiles is the dashboard's WorkflowRepoFiles dependency
+// (POST /api/workflows/ingest, RFC #17620 P3): it downloads the repo's
+// default branch read-only, keeps only the CI definition files and
+// removes the checkout. client is read per call so a credential rebuild is
+// followed.
+func workflowRepoFiles(client func() *github.Client) func(ctx context.Context, repo string) (map[string][]byte, string, error) {
+	return func(ctx context.Context, repo string) (map[string][]byte, string, error) {
+		c := client()
+		if c == nil {
+			return nil, "", errNoForgeClient
+		}
+		co, err := c.FetchDefaultBranchTree(ctx, repo)
+		if err != nil {
+			return nil, "", err
+		}
+		defer co.Cleanup()
+		files, err := ingest.ReadTree(co.Dir)
+		if err != nil {
+			return nil, "", err
+		}
+		return files, co.Ref, nil
+	}
+}

src/cmd/hive/workflow_ingest_test.go

@@ -0,0 +1,16 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/hivecommons/hive/pkg/github"
+)
+
+func TestWorkflowRepoFilesWithoutClient(t *testing.T) {
+	fetch := workflowRepoFiles(func() *github.Client { return nil })
+	if _, _, err := fetch(context.Background(), "acme/widgets"); !errors.Is(err, errNoForgeClient) {
+		t.Fatalf("err = %v", err)
+	}
+}

src/cmd/workflow-survey/main.go

@@ -68,6 +68,12 @@ type repoResult struct {
 	ValidateError  string                         `json:"validateError,omitempty"`
 	Hash           string                         `json:"hash"`
 	Severities     map[workflowmodel.Severity]int `json:"severities,omitempty"`
+	// Compiled says whether the lifecycle model (ingest.Lifecycle) compiles;
+	// CompileCodes counts its diagnostics by code, errors and warnings.
+	Compiled      bool           `json:"compiled"`
+	CompileError  string         `json:"compileError,omitempty"`
+	CompileCodes  map[string]int `json:"compileCodes,omitempty"`
+	LifecycleCols int            `json:"lifecycleColumns"`
 }
 
 type summary struct {
@@ -78,6 +84,8 @@ type summary struct {
 	Parsed           int                            `json:"parsed"`
 	Pipelines        int                            `json:"pipelines"`
 	Valid            int                            `json:"valid"`
+	Compiled         int                            `json:"compiled"`
+	CompileCodes     map[string]int                 `json:"compileCodes"`
 	Roles            map[workflowmodel.Role]int     `json:"roles"`
 	Constructs       map[string]int                 `json:"constructs"`
 	FindingCodes     map[string]int                 `json:"findingCodes"`
@@ -160,7 +168,7 @@ func run(corpus, out string, filter map[string]bool, stdout, stderr io.Writer) e
 		return err
 	}
 	maturity := loadProjects(corpus)
-	sum := &summary{Roles: map[workflowmodel.Role]int{}, Constructs: map[string]int{}, FindingCodes: map[string]int{}, Severities: map[workflowmodel.Severity]int{}, ColumnSets: map[string]int{}, ByProject: map[string]*projectStats{}}
+	sum := &summary{Roles: map[workflowmodel.Role]int{}, Constructs: map[string]int{}, FindingCodes: map[string]int{}, Severities: map[workflowmodel.Severity]int{}, CompileCodes: map[string]int{}, ColumnSets: map[string]int{}, ByProject: map[string]*projectStats{}}
 	for _, d := range dirs {
 		if !d.IsDir() {
 			continue
@@ -252,9 +260,25 @@ func surveyRepo(dir, repo, outDir string) (repoResult, error) {
 	}
 	res.ColumnDiff = render.Diff(m)
 
+	lm := ingest.Lifecycle(workflowmodel.Scope{Forge: defaultForge, Repository: repo}, files)
+	res.LifecycleCols = len(lm.Spec.Columns)
+	res.CompileCodes = map[string]int{}
+	_, diags, cerr := workflowmodel.Compile(lm)
+	for _, d := range diags {
+		res.CompileCodes[d.Code]++
+	}
+	if cerr != nil {
+		res.CompileError = cerr.Error()
+	} else {
+		res.Compiled = true
+	}
+
 	if err := os.MkdirAll(outDir, 0o755); err != nil {
 		return res, err
 	}
+	if err := writeJSON(filepath.Join(outDir, "lifecycle.json"), lm); err != nil {
+		return res, err
+	}
 	if err := writeJSON(filepath.Join(outDir, "model.json"), m); err != nil {
 		return res, err
 	}
@@ -324,6 +348,12 @@ func accumulate(s *summary, r repoResult) {
 	if r.Valid {
 		s.Valid++
 	}
+	if r.Compiled {
+		s.Compiled++
+	}
+	for k, v := range r.CompileCodes {
+		s.CompileCodes[k] += v
+	}
 	if len(r.ColumnDiff) > 0 {
 		s.ReposChangedCols++
 	}
@@ -375,7 +405,12 @@ func markdown(s *summary) string {
 	fmt.Fprintf(&b, "# Workflow model survey\n\n")
 	fmt.Fprintf(&b, "| | |\n|---|---|\n")
 	fmt.Fprintf(&b, "| Repositories | %d |\n| with GitHub Actions pipelines | %d |\n| with Tekton pipelines | %d |\n", s.Repos, s.ReposWithGHA, s.ReposWithTekton)
-	fmt.Fprintf(&b, "| Files read | %d |\n| Files parsed | %d (%.1f%%) |\n| Pipelines | %d |\n| Models valid | %d / %d |\n| Repos whose columns change | %d |\n\n", s.Files, s.Parsed, pct(s.Parsed, s.Files), s.Pipelines, s.Valid, s.Repos, s.ReposChangedCols)
+	fmt.Fprintf(&b, "| Files read | %d |\n| Files parsed | %d (%.1f%%) |\n| Pipelines | %d |\n| Models valid | %d / %d |\n| Lifecycle models compiled | %d / %d |\n| Repos whose columns change | %d |\n\n", s.Files, s.Parsed, pct(s.Parsed, s.Files), s.Pipelines, s.Valid, s.Repos, s.Compiled, s.Repos, s.ReposChangedCols)
+	b.WriteString("## Lifecycle compile diagnostics\n\n| Code | Count |\n|---|---|\n")
+	for _, k := range sortedKeys(s.CompileCodes) {
+		fmt.Fprintf(&b, "| %s | %d |\n", k, s.CompileCodes[k])
+	}
+	b.WriteString("\n")
 	b.WriteString("## Pipeline roles\n\n| Role | Pipelines |\n|---|---|\n")
 	for _, k := range sortedKeys(s.Roles) {
 		fmt.Fprintf(&b, "| %s | %d |\n", k, s.Roles[workflowmodel.Role(k)])

src/docs/design/workflow-models.md

@@ -125,3 +125,33 @@ kind is not covered. Models are whole (no inheritance) through phase 4.
   columns, numbers, error}}}` (nil when nothing is bound). The JS labels the model column,
   renders a per-row column strip by stage number, and shows an amber "Not covered" chevron for
   abstentions. Settings → Repos has a per-repo Workflow select that PUTs the binding.
+
+## Ingest → lifecycle (RFC phase 3a)
+
+`ingest.Repository` (phase 0) is unchanged and still produces the process half. `ingest.Lifecycle`
+composes on top of it:
+
+1. Start from `hivedefault.Generate()` — the compiled hive-default lifecycle, `derivation: builtin`.
+2. Copy the ingested pipelines, sources and coverage; `meta.source` is the system(s) read,
+   `meta.provenance` is `ingested-unreviewed`, `meta.version` is `1` (the store assigns later
+   versions).
+3. PR-verify pipelines are bound to every state in the CI / Review column, each with a citation to
+   the file the pipeline came from.
+4. Every post-merge state the process half produced (main CI, release, `deploy-<env>`, done)
+   becomes a state in its own column after Merged, `confidence: inferred`, with the environment
+   guards carried over. The chain's transitions fire on `pipeline-succeeded` =
+   `pipeline.status is green`. The engine does not observe post-merge runs yet and reports
+   `unknown` for that fact, so a bound ingested model derives exactly like hive-default today;
+   the extra columns render, nothing lands in them until post-merge observation exists.
+
+The compiler warns once per inferred element; a human confirming them in the editor (phase 4)
+is what moves provenance to `local`/`curated`.
+
+`POST /api/workflows/ingest` is the spoke entry point (owner role): `{repo}` fetches the default
+branch through `FetchDefaultBranchTree`, `ingest.ReadTree` keeps GitHub Actions workflows, YAML
+under Tekton directories and pipeline-named YAML that looks like Tekton (the survey collector's
+rules), and the checkout is removed. `{files}` accepts pasted definitions. `save: true` stores the
+result at the next integer version under `name` (default: the repository slug).
+
+`cmd/workflow-survey` now compiles every corpus repository's lifecycle model (`lifecycle.json`
+alongside `model.json`) and reports pass/refusal counts by diagnostic code.

src/pkg/dashboard/api_workflows.go

@@ -20,6 +20,7 @@ func (s *Server) registerWorkflowRoutes() {
 	s.mux.HandleFunc("GET /api/workflows", s.handleWorkflowList)
 	s.mux.HandleFunc("POST /api/workflows", s.handleWorkflowSave)
 	s.mux.HandleFunc("POST /api/workflows/compile", s.handleWorkflowCompile)
+	s.mux.HandleFunc("POST /api/workflows/ingest", s.handleWorkflowIngest)
 	s.mux.HandleFunc("GET /api/workflows/bindings", s.handleWorkflowBindingsGet)
 	s.mux.HandleFunc("PUT /api/workflows/bindings", s.handleWorkflowBindingPut)
 	s.mux.HandleFunc("GET /api/workflows/{name}/{version}", s.handleWorkflowGet)