refactor[next]: rename Backend to Toolchain and restructure artifact loading

docs/development/ADRs/next/0011-On_The_Fly_Compilation.md

@@ -14,6 +14,8 @@ This supersedes [0009 - Compiled Backend Integration](0009-Compiled-Backend-Inte
 > [!NOTE]
 > The stage and step names introduced below have since been renamed and redistributed over two modules: `ProgramCall` is now `otf.stages.CompilableProgramDef`; `ProgramSource`, `CompilableSource` and `CompiledProgram` are now `otf.artifacts.ProgramSource`, `otf.artifacts.ExtensionSource` and `otf.artifacts.CompilationArtifact` (whose `load()` returns an `otf.artifacts.ExecutableProgram`); and `otf.step_types` was merged into `otf.stages`, where `BindingStep` no longer exists. The architecture described here — composable, statically typed workflows — is unchanged.
 
+Partially superseded by [0029 - Toolchain Naming and Pipeline Simplification](0029-Toolchain-Naming-and-Pipeline-Simplification.md) (workflow combinators and step-type naming).
+
 ## Context
 
 The on-the-fly compilation (OTFC) in gt4py encompasses everything necessary to go from an IR representation of a GT4Py program to an executable Python function. Depending on the chosen route, this may include:

docs/development/ADRs/next/0017-Toolchain-Configuration.md

@@ -11,6 +11,8 @@ tags: [backend, otf, workflows, toolchain]
 
 In order to provide a streamlined user experience, we attempt to standardize how users of GT4Py stencils can configure how those stencils are optimized without editing GT4Py code. This describes the design of the first minimal implementation.
 
+Refined by [0029 - Toolchain Naming and Pipeline Simplification](0029-Toolchain-Naming-and-Pipeline-Simplification.md) (the term *toolchain* becomes the name of the root object).
+
 ## Context
 
 In this document the word toolchain is used to mean all the code components that work together to go from DSL code to an optimized, runnable python callable.

docs/development/ADRs/next/0027-External_Workspace_Memory.md

@@ -9,6 +9,8 @@ tags: []
 - **Created**: 2026-07-27
 - **Updated**: 2026-08-03
 
+Partially superseded by [0029 - Toolchain Naming and Pipeline Simplification](0029-Toolchain-Naming-and-Pipeline-Simplification.md) (the workspace is carried by an explicit `Toolchain.loading` step instead of a `DaCeBackend` subclass overriding `Backend.load_artifact`; the decision to inject the workspace at load time is unchanged).
+
 In the context of `gt4py.next` DaCe backends that run the same compiled SDFG
 many times (e.g. inside a time loop), facing per-call GPU transient allocation
 overhead and the desire to bound workspace memory to one SDFG's transient

docs/development/ADRs/next/0029-Toolchain-Naming-and-Pipeline-Simplification.md

@@ -0,0 +1,176 @@
+---
+tags: [backend, otf, toolchain, workflows, naming, observability]
+---
+
+# Toolchain Naming and Pipeline Simplification
+
+- **Status**: valid
+- **Authors**: Enrique González Paredes (@egparedes)
+- **Created**: 2026-07-30
+- **Updated**: 2026-07-30
+
+In the context of the on-the-fly compilation toolchain, facing a root object
+whose names (`Backend`, `transforms`, `executor`) no longer match the
+established *toolchain* vocabulary and a workflow-combinator framework whose
+reading cost exceeds its value, we decided to rename the root object to
+`Toolchain` (with `frontend` / `backend` halves), rename the envelope and
+stage types accordingly, replace the combinator tower with explicit, fully
+typed pipelines, and add a sanctioned stage-observability seam. We considered
+keeping the combinators and trimming only the dead ones, and accept that the
+renames break the old names outright, rotate the persistent translation-cache
+keys (the second such rotation in this stack), and impose a migration burden
+on downstream code that subclassed the combinators.
+
+This partially supersedes [0011 - On The Fly Compilation](0011-On_The_Fly_Compilation.md)
+(the workflow-combinator framework and the `otf.step_types` naming — its stage
+vocabulary and build-system decisions remain valid) and refines
+[0017 - Toolchain Configuration](0017-Toolchain-Configuration.md) (whose term
+*toolchain* becomes the name of the root object; its configuration decisions
+remain valid).
+
+This builds on [0028 - Plain Builders Instead of factory-boy Factories](0028-Plain-Builders-Instead-of-Factories.md),
+which replaced the factory classes that constructed these objects, so the
+renames below land in plain, type-checked builder code.
+
+## Context
+
+ADR 0017 already defines **toolchain** as "all the code components that work
+together to go from DSL code to an optimized, runnable python callable" —
+JIT/OTF pipelines, transformation passes, lowerings, parsers. The code grew
+into exactly that shape while keeping older names: the root object is called
+`Backend`, its frontend half `transforms`, its backend half `executor`, and
+the pipeline pieces are spread over five vaguely-named modules (`workflow`,
+`toolchain`, `definitions`, `stages`, `recipes`).
+
+Separately, the workflow-combinator framework introduced by ADR 0011 had grown
+to thirteen abstractions (`Workflow`, two mixins, `NamedStepSequence`,
+`MultiWorkflow`, `StepSequence`, `CachedStep`, `SkippableStep`, `make_step`,
+the `ConcreteArtifact` envelope and three adapters) to express function
+composition (some of them already deleted as dead code in the preparatory
+cleanup PRs). Measured against actual use, only `CachedStep` is a deep module;
+the others are shallow wrappers around `Callable[[S], T]`, and the
+`NamedStepSequence.__call__` reflection loop is `Any`-typed, defeating the
+static typing ADR 0011 prized.
+
+Finally, the seam between the toolchain and its consumers has no sanctioned
+interface for observing or running intermediate stages: the one consumer that
+needs an intermediate stage (the dace `__sdfg__` path) duck-types into the
+default pipeline and mutates its stages.
+
+## Decision
+
+### Naming
+
+| Old                          | New                               |
+| ---------------------------- | --------------------------------- |
+| `Backend`                    | `Toolchain`                       |
+| `Backend.transforms`         | `Toolchain.frontend`              |
+| `Backend.executor`           | `Toolchain.backend`               |
+| `Backend.load_artifact()`    | `Toolchain.loading`               |
+| `OTFCompileWorkflow`         | `CompilePipeline`                 |
+| `ConcreteArtifact` / `.data` | `ProgramWithArgs` / `.definition` |
+| `CompilableProgramDef`       | `CompilableProgram`               |
+
+These are hard renames: no deprecation aliases are kept, so the old names stop
+resolving in the same release. The public `gtx.*` names (`gtx.gtfn_cpu`,
+`gtx.wait_for_compilation`, ...) are unchanged, with one exception —
+`gtx.typing.Backend` becomes `gtx.typing.Toolchain`.
+
+We considered shipping one-release read aliases for the renamed attributes and
+re-export shims for the moved modules. We rejected that: an alias that silently
+keeps working is exactly what lets a downstream stay on the old vocabulary past
+the release where it was supposed to migrate, and the aliases would have been
+partial anyway (constructing with the old keyword names could not be aliased
+without hand-writing `__init__`, so `Backend(executor=...)` would have raised
+`TypeError` regardless). A single loud break at a known release is easier to
+act on than a half-working compatibility layer. `CompilableProgram` stays a type
+alias (of the parameterized envelope) for now.
+
+The last row is not a pure rename. `Backend.load_artifact()` was an overridable
+method, and a toolchain that needed backend-specific runtime data in the loaded
+program subclassed `Toolchain` to override it (see
+[0027 - External Workspace Memory for DaCe Transients](0027-External_Workspace_Memory.md)).
+`Toolchain.loading` makes that seam a field like the other three — a plain step
+defaulting to `artifacts.load_artifact` — so customization stays
+composition-time, consistent with the rest of this ADR. It also keeps
+subclassing off the load path: per
+[0024 - Compilation Runners](0024-Compilation-Runners.md), a toolchain that
+customizes `compile` is opaque to the process-pool runner, and a
+customization mechanism built on subclassing invites exactly that. The step
+lives on `Toolchain` rather than inside `CompilePipeline` because loading also
+happens off the pipeline — `CompiledProgramsPool` loads artifacts straight from
+a compilation future — and because monolithic backends have no `CompilePipeline`
+to hang it on.
+
+### Pipeline, not combinators
+
+Named pipelines become frozen dataclasses with an explicit, fully typed
+`__call__`; steps are plain callables (`Step[S, T] = Callable[[S], T]`);
+customization stays composition-time via `dataclasses.replace`. The
+combinator framework (both mixins, `NamedStepSequence`, `MultiWorkflow`,
+`StepSequence`, `make_step`, `.chain`, the three adapters) is deleted;
+`CachedStep` is kept unchanged.
+
+What ADR 0011's decisions become:
+
+| ADR 0011 requirement                        | Kept by                                                                 |
+| ------------------------------------------- | ----------------------------------------------------------------------- |
+| Named steps, order visible                  | dataclass fields + explicit `__call__` (order literally visible)        |
+| Statically typed composition                | explicit `__call__` — checked end-to-end, unlike the reflection loop    |
+| Customization at composition, not via flags | `dataclasses.replace` on frozen pipelines                               |
+| Steps compose across backends               | `Step[S, T]` is a `Callable` — every existing step already satisfies it |
+| Linear workflows                            | unchanged (`Transforms` keeps its input-dependent step *selection*)     |
+
+### Stage observability
+
+A `stage_hook(name, artifact)` event hook (on the existing
+`instrumentation.hook_machinery`) is emitted by `CompilePipeline` and
+`Transforms` after each step; artifacts are treated opaquely.
+`GT4PY_DUMP_STAGES=<dir>` registers a subscriber that writes each stage
+artifact per program. `Toolchain.translate(definition, compile_time_args)` is
+the sanctioned partial run (frontend + translation only); it narrows to the
+standard `CompilePipeline` shape and raises a clear error on monolithic
+backends. Per-call step options are deliberately not offered — a caller
+needing a variant step builds a variant pipeline with `dataclasses.replace`.
+
+### Phasing
+
+The decisions in this ADR land as a stack of PRs. The naming decisions above
+land first, except the `OTFCompileWorkflow` → `CompilePipeline` rename, which
+lands with the pipeline PR; the "Pipeline, not combinators" and "Stage
+observability" decisions are implemented in follow-up PRs of the same stack.
+
+## Consequences
+
+- The dace `__sdfg__` reach-in (duck-typing through `executor.translation`
+  and mutating frozen stages) will be replaced by a dace-owned translate-only
+  step built with `dataclasses.replace`.
+- Downstream code subclassing the deleted combinators or overriding
+  `Transforms.step_order` must migrate to explicit `__call__` composition.
+- Downstream code must migrate to the new names in the same release: the old
+  ones are removed, not aliased. That includes the modules dissolved along the
+  way — `otf.definitions`, `otf.code_specs`, `otf.recipes` and `otf.toolchain`
+  cease to exist, and `gtx.typing.Backend` becomes `gtx.typing.Toolchain`.
+- ADR 0011's `otf.step_types` naming section no longer describes the code.
+- Cache keys rotate: fingerprints embed the qualified class and field names,
+  so the `ProgramWithArgs` / `definition` rename changes the keys of the
+  persistent translation caches (gtfn and dace). This is the *second* of two
+  rotations in this stack — moving the same class from `otf.toolchain` to
+  `otf.workflow` already rotated them once, since the qualified name alone
+  is enough. What makes each a cold rebuild rather than a stale hit is that
+  the key itself changes, so entries written under the old name are never
+  looked up; `BUILD_CACHE_VERSION_ID` (the gt4py version) is a separate salt
+  on top, which means the rotation costs nothing extra to anyone who crosses
+  a release — or, for a source install, any commit — boundary anyway. The two
+  collapse into a single cold rebuild if the whole stack lands in one
+  release, which is the concrete argument for not spreading it across
+  releases.
+
+## Alternatives considered
+
+- **Keep the combinators but trim dead ones**: rejected — the tower is a
+  shallow-module archetype and every recorded ADR 0011 requirement survives
+  its removal; trimming keeps the reading cost and the `Any`-typed
+  composition.
+- **A per-call options API on `translate()`**: rejected — reconfiguration
+  stays composition-time, ADR 0011's own rule.

docs/development/ADRs/next/README.md

@@ -53,6 +53,7 @@ Writing a new ADR is simple:
 - [0017 - Toolchain Configuration](0017-Toolchain-Configuration.md)
 - [0027 - External Workspace Memory for DaCe Transients](0027-External_Workspace_Memory.md)
 - [0028 - Plain Builders Instead of factory-boy Factories](0028-Plain-Builders-Instead-of-Factories.md)
+- [0029 - Toolchain Naming and Pipeline Simplification](0029-Toolchain-Naming-and-Pipeline-Simplification.md)
 
 ### Python Integration
 

docs/user/next/advanced/HackTheToolchain.md

@@ -22,8 +22,8 @@ cached_lowering_toolchain = gtx.backend.DEFAULT_TRANSFORMS.replace(
 ## Skip Steps / Change Order
 
 ```python
-DUMMY_FOP = workflow.ConcreteArtifact(
-    data=ff_stages.DSLFieldOperatorDef(definition=None), args=None
+DUMMY_FOP = workflow.ProgramWithArgs(
+    definition=ff_stages.DSLFieldOperatorDef(definition=None), args=None
 )
 ```
 

docs/user/next/advanced/WorkflowPatterns.md

@@ -399,7 +399,7 @@ gtx.backend.DEFAULT_PROG_TRANSFORMS??
 ```
 
 ```python
-gtx.program_processors.runners.gtfn.run_gtfn_gpu.executor.otf_workflow??
+gtx.program_processors.runners.gtfn.run_gtfn_gpu.backend??
 ```
 
 ```python

src/gt4py/next/AGENTS.md

@@ -31,6 +31,9 @@ field operators / programs (ffront)
   Distinct from `iterator/embedded.py`, which runs one level lower.
 - `otf/` — on-the-fly compilation toolchain (workflow steps, caching,
   argument descriptors).
+- `backend.py` — the toolchain root object `Toolchain` (formerly `Backend`):
+  `frontend` (definition transforms) + `backend` (compile pipeline) +
+  allocator.
 - `program_processors/runners/` — the backends: `gtfn` (GridTools C++),
   `dace` (DaCe SDFG), `roundtrip` / `double_roundtrip` (pure Python).
 - `type_system/` — `next` type specifications and the type inference the

src/gt4py/next/backend.py

@@ -9,7 +9,7 @@
 from __future__ import annotations
 
 import dataclasses
-from typing import Generic
+from typing import Callable, Generic
 
 from gt4py._core import definitions as core_defs
 from gt4py.next import config, custom_layout_allocators as next_allocators
@@ -45,7 +45,7 @@ def adapted_jit_to_aot_args_factory() -> workflow.Workflow[
 class Transforms(
     workflow.MultiWorkflow[
         stages.ConcreteProgramDef[stages.IRDefinitionT, stages.ArgsDefinitionT],
-        stages.CompilableProgramDef,
+        stages.CompilableProgram,
     ]
 ):
     """
@@ -92,14 +92,14 @@ class Transforms(
     ] = dataclasses.field(default_factory=past_process_args.transform_program_args_factory)
 
     past_to_itir: workflow.Workflow[
-        ffront_stages.ConcretePASTProgramDef, stages.CompilableProgramDef
+        ffront_stages.ConcretePASTProgramDef, stages.CompilableProgram
     ] = dataclasses.field(default_factory=past_to_itir.past_to_gtir_factory)
 
     def step_order(self, inp: stages.ConcreteProgramDef) -> list[str]:
         steps: list[str] = []
         if isinstance(inp.args, arguments.JITArgs):
             steps.append("aotify_args")
-        match inp.data:
+        match inp.definition:
             case ffront_stages.DSLFieldOperatorDef():
                 steps.extend(
                     [
@@ -140,32 +140,36 @@ def step_order(self, inp: stages.ConcreteProgramDef) -> list[str]:
 DEFAULT_TRANSFORMS: Transforms = Transforms()
 
 
-# TODO(tehrengruber): Rename class and `executor` & `transforms` attribute. Maybe:
-#  `Backend` -> `Toolchain`
-#  `transforms` -> `frontend_transforms`
-#  `executor` -> `backend_transforms`
 @dataclasses.dataclass(frozen=True)
-class Backend(Generic[core_defs.DeviceTypeT]):
+class Toolchain(Generic[core_defs.DeviceTypeT]):
+    """
+    Complete pipeline from a program definition to an executable program.
+
+    The `frontend` workflow transforms any supported program definition into a
+    `CompilableProgram`, which the `backend` workflow then compiles into a
+    loadable compilation artifact. The `loading` step turns that artifact into a
+    directly-callable program; toolchains needing to inject backend-specific
+    runtime data supply their own step here. The `allocator` describes the
+    device the compiled program expects its buffers on.
+    """
+
     name: str
-    executor: workflow.Workflow[stages.CompilableProgramDef, artifacts.CompilationArtifact]
+    backend: workflow.Workflow[stages.CompilableProgram, artifacts.CompilationArtifact]
     allocator: next_allocators.FieldBufferAllocatorProtocol[core_defs.DeviceTypeT]
-    transforms: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgramDef]
+    frontend: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgram]
+    # A plain `Callable`, not `workflow.Workflow`: the protocol names its
+    # parameter `inp`, which would force every loading step to use that name.
+    loading: Callable[[artifacts.CompilationArtifact], artifacts.ExecutableProgram] = (
+        dataclasses.field(default=artifacts.load_artifact)
+    )
 
     def compile(
         self, program: stages.IRDefinitionT, compile_time_args: arguments.CompileTimeArgs
     ) -> artifacts.ExecutableProgram:
-        artifact = self.executor(
-            self.transforms(stages.ConcreteProgramDef(data=program, args=compile_time_args))
+        artifact = self.backend(
+            self.frontend(stages.ConcreteProgramDef(definition=program, args=compile_time_args))
         )
-        return self.load_artifact(artifact)
-
-    def load_artifact(self, artifact: artifacts.CompilationArtifact) -> artifacts.ExecutableProgram:
-        """Load an artifact into an executable program.
-
-        Backends may override this method to inject backend-specific runtime data
-        into the loaded program.
-        """
-        return artifact.load()
+        return self.loading(artifact)
 
     @property
     def __gt_allocator__(

src/gt4py/next/config.py

@@ -48,10 +48,10 @@ class BuildJobsMode(enum.Enum):
     #: Run compilation in a ``ThreadPoolExecutor``.
     THREAD = "thread"
     #: Run compilation in a ``ProcessPoolExecutor`` with the ``spawn`` start
-    #: method. Requires the backend's ``executor`` to be stdlib-picklable
+    #: method. Requires the toolchain's ``backend`` pipeline to be stdlib-picklable
     #: (standard runners and factory-constructed variants are) and to return a
     #: picklable ``CompilationArtifact``; backends that don't qualify (or that
-    #: customize ``Backend.compile``) are compiled in the calling thread
+    #: customize ``Toolchain.compile``) are compiled in the calling thread
     #: instead, with a warning.
     PROCESS = "process"