SDK-style deploy/debug: selected-device + checksum pre-check + pluggable engine-binding seam

docs/plans/sdk-style-projects/00-overview.md

@@ -0,0 +1,171 @@
+# nanoFramework SDK-Style Project System Migration — Specification Set
+
+**Status:** Draft v2
+**Scope:** Migration of .NET nanoFramework from the legacy `.nfproj` flavored
+project system to a first-class MSBuild SDK — **managed project system only**.
+**Audience:** nanoFramework core contributors with CLR-internals and MSBuild
+expertise.
+
+---
+
+## 0.1 Why this exists
+
+The legacy project system relies on a project flavor, MSBuild targets shipped via
+the VS / VS Code extensions, `packages.config`, hand-written `.nuspec`, AnyCPU-only
+builds, and an x64-task / `nodeReuse` workaround. That couples builds to the IDE
+extensions and diverges from mainstream .NET tooling, making the CLI and CI story
+harder than it should be. This set specifies a move to an MSBuild **SDK**, so that
+`dotnet build` / `dotnet pack` / `dotnet test` work on a clean machine with only
+the .NET SDK and a NuGet restore.
+
+## 0.2 Scope boundary (read this first)
+
+This is a **managed project-system migration**. The following are **out of scope**
+and belong to a separate, later effort — they are deliberately absent from these
+specs and must not be reintroduced:
+
+- OTA update system.
+- Modular / relocatable native firmware packaging, native compile/link, and any
+  native binaries shipped inside NuGet packages (`runtimes/{rid}/native/`,
+  pre-linked modules, module/ABI manifests, CoreRuntime firmware packs, toolchain
+  packs).
+- Any firmware- or device-side changes.
+
+NuGet packages in scope ship **managed** assets only (`.pe` + reference `.dll` +
+`.pdbx` + `.xml`).
+
+## 0.3 The blocker — RESOLVED by the POC ✅
+
+The maintainer attributed the SDK-style block to the **VS debugger**
+([#1635](https://github.com/orgs/nanoframework/discussions/1635)). A code-level
+read of `nf-Visual-Studio-extension` (`develop`) **decomposed** that into a more
+tractable picture — and the executed POC then **confirmed** the decomposition by
+achieving deploy + F5 + source breakpoints on a real ESP32_S3_OCTAL:
+
+- The VS project system is **already CPS**, not a legacy MPF flavor
+  (`NanoCSharpProject{Unconfigured,Configured}.cs`;
+  `<ProjectCapability Include="CPS" />` in `NFProjectSystem.targets`).
+- Deploy (`DeployProvider : IDeployProvider`) and debug-launch
+  (`NanoDebuggerLaunchProvider : DebugLaunchProviderBase`) are **CPS providers**
+  keyed off the `NanoCSharpProject` capability, and the engine is **launched by
+  GUID** (`LaunchDebugEngineGuid = CorDebug.EngineGuid`). None of this inspects the
+  project-file format.
+
+So the concrete gate was **(1) build-targets composition** — the nano targets import
+the legacy MSBuild chain and collide with `Microsoft.NET.Sdk` (#1635) — and
+**(2) project-type registration / capability injection** onto SDK-style projects. The
+**AD7 debug engine (`CorDebug`) is orthogonal** — confirmed: launched by GUID, it
+attaches to the SDK-style CPS project unchanged once the project carries the
+`NanoCSharpProject` capability. Migrating the engine **AD7 → Concord** is separate
+modernization (future-proofing against AD7 deprecation), **not** the unlock.
+
+**Plan of record (executed):** an A+C proof-of-concept — author a minimal
+`nanoFramework.Sdk` + inject the capability, keep the AD7 engine behind an
+engine-binding abstraction so Concord can be swapped later
+([poc-sdk-style-debugging-plan.md](poc-sdk-style-debugging-plan.md)). **The POC is
+done and the gate is cleared on hardware.** Results in
+[poc-findings/RESULTS.md](poc-findings/RESULTS.md); the full decision record
+(every blocker + fix, §1–§6) in
+[poc-findings/DEBUGGING-LOG.md](poc-findings/DEBUGGING-LOG.md). Four issues
+surfaced and were fixed: F5-console (LaunchProfiles removed + `DebuggerFlavor`),
+deploy version mismatch (checksum-only pre-check), breakpoints (Debug must emit a
+**Windows/full** PDB, not portable), and dev-only legacy `.nfproj` load (surface the
+`InstallRoot="MSBuild"` assets).
+
+What this means in practice now:
+
+## 0.3.1 Blocked vs. not blocked → all paths proven
+
+- **Never blocked:** build, pack, and test via the CLI (cross-platform). MDP and the
+  test adapter look only at build outputs and standard MSBuild items, so they're
+  project-type agnostic.
+- **Was blocked, now proven:** VS deploy + debugging on SDK-style projects. The POC
+  demonstrated it on real hardware, so the flavor *can* be retired.
+
+The remaining work is **productization** — packaging/publishing the `nanoFramework.Sdk`,
+folding the POC fixes back into the shipped extension, and fleet migration (doc 09) —
+**not** a feasibility question.
+
+## 0.4 Corrected premises
+
+Two framing assumptions from the original prompt that the codebase / ecosystem
+have invalidated:
+
+1. **MDP is already an MSBuild task**, not an external post-build tool. It ships as
+   `nanoFramework.Tools.MetadataProcessor.MsBuildTask` wired into
+   `NFProjectSystem.MDP.targets` (`GenerateBinaryOutputTask` et al.), with a
+   `.CLI` variant for runtime-codegen. The work is **re-hosting** it inside an SDK
+   with proper incrementality and ordering (doc 04).
+2. **`netnano1.0` is a real, recognized TFM.** It appears in the
+   [Microsoft TFM table](https://learn.microsoft.com/en-us/dotnet/standard/frameworks#supported-target-frameworks)
+   (".NET nanoFramework → `netnano1.0`"), recognized by the .NET SDK and NuGet
+   client. The real gap is narrower: nanoFramework's **packages aren't published
+   against `netnano1.0`** yet (consumers fall back to `net` to restore — see
+   [#1635](https://github.com/orgs/nanoframework/discussions/1635)), and projects
+   still use `packages.config`. Closing that is unblocked work (doc 02 §2.2).
+
+Two further realities the specs build on:
+
+- The current project-system files (`NFProjectSystem.Default.props`,
+  `NFProjectSystem.props`, `NFProjectSystem.CSharp.targets`,
+  `NFProjectSystem.MDP.targets`) are distributed via the **VS extension**
+  (`$(MSBuildExtensionsPath)\nanoFramework\v1.0\`, shipped as `InstallRoot="MSBuild"`
+  VSIX assets — note a non-elevated *experimental-instance* deploy can't surface these,
+  which is why a legacy `.nfproj` fails to load there until restored; see DEBUGGING-LOG
+  §6) and the **VS Code extension** (`dist/utils/nanoFramework/v1.0/`), located via
+  `$(NanoFrameworkProjectSystemPath)`. One of them
+  (`NFProjectSystem.CSharp.targets`) re-imports
+  `Microsoft.CSharp.CurrentVersion.targets`, which collides in SDK-style/imported
+  contexts (#1635, #1067) — the SDK must own the import chain.
+- The project flavor GUID `{11A8DD76-328B-46DF-9F39-F559912D0360}` (plus the C#
+  GUID `{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}`) is what makes VS load the custom
+  project system, and is tied to the debugger gate (doc 06).
+
+## 0.5 Document map
+
+| Doc | Title |
+|-----|-------|
+| 00 | Overview (this doc) |
+| [01](01-current-state.md) | Current State Analysis |
+| [02](02-sdk-design.md) | SDK Design: `nanoFramework.Sdk`, the TFM, the target graph |
+| [03](03-project-file-migration.md) | Project File Migration |
+| [04](04-mdp-native-integration.md) | Metadata Processor (MDP) Integration |
+| [05](05-cli-experience.md) | CLI Experience (`dotnet build/deploy/new/watch`) |
+| [06](06-ide-integration.md) | Visual Studio & VS Code Integration (debugger gate cleared ✅) |
+| [07](07-library-migration.md) | Library Repository Migration (~100+ repos) |
+| [08](08-nuget-pipeline.md) | NuGet Pipeline (managed `pack`) |
+| [09](09-implementation-strategy.md) | Implementation Strategy & Phasing |
+| [10](10-tooling-specs.md) | Tooling Specifications, Package Layouts, Templates |
+| [POC](poc-sdk-style-debugging-plan.md) | A+C debugging proof-of-concept (executed: [results](poc-findings/RESULTS.md)) |
+| [VSCode](vscode-extension-impact.md) | VS Code extension migration impact |
+
+## 0.6 Naming conventions used across the set
+
+- **SDK package:** `nanoFramework.Sdk` (the MSBuild project SDK). Referenced as
+  `<Project Sdk="nanoFramework.Sdk/<version>">`. The SDK package version is
+  independent of the TFM.
+- **TFM:** `netnano1.0`, long form `.NETnanoFramework,Version=v1.0`.
+- **Deploy tool:** `nanoff` remains the on-device executor; the SDK target
+  `Deploy` orchestrates it (doc 05).
+
+## 0.7 What "done" looks like (for the unblocked, managed part)
+
+A minimal nanoFramework app is a single `.csproj`:
+
+```xml
+<Project Sdk="nanoFramework.Sdk/1.0.0">
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFramework>netnano1.0</TargetFramework>
+  </PropertyGroup>
+</Project>
+```
+
+`dotnet build` produces a `.pe` (+ `.pdbx`); `dotnet pack` emits a package with the
+managed assets under `lib/netnano1.0/`; `dotnet test` runs the unit tests. No VS
+extension is required to build, pack, or test. VS debugging of SDK-style projects is now
+**proven** (the POC: F5 + source breakpoints on real hardware — doc 06, doc 09 §9.5);
+productizing it in the shipped extension is the remaining step. The MSBuild SDK itself now
+exists as the official repo
+[`nanoframework/nanoFramework.Sdk`](https://github.com/nanoframework/nanoFramework.Sdk)
+(WIP on `move-to-sdk`).

docs/plans/sdk-style-projects/01-current-state.md

@@ -0,0 +1,113 @@
+# 01 — Current State Analysis
+
+This document inventories what the legacy `.nfproj` system actually does, so the SDK design (doc 02) has a concrete migration target. Where a fact is load-bearing for the migration, it is called out.
+
+---
+
+## 1.1 The three layers of the current build
+
+The current managed build is split across three components with poorly-defined boundaries. The migration's first job is to draw those boundaries cleanly.
+
+### Layer A — Visual Studio extension (`nf-Visual-Studio-Extension`)
+
+Provides:
+
+- **The project flavor.** Registered under project type GUID `{11A8DD76-328B-46DF-9F39-F559912D0360}` (composed with the C# project GUID `{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}` via `<ProjectTypeGuids>`). This is what makes VS load `.nfproj` with the nanoFramework property pages, references UI, and build behavior.
+- **The MSBuild props/targets payload.** Drops `NFProjectSystem.*.props/targets` into `$(MSBuildExtensionsPath)\nanoFramework\v1.0\`, which `.nfproj` files locate via `$(NanoFrameworkProjectSystemPath)`.
+- **Device Explorer** — serial/USB device discovery, ping, device capabilities, deploy UI.
+- **Deploy button** — builds, then pushes PE assemblies to the device.
+- **Debugger integration** — the wire-protocol debug engine (breakpoints, stepping, the managed debugging session against nanoCLR).
+
+The problem: **build logic lives here that should live in the SDK.** The deploy button, in particular, encodes deploy orchestration that ought to be an MSBuild target callable from CLI. Today CLI users can't `deploy` without `nanoff` invoked separately.
+
+### Layer B — MSBuild project system (`NFProjectSystem.*`)
+
+Distributed as four files (names/contents reconstructed from public sources; exact internals to be confirmed against the repo during Phase 0):
+
+| File | Role |
+|------|------|
+| `NFProjectSystem.Default.props` | Default property values; imported at top of `.nfproj`. Sets `NanoFrameworkProjectSystemPath`, output type, framework version defaults. |
+| `NFProjectSystem.props` | Core properties; references the C# common props. |
+| `NFProjectSystem.CSharp.targets` | Hooks Roslyn (`csc`) compilation into the nanoFramework graph; AnyCPU enforcement. |
+| `NFProjectSystem.MDP.targets` | The MDP stage. Defines/invokes the metadata-processor task(s) — e.g. `GenerateBinaryOutputTask` (~line 718) — after the C# compile. Produces `.pe`, `.pdbx`, native stubs, checksum. |
+
+These import the standard `Microsoft.CSharp.targets` underneath, then layer the nanoFramework PE stage on top. This is *exactly* the structure an MSBuild SDK formalizes (`Sdk.props` at the top, `Sdk.targets` at the bottom, replacing the explicit imports).
+
+### Layer C — Metadata Processor (MDP)
+
+Two shipping forms:
+
+- **`nanoFramework.Tools.MetadataProcessor.MsBuildTask`** — the managed MSBuild task DLL (built x64 to match VS2022's host). This is the integrated path; `NFProjectSystem.MDP.targets` calls it. **MDP is already a task**, contradicting the "external post-build tool" framing.
+- **`nanoFramework.Tools.MetadataProcessor.CLI`** — the standalone CLI (`-loadhints`, `-parse`, `-compile` in order) for runtime-codegen scenarios (generate C# at runtime, compile with Roslyn, convert IL→PE on the fly). Distributed as a *content* package; on non-SDK projects you must set Copy-to-Output manually.
+
+What MDP does (the core of the managed pipeline):
+
+1. Parses the IL assembly Roslyn emitted.
+2. Emits the **`.pe`** (nanoFramework Portable Executable) — the actual on-device assembly format.
+3. Emits **`.pdbx`** (nanoFramework debug DB) for the VS debugger.
+4. For `[MethodImpl(InternalCall)]` / native interop declarations, generates **native stub C++** into `bin/<config>/stubs/<assembly-name>/`.
+5. Computes the **`NativeMethodsChecksum`** (the PE↔native ABI hash), writes it into `corlib_native.cpp` / the stub headers, and embeds it in the PE. This checksum is what the runtime checks at load time to refuse a PE whose native counterpart doesn't match.
+
+> **Migration-relevant:** the SDK should surface this checksum as a build output
+> (an MSBuild property), not bury it in generated C++, so an optional build-time
+> ABI gate can consume it (doc 04 §4.5). This is a managed-build concern; the
+> stubs themselves feed the firmware build and are never shipped in a package.
+
+## 1.2 The PE build pipeline, today
+
+```
+                 .nfproj (MSBuild, AnyCPU, flavored)
+                        │
+          ┌─────────────┴──────────────┐
+          │  CoreCompile (Roslyn csc)  │   →  obj/<cfg>/<Asm>.dll  (standard IL)
+          └─────────────┬──────────────┘      + .pdb
+                        │
+          ┌─────────────┴──────────────┐
+          │  MDP MSBuild task           │
+          │  (NFProjectSystem.MDP.      │   →  bin/<cfg>/<Asm>.pe      (nano PE)
+          │   targets, GenerateBinary   │   →  bin/<cfg>/<Asm>.pdbx
+          │   OutputTask et al.)        │   →  bin/<cfg>/stubs/<Asm>/*.cpp,*.h
+          └─────────────┬──────────────┘   →  NativeMethodsChecksum (embedded)
+                        │
+                 NuGet pack (.nuspec) → lib/netnanoframework1.0/<Asm>.{pe,pdbx,dll,xml}
+```
+
+Notes that matter:
+
+- The `.dll` Roslyn produces is a throwaway intermediate; the **`.pe` is the artifact**. (NuGet packages ship the `.pe` *and* the reference `.dll` so other projects can compile against the API.)
+- AnyCPU is mandatory; the MDP task DLL being x64 + `nodeReuse` interactions cause the well-known "build the test project's nfproj in a pre-build event with `-nr=False`" workaround.
+- There is **no RID** anywhere. The build is target-agnostic; the *same* PE runs on any device whose CLR exports the matching native methods (validated by checksum). This stays true for the managed migration — there is no per-RID native artifact in scope.
+
+## 1.3 NuGet packaging, today
+
+- Packages are produced from a hand-written **`.nuspec`** alongside the `.nfproj` (see the IoT.Device repo layout: `Binding1.nuspec`, `version.json`). `dotnet pack` / SDK pack is *not* used; packaging is `nuget pack` against the nuspec.
+- Versioning via **Nerdbank.GitVersioning** (`version.json`).
+- Package contents: managed only — `lib/netnanoframework1.0/<Asm>.pe`, `.pdbx`, the reference `.dll`, and `.xml` docs. **No native payload.** Native code (where it exists) lives in `nf-interpreter/targets/` and is compiled into the monolithic firmware, *not* shipped in the library's package.
+
+The managed migration keeps this property: packages ship managed assets only.
+What changes is the folder (`lib/netnano1.0/`) and the mechanics (`dotnet pack`
+from MSBuild properties instead of `nuget pack` from a `.nuspec`) — see doc 08.
+
+## 1.4 The firmware build (out of scope, but adjacent)
+
+`nf-interpreter` builds nanoCLR/nanoBooter with **CMake**, entirely separate from MSBuild. `nanoff` downloads pre-built firmware images and flashes them. The SDK migration does **not** absorb the CMake firmware build, and does not introduce any native build path of its own — native compilation and firmware packaging are out of scope (separate effort). For native interop, MDP continues to emit stubs that the CMake firmware build consumes; the managed SDK's job stops at producing the PE.
+
+## 1.5 Inventory: properties / items / tasks to migrate
+
+The SDK must preserve or replace these. Exact set to be enumerated from the repo in Phase 0; known/expected members:
+
+**Properties**
+- `NanoFrameworkProjectSystemPath` — *eliminated* (SDK resolves itself).
+- `TargetFrameworkVersion` = `v1.0` — *replaced* by `TargetFramework=netnano1.0`.
+- `ProjectTypeGuids` — *eliminated* (no flavor).
+- AnyCPU `Platform` — *retained as default*, but RID becomes the meaningful axis.
+- `NativeMethodsChecksum` (output) — *surfaced as a public output property/item*.
+
+**Tasks**
+- `GenerateBinaryOutputTask` and siblings in MDP.targets — *re-hosted* into `nanoFramework.Sdk` targets (doc 04).
+
+**Items**
+- Implicit `Compile` globs do not exist today (every `.cs` is listed). SDK introduces default globbing — a major project-file simplification (doc 03).
+
+**Targets**
+- The Roslyn→MDP chain — *reimplemented* as SDK targets ordered against `AfterCompile`/`CoreCompile` (doc 04 §4.4).