app-modeling: replace the modeling prose with a check on the compiled output

[AzureMike]

The app-modeling skill reads a repository and writes a Radius app.bicep. All of its guidance used to be prose, spread across eight reference documents and another 255 lines in SKILL.md. This change cuts the general guidance to one document, keeps the custom-resource-type guide separate, and adds a script that inspects the file the skill just generated.

The old design

Every rule lived in a document the model was expected to have read and kept in mind. bicep-structure-rules.md was 278 lines of property names, required fields, and which things are written as objects rather than arrays. Seven more covered naming, secrets, connections, the runtime contract, architecture patterns, a catalog of components, and a worked example.

When a generated model came out wrong, the obvious fix was another rule, and that works for a while. Each rule gets written after watching one repository fail, so it covers that repository. But past a certain point the model can no longer hold all of the rules in mind while reading unfamiliar source code, and a rule that only ever mattered for one repository is still sitting there being read on every other one.

Why more words weren't the answer

Take a real rule: an app.bicep must resolve the Radius extension. If it doesn't, Bicep can leave the resource types unresolved instead of compiling the model the skill meant to write.

This is a plain fact about the finished file. A script can settle it directly instead of asking the model to carry the rule in mind through every source file it reads.

The same is true of rules such as declaring exactly one application, using valid resource shapes, reading only Recipe outputs that exist, binding only published secret keys, connecting resources the workload consumes, and pinning source builds to an immutable revision.

Most of the old prose was like this: claims about the finished file that a script can check.

What the model is better at

A script can only check the file that was written. It cannot read the source repository and work out which of several ways to run an application is the one worth modeling. It also cannot decide whether the source needs TLS, session state, bootstrap users, writable storage, or a setting whose default breaks the main operation.

The model still does this part. It follows one real operation from input, through the workloads and backing services, to an observable result. After the checker passes, it compares that same source trace with the final model so a checker repair does not quietly remove something the application needs.

The new design

The skill now runs in nine steps.

1. Check the repository. Confirm that the repository and required Dockerfile are present before writing anything.

2. Resolve the target. Read the exact Radius schemas and Azure Recipe Pack the model will use.

3. Read the source. Start with Dockerfiles, manifests, startup scripts, and the configuration the application actually reads.

4. Pick one way to run it. Follow one complete operation and list the workloads and backing services that operation needs.

5. Prove the profile is complete. Check the listener, protocol, TLS, authentication, startup, migrations, persistence, image user, and provider requirements. If a required fact is missing, stop instead of guessing.

6. Resolve every Radius field. Write .radius/bicepconfig.json, then check each type, property, output, and secret key against the selected schema and Recipe.

7. Check the image. Build from a clean checkout, or pull the exact digest for an allowed release image, then inspect the image's user, command, working directory, platform, and volumes.

8. Write the model. Generate .radius/app.bicep with the workloads, resources, connections, secrets, mounts, and lifecycle settings the selected profile needs.

9. Compile and check. Fix every proven checker finding, then compare the final model with the source trace again.

When no predefined Radius type fits a required Azure service, the existing custom-resource-type guide still describes the generated schema, extension, Recipe, and Recipe Pack.

Compile and check

The skill compiles in a fresh temporary directory and runs the checker even when Bicep reports an error:

out=$(mktemp -d)
bicep build .radius/app.bicep --diagnostics-format sarif --stdout \
  > "$out/app.json" 2> "$out/app.sarif" || true
node <loaded-skill-base>/scripts/check.mjs "$out/app.json" \
  --diagnostics "$out/app.sarif"

The checker is scripts/check.mjs. It reads the compiled ARM, the compiler diagnostics, and the Azure Recipe Pack pinned to an immutable resource-types-contrib commit.

Compiled ARM can still contain expressions, so the checker resolves only the forms it can prove. It rejects known compiler, ARM, schema, and Recipe-contract defects, but leaves uncertain source and runtime questions to the skill. This is deliberate: letting a bad model pass is safer than rejecting a correct one.

The pinned Recipe contract also supplies provider constraints that compiled ARM can prove, including safe forms for globally scoped names and required managed connection secrets.

An ALLOW result means the checks the script can prove passed. It does not mean the application was deployed or that its main operation worked.

Tests

tests/app-modeling/test_check.mjs contains 107 Node tests. They cover each checker rule, valid counterexamples, malformed input, Recipe contracts, finding deduplication, stable signatures, and expressions the checker cannot safely classify.

[kachawla]

I see that the old reference files have mostly been combined into authoring.md, while rules that can be checked automatically were moved into check.mjs. Could we still keep some separation by topic? Both files are now quite large, and smaller focused files would be easier to understand, test, and update.

It also looks like some useful guidance was removed, including architecture patterns, component detection clues, and information about services without predefined Radius types. These help the skill discover dependencies that the checker cannot catch later.

Thinking about this further.. could we split this work into smaller PRs and move one area at a time? This PR changes a lot at once, which makes it easier to miss behavior and guidance that we have added to the skill over time.

[kachawla]

Since this Recipe Pack is pinned to a specific commit, do we need to download and compile it every time? A temporary GitHub, network, or Bicep issue could make the check fail even when nothing has changed. Could we cache it using the commit SHA?

[kachawla]

Why delete this example?

[kachawla]

Why does this need to be updated?

[kachawla]

Same question here, was the previous format causing issue with the new script?

[kachawla]

Not sure if I'm following the updated wording here.