Support trimming-safe message metadata without assembly scanning

[danielmarbach]

Summary

Short version: this PR gives scanner-disabled trimmed and NativeAOT endpoints a closed-world message metadata model without changing how normal JIT endpoints discover messages.

Today, creating logical message metadata can fall back to runtime type loading and hierarchy reflection. That works for regular JIT deployments, but it is not a reliable contract once trimming or NativeAOT enters the picture.

The safe input is metadata we already resolved at build or startup time: the logical message type and its hierarchy. This PR adds that path, closes the generated registration gaps, proves it with executable trimmed and NativeAOT tests, and only then activates strict registered-only behavior.

This is a larger PR than I would normally prefer. I kept the work together because strict activation is only acceptable once generated registration coverage and executable validation are present. The commits are ordered so reviewers can look at the characterization, safe capability, generated coverage, executable proof, and activation separately.

Compatibility boundary

Strict registered-only mode activates only when:

AssemblyScanner.Disabled &&
(strictRegisteredOnlyMessageMetadataSwitch || !RuntimeFeature.IsDynamicCodeSupported)

The resulting behavior is:

Assembly scanning	Deployment	Behavior
Enabled	Normal JIT	Existing scanning and runtime discovery
Disabled	Normal JIT	Existing runtime fallback
Enabled	Trimmed or NativeAOT	Existing scanning compatibility guard
Disabled	Trimmed or NativeAOT	Pre-registered metadata only

Keeping scanner-disabled normal JIT behavior unchanged is deliberate. Those endpoints exist today, including multi-endpoint hosting scenarios, and a minor release should not silently turn them into closed-world applications.

Strict mode also overrides DynamicTypeLoadingEnabled. That setting continues to control legacy Type.GetType resolution in normal JIT mode, but it cannot make dynamic type loading safe in a trimmed or NativeAOT deployment.

What changed

Cache-only metadata resolution

MessageMetadataRegistry now has cache-only lookups by Type and string identifier. These methods do not invoke conventions, load types, inspect hierarchies, or register metadata on a miss.

The existing runtime discovery path remains available for normal JIT compatibility.

LogicalMessageFactory also has a canonical overload that accepts existing MessageMetadata. Runtime resolution is now:

runtime type
  -> registered metadata
  -> mapper normalization
  -> registered mapped-type metadata
  -> strict failure or legacy runtime discovery

The mapper step matters for interface messages because serializers can return generated proxy instances while the logical message type remains the interface contract.

Generated registration coverage

Handlers, sagas, finder-only saga messages, and explicit registrations now share the same compile-time hierarchy calculation and metadata emitter.

For message types that are not visible through a local handler or saga, endpoints can declare them explicitly:

endpointConfiguration.AddMessageType<OrderPlaced>();

This covers published-only events, sent-only commands, replies, externally supplied contracts, and unobtrusive message conventions.

The method retains an honest reflection fallback for normal JIT applications. Its interceptor replaces that fallback with generated hierarchy registration when source generation succeeds.

Trimmed deployment feature switch

There is no runtime API that reliably tells us an ordinary CoreCLR application was published with trimming. Instead of emitting an assembly marker, Core now follows the same feature-switch pattern used by System.Text.Json.

The internal NServiceBus.EnableStrictRegisteredOnlyMessageMetadata switch is backed by AppContext and annotated with FeatureSwitchDefinition. The build-transitive targets default it to true for executable projects when PublishTrimmed or PublishAot is enabled, then emit it as a RuntimeHostConfigurationOption with Trim="true". This puts the value in the executable's runtime configuration and also lets ILLink treat it as a feature setting.

IsTrimmable, IsAotCompatible, and EnableTrimAnalyzer intentionally do not activate strict runtime behavior. They describe compatibility or enable analysis; they do not prove that the executable being run was trimmed.

Applications can enforce or override the behavior explicitly with:

<NServiceBusEnableStrictRegisteredOnlyMessageMetadata>true</NServiceBusEnableStrictRegisteredOnlyMessageMetadata>

The value can also be supplied through runtimeconfig.template.json or by calling AppContext.SetSwitch before constructing the endpoint configuration. An explicit false value wins over the automatic publish default.

Like other runtime host options, a value introduced only during dotnet publish --no-build cannot update a runtime configuration produced by an earlier build. Split build/publish pipelines must therefore provide the explicit switch during the original build or through the runtime configuration template.

Strict cache-miss behavior

Strict mode is established before the metadata registry initializes. Generated hierarchy registrations can initialize normally, while a bare registration that would require hierarchy reflection fails immediately.

Known handler and saga gaps therefore fail during startup. An outgoing-only type omitted from AddMessageType<T>() can only be detected when the endpoint first sends or publishes it, so that remains a first-use failure.

The exception points to AddMessageType<T>(), AddHandler<T>(), and AddSaga<T>().

Tradeoffs

Explicit registration instead of messaging call-site discovery

I considered discovering every Send<T>, Publish<T>, Reply<T>, and RequestTimeout<T> call automatically.

I am not convinced that gives us a reliable closed-world model. Those calls can sit behind generic application abstractions or live in referenced assemblies, and finding a call does not give the generator a dependable endpoint configuration instance or startup hook.

AddMessageType<T>() is less magical. It is also predictable, additive, and works with unobtrusive conventions. Automatic discovery can still be explored later as an ergonomic layer.

Runtime objects still determine logical metadata

The incoming pipeline does not assign metadata positionally from NServiceBus.EnclosedMessageTypes.

Serializer results do not have a universal one-to-one relationship with that header. JSON and XML can deduplicate polymorphic roots, XML supports legacy multi-message payloads, content-type inference can operate without the header, and interface contracts can deserialize into concrete proxies.

The existing runtime-instance semantics remain intact. The difference is that strict deployments must resolve the resulting logical type from registered metadata.

Metadata registration does not preserve serializer members

MessageMetadata carries logical type identity and a precomputed hierarchy. It does not claim that every constructor or public property required by a serializer is preserved.

That responsibility stays with typed outgoing APIs, generated handler and saga registration, or serializer-specific source generation. Broadly applying the outgoing serialization contract to every metadata carrier would retain more code than this path needs.

Saga accessor issue found along the way

Generated saga correlation accessors were already passed into SagaMetadata.Create, but SagaMetadata did not forward them to SagaMapper.

Making that path live exposed two problems in the generated accessor:

• the receiver was IContainSagaData, while UnsafeAccessor resolves the member against the declared receiver type;
• generated setters had the property type as their return value instead of void.

Accessors are now generated for the concrete saga-data type and keyed by saga-data type plus property identity. The tests execute two accessors whose saga-data classes use the same correlation property name and type.

This bug also exists in the 10.2 line and should be backported as a cohesive fix.

Validation

Local validation includes:

• Release builds for each feature commit;
• the full Core and analyzer test suites;
• normal JIT compatibility tests, including scanner-disabled endpoints;
• package feature-switch tests covering build and buildTransitive assets, executable-only defaults, explicit true/false overrides, and the distinction from IsTrimmable/IsAotCompatible;
• a package-only consumer proving interceptor delivery and the emitted runtime configuration;
• an executable PublishTrimmed endpoint;
• an executable NativeAOT endpoint on macOS ARM64;
• strict-mode failures for missing generated registration.

The ordinary trimmed executable covers handlers, outgoing-only registration, duplicates, and strict diagnostics.

The .NET 10 linker currently crashes while reporting warnings for the full source-generated saga scenario. The NativeAOT executable therefore carries the saga start, handle, and timeout paths. Finder-only registration remains covered by generator and runtime tests because LearningPersistence does not support custom saga finders. Pulling NonDurable persistence into Core would introduce a downstream dependency on a previously released Core package, which seems like the wrong dependency direction for this PR.

XML remains unsupported for trimming. The executable validation uses System.Text.Json.

Follow-ups

The serialization security documentation for DisableDynamicTypeLoading currently says all expected message types must be discovered through assembly scanning. Once this ships, it should also mention explicit and source-generated registration as supported alternatives.

Automatic outgoing call-site discovery remains an optional follow-up.

Please challenge the compatibility boundary and the explicit registration decision in particular. Those are the two choices that shape most of the implementation.

[danielmarbach]

c55ef09 needs to be backported because we have a bug.

[danielmarbach]

@bording if you could give the msbuild stuff a review I'd be grateful

[bording]

@bording if you could give the msbuild stuff a review I'd be grateful

As you've already noticed, I pushed up some tweaks.

[danielmarbach]

@bording if you could give the msbuild stuff a review I'd be grateful

As you've already noticed, I pushed up some tweaks.

Splendid. Thanks!

[DavidBoike]

Does returning null result in "no handlers found for this message type" which results in error queue?

[danielmarbach]

• Returning null does not directly cause "no handlers found."
• With allowContentTypeInference enabled, deserialization continues and serializers such as XML can infer the type from the body.
• With inference disabled, deserialization fails with MessageDeserializationException and the unrecoverable message goes to the error queue.
• "No handlers found" can occur later only if inference succeeds but the inferred type has no handler.

[DavidBoike]

Cool - the final result of error queue (and not message loss) is all I was wanting to verify.

[DavidBoike]

Cool - the final result of error queue (and not message loss) is all I was wanting to verify.