Skip to content

Add Entra authentication to Durable distributed tracing - #3512

Merged
wangbill (YunchuWang) merged 22 commits into
Azure:devfrom
YunchuWang:yunchuwang-fix-3497-entra-tracing
Aug 26, 2026
Merged

wangbill (YunchuWang) merged 22 commits into
Azure:devfrom
YunchuWang:yunchuwang-fix-3497-entra-tracing

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Aug 6, 2026 •

Copy link
Copy Markdown
Member

Summary

  • apply APPLICATIONINSIGHTS_AUTHENTICATION_STRING to Durable's private Application Insights pipeline
  • support both system-assigned and user-assigned managed identities
  • forward Durable telemetry to the Functions host's Application Insights channel, which already carries the host's Entra credential, so Durable never creates or reads an Azure.Core type
  • preserve Durable's processor-free telemetry configuration, V1/V2 behavior, test hooks, and lifecycle ownership
  • log only the selected identity type; configured IDs and authentication-string contents are never logged

Fixes #3497

Root cause

Durable distributed tracing creates a private TelemetryConfiguration. It copied the Application Insights instrumentation key or connection string, but not the Microsoft Entra credential configured by the Functions host. With DisableLocalAuth=true, host telemetry stayed authenticated while Durable Request/Dependency telemetry was rejected at ingestion.

The obvious fix — construct a ManagedIdentityCredential and call SetAzureTokenCredential on Durable's configuration — does not work inside the Functions host. Application Insights 2.22 resolves Azure.Core.TokenCredential through ReflectionCredentialEnvelope, which binds the type in the host/default load context. A credential constructed in the function app load context is a different type identity and is rejected with ArgumentException. This was confirmed on a live Function App, not inferred.

Implementation

Durable does not obtain a credential at all. It wraps the host's ITelemetryChannel and forwards its own telemetry through it. The host channel already holds the credential envelope and the Entra ingestion endpoint, so the authentication problem disappears without Durable ever touching an Azure.Core type — no reflection, and no coupling to the host's load context or to Application Insights internals.

  • parse the documented authentication-string forms using the WebJobs Application Insights parser:
    • system-assigned: Authorization=AAD
    • user-assigned: Authorization=AAD;ClientId=<CLIENT_ID>
  • add a host-aware TelemetryActivator constructor while retaining the existing two-argument public constructor
  • add HostForwardingTelemetryChannel, which forwards Send/Flush to the host channel
  • retain direct ManagedIdentityCredential fallback behavior for standalone WebJobs hosts, where there is no host channel to reuse
  • emit sanitized warnings for malformed settings or configurations that cannot be applied
  • document the required settings and Monitoring Metrics Publisher role

Three properties of the Application Insights SDK make the wrapper safe, and they are structural rather than conventions this code has to uphold:

  • ISupportCredentialEnvelope is internal, so an external wrapper cannot implement it. The TelemetryChannel setter's call to SetTelemetryChannelCredentialEnvelope is therefore a no-op on the wrapper, and Durable's credential-free configuration cannot clear the host's credential.
  • SetTelemetryChannelEndpoint only writes EndpointAddress for InMemoryChannel and ServerTelemetryChannel. A custom channel is never written to, so configuring Durable's connection string cannot downgrade the host to the non-Entra endpoint.
  • Dispose is intentionally empty. The host owns the channel's lifetime, and Durable's configuration must not be able to stop host telemetry.

The host TelemetryConfiguration itself is still intentionally not reused for telemetry. Host telemetry processors include operation filtering that is incompatible with Durable operations that do not carry the host Category property; retaining the private configuration avoids the missing-span behavior that led to the earlier revert in #3053. Only the channel — the transport — is shared.

Real Azure-hosted validation

All runs used temporary Azure Functions v4 Windows Consumption apps, real Azure Storage, workspace-based Application Insights with DisableLocalAuth=true, and real managed identities. Exactly one identity at a time had Monitoring Metrics Publisher scoped to the Application Insights component. No application code used AzureCliCredential.

Current head 7a435487 — channel forwarding, .NET 8 in-process

A diagnostic endpoint resolved the production-registered ITelemetryActivator from the live host and reported the wiring:

{
  "activatorType": "...DurableTask.Correlation.TelemetryActivator",
  "durableChannelType": "...DurableTask.Correlation.HostForwardingTelemetryChannel",
  "hostChannelType": "Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel",
  "hostChannelAssembly": "Microsoft.AI.ServerTelemetryChannel",
  "wrappedIsHostChannel": true,
  "durableConfigurationIsHostConfiguration": false
}

The wrapper holds the host's real ServerTelemetryChannel instance, and Durable's configuration is still its own — so the isolated processor/module pipeline that #3053 depended on is intact.

Identity Orchestration Durable Requests Durable Dependencies Result
User-assigned Completed 2 2 One operation ID; full parent chain
System-assigned Completed 2 2 One operation ID; full parent chain

Ingested span chain, identical in both runs (user-assigned shown):

request     Issue3497Start                     <- host control telemetry
dependency  create_orchestration:Orchestrator  <- parent: Issue3497Start
request     orchestration:Orchestrator         <- parent: create_orchestration
dependency  activity:Activity                  <- parent: orchestration
request     activity:Activity                  <- parent: activity dependency
request     Issue3497Status                    <- host control telemetry

Initialization logged the expected identity type for each run, and the host-channel-unavailable warning never fired. Ordinary host telemetry continued to flow throughout, confirming the shared channel was not disturbed.

Earlier heads

The same harness previously reproduced the bug and validated the credential-reuse approach. Those results still establish the failure mode and the breadth of the fix.

Phase Runtime Identity Durable Requests Durable Dependencies Result
Official 3.14.0 .NET 8 in-process UAMI 0 0 Reproduced #3497
Official 3.14.0 .NET 8 isolated UAMI 0 0 Host/control telemetry and MI token acquisition succeeded
Patched .NET 8 in-process UAMI / System 2 2 Full parent chain
Patched host DLL .NET 8 isolated UAMI / System 2 2 Full parent chain
Patched host DLL Node.js out-of-process System 2 2 Full parent chain

The isolated and Node.js runs confirm the fix applies to host-generated Durable spans independently of the language worker. Worker-direct Application Insights/OpenTelemetry remains a separate pipeline. Normal non-.NET rollout still requires a Durable extension release followed by an extension-bundle release.

Automated validation

  • TelemetryActivatorTests: 10/10 on net8.0 and net10.0
  • CorrelationEndToEndTests: 37/37 on net8.0 and net10.0
  • Release build/package: 0 warnings, 0 errors
  • deployed host DLL hashes matched their local publish payloads exactly

New coverage for this design:

  • forwarding is selected when the host exposes a configuration, and the wrapper holds the host's own channel
  • telemetry tracked on Durable's configuration physically arrives on the host channel
  • the host's credential survives Durable's wiring, and Durable's configuration still has none
  • configuring Durable's connection string does not overwrite the host channel's endpoint
  • disposing Durable's configuration does not dispose the host channel, which keeps sending afterwards
  • the OnSend test channel is preserved rather than replaced
  • standalone hosts still fall back to a managed identity credential, for both identity forms
  • end to end: with the activator built purely by production DI, all four Durable spans arrive through the forwarding channel with the correct parent chain

Cleanup and security

  • all temporary resource groups were deleted
  • ARM/subscription checks confirmed no matching Function App, storage account, managed identity, workspace, Application Insights component, or scoped role assignment remained
  • deployment archives, publishing material, binaries, and transient credentials were removed
  • retained evidence and source artifacts passed secret/resource-identifier scans

Configuration

The connection string remains required:

APPLICATIONINSIGHTS_CONNECTION_STRING=<connection string>

Choose one authentication form:

# System-assigned identity
APPLICATIONINSIGHTS_AUTHENTICATION_STRING=Authorization=AAD

# User-assigned identity
APPLICATIONINSIGHTS_AUTHENTICATION_STRING=Authorization=AAD;ClientId=<client ID>

The selected identity needs Monitoring Metrics Publisher on the Application Insights component.

Official references:

Reuse the WebJobs Application Insights authentication-string parser and apply the selected managed identity credential to Durable's owned telemetry configuration. Keep the existing lifecycle so host telemetry processors cannot filter Durable spans.

Add system/user identity parsing coverage, malformed-setting warnings, and a V2 request/dependency parent-chain regression test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01711e67-44d5-4e79-915a-a9e6d7dafe26
Copilot AI lite review requested due to automatic review settings August 6, 2026 00:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables Microsoft Entra (AAD) authentication for Durable Functions’ private Application Insights telemetry pipeline used by Distributed Tracing, by honoring APPLICATIONINSIGHTS_AUTHENTICATION_STRING and configuring a managed-identity TokenCredential on the TelemetryConfiguration. This addresses missing Durable V2 spans when the App Insights resource has DisableLocalAuth=true.

Changes:

  • Update TelemetryActivator to parse APPLICATIONINSIGHTS_AUTHENTICATION_STRING (using WebJobs’ parser semantics) and apply a ManagedIdentityCredential via TelemetryConfiguration.SetAzureTokenCredential(...), with a safe warning on invalid strings.
  • Add E2E/activation tests covering valid/invalid authentication strings and verifying that V2 module span parent/child chains are preserved under Entra auth.
  • Add the necessary WebJobs Application Insights logging package reference and document Entra-only setup in the V2 tracing sample.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/WebJobs.Extensions.DurableTask/Correlation/TelemetryActivator.cs Reads and parses APPLICATIONINSIGHTS_AUTHENTICATION_STRING, applies a managed-identity credential to Durable’s owned AI telemetry pipeline, and logs success/warning without exposing configured values.
src/WebJobs.Extensions.DurableTask/WebJobs.Extensions.DurableTask.csproj Adds a runtime package reference needed to reuse WebJobs’ authentication-string parsing semantics.
Directory.Packages.props Centralizes the Microsoft.Azure.WebJobs.Logging.ApplicationInsights version for consistent dependency management.
test/Directory.Packages.props Removes the redundant per-test package version entry now that it is centrally managed.
test/FunctionsV2/Tests/E2E/CorrelationEndToEndTests.cs Adds coverage for Entra auth enablement/warnings and validates V2 telemetry span chaining with Entra auth enabled.
samples/distributed-tracing/v2/DistributedTracingSample/README.md Documents Entra-only App Insights setup (APPLICATIONINSIGHTS_AUTHENTICATION_STRING) for Distributed Tracing V2.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings August 6, 2026 18:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

wangbill (YunchuWang) and others added 2 commits August 6, 2026 12:49
Reuse the host-configured Application Insights credential when available so Durable's private telemetry pipeline uses the Functions host assembly-load context. Retain the standalone fallback and cover credential extraction and DI construction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 00d274e1-e73b-45f4-9592-18fa90d81936
Verify dependency injection selects the host-aware constructor and distinguish malformed authentication strings from credentials that cannot be applied. Document why the host credential must be reused across Functions load contexts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01711e67-44d5-4e79-915a-a9e6d7dafe26
Copilot AI review requested due to automatic review settings August 6, 2026 20:02
Extract the host-reuse and standalone-fallback decision into a focused helper. Verify the host credential is applied to Durable's owned telemetry configuration and both managed identity authentication forms retain their fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 00d274e1-e73b-45f4-9592-18fa90d81936

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@YunchuWang
wangbill (YunchuWang) requested a balanced review from Copilot August 6, 2026 22:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

ApplyAzureTokenCredential now reports whether the Functions host credential
was reused. When a host TelemetryConfiguration is present but the credential
cannot be read, Durable falls back to creating its own managed identity
credential, which the host Application Insights SDK rejects when the host and
the function app load different Azure.Core versions. That failure was silent
and reproduced the original missing-span symptom. It is now logged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 01711e67-44d5-4e79-915a-a9e6d7dafe26
Comment thread Directory.Packages.props Resolved
Microsoft.Azure.WebJobs.Logging.ApplicationInsights 3.0.41 transitively
resolved System.Drawing.Common 4.7.0 (GHSA-rxg9-xrhp-64gj, Critical) on
both target frameworks. Move to 3.0.45 so it lines up with the existing
Microsoft.Azure.WebJobs 3.0.45 reference; the transitive graph now
resolves System.Drawing.Common 6.0.0 and the audit is clean. That newer
graph also raises the transitive requirement above the test-side pin of
4.7.3, so raise that pin to 6.0.0 to keep central pinning consistent.

The correlation E2E test asserted on an activator the test helper built
by hand with the two-argument constructor, so it never exercised the
host-aware constructor that the fix depends on. Let the container build
the activator and only attach the OnSend hook, register a host
TelemetryConfiguration the way the Functions host does, and assert the
credential Durable applies to its own configuration is the very same
instance the host owns. The fallback case is covered too: no host
credential means a standalone managed identity credential plus the
warning. Reverting the reuse logic fails both new assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01711e67-44d5-4e79-915a-a9e6d7dafe26
Copilot AI review requested due to automatic review settings August 13, 2026 21:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Upgrading Microsoft.Azure.WebJobs.Logging.ApplicationInsights to 3.0.45
changes the transitive graph so Microsoft.Data.SqlClient's
System.Configuration.ConfigurationManager 6.0.0 chain now requires
System.Drawing.Common >= 6.0.0. Two remaining 4.7.3 pins were lower than
that and failed restore with NU1605/NU1109 (warnings are errors).

Verified by control: with 3.0.41 both restore clean, with 3.0.45 both
fail; after raising the pins all 25 projects that reference the Durable
extension restore clean.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01711e67-44d5-4e79-915a-a9e6d7dafe26
Copilot AI review requested due to automatic review settings August 13, 2026 21:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/WebJobs.Extensions.DurableTask/Correlation/TelemetryActivator.cs:270

  • APPLICATIONINSIGHTS_CONNECTION_STRING is required for Microsoft Entra ingestion, but this branch also runs when only APPINSIGHTS_INSTRUMENTATIONKEY is present. That unsupported configuration receives a credential and the “authentication enabled” success log even though its Durable spans cannot be ingested. Gate authentication on connectionStringProvided and emit a sanitized warning when the authentication string is configured without a connection string.
            if (!string.IsNullOrWhiteSpace(resolvedAuthenticationString))

samples/distributed-tracing/v2/DistributedTracingSample/README.md:31

  • The preceding paragraph permits either a connection string or instrumentation key, so “also configure” incorrectly presents APPINSIGHTS_INSTRUMENTATIONKEY plus this authentication string as supported. Microsoft Entra ingestion requires APPLICATIONINSIGHTS_CONNECTION_STRING; make that prerequisite explicit here.
If the Application Insights resource has local authentication disabled, also configure `APPLICATIONINSIGHTS_AUTHENTICATION_STRING`. Use `Authorization=AAD` for the function app's system-assigned managed identity, or `Authorization=AAD;ClientId=<USER_ASSIGNED_CLIENT_ID>` for a user-assigned managed identity. The selected identity needs the `Monitoring Metrics Publisher` role on the Application Insights resource. Microsoft Entra authentication for Application Insights isn't supported by the Functions host during local development.

Copilot AI review requested due to automatic review settings August 18, 2026 23:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 19, 2026 22:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

@YunchuWang

wangbill (YunchuWang) commented Aug 20, 2026 •

Copy link
Copy Markdown
Member Author

query_data.csv
cc Varshitha Bachu (@bachuv) for local manual test traces captured from app insights

Copilot AI review requested due to automatic review settings August 20, 2026 17:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 21, 2026 00:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/WebJobs.Extensions.DurableTask/Correlation/HostForwardingTelemetryChannel.cs:97

  • HostForwardingTelemetryChannel advertises the IAsyncFlushable contract, where false means the telemetry transfer failed (or was cancelled), but this path returns false after Flush() completes successfully. Consequently, TelemetryClient.FlushAsync reports a failure for every synchronous host channel even though no failure occurred. Return true after the successful synchronous flush (and update the corresponding assertion in TelemetryActivatorTests).
            this.hostChannel.Flush();
            return Task.FromResult(false);

Copilot AI review requested due to automatic review settings August 21, 2026 22:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Directory.Build.props pins System.Drawing.Common 6.0.0 for the generated
WorkerExtensions.csproj, but two companion comments still described a 4.7.3
override. Update them so future maintenance reflects the actual pin.

Comment-only change; no MSBuild behavior is affected. The repo-root
Directory.Packages.props 4.7.3 CPM pin is intentionally unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bb9b7efb-7caf-485a-bc72-5eeee04310b8
Copilot AI review requested due to automatic review settings August 25, 2026 13:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

@YunchuWang
wangbill (YunchuWang) merged commit f0688f8 into Azure:dev Aug 26, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Distributed Tracing V2 emits no telemetry when Application Insights local auth is disabled (regression of #2870, fix reverted by #3053)

3 participants