Skip to content

Reject starts for disabled orchestrators - #3514

Merged
wangbill (YunchuWang) merged 16 commits into
devfrom
fix-1933-disabled-orchestrator
Aug 19, 2026
Merged

wangbill (YunchuWang) merged 16 commits into
devfrom
fix-1933-disabled-orchestrator

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Aug 6, 2026 •

Copy link
Copy Markdown
Member

Fixes #1933

Problem

A function disabled at host startup is still indexed by name, but its RegisteredFunctionInfo remains null because no listener is created. Starts could therefore appear successful and remain pending indefinitely.

Change

  • Reject new and restarted current-app instances only when the orchestrator is known to be startup-disabled.
  • Apply the disabled-function check to direct clients, the HTTP management API, and the gRPC sidecar before provider scheduling.
  • Return HTTP 400 and gRPC InvalidArgument while retaining ArgumentException compatibility for direct clients.
  • Keep active and deregistered listener-drain behavior and deterministic replay unchanged.

Compatibility intentionally preserved

Disabled validation deliberately reuses the existing ClientReferencesCurrentApp predicate used by direct missing-function validation. This PR does not change current-app classification:

  • ExternalClient=true bypasses local validation.
  • TaskHub comparison remains exact-case.
  • Connection comparison retains its existing behavior, including same-hub/different-connection self-comparison.
  • Missing orchestrator/entity behavior is unchanged.
  • HTTP and gRPC still allow unknown function names.
  • No provider-specific TaskHub casing or host-default-provider connection correction is included.

Tests

Coverage includes direct, HTTP, and gRPC starts and restarts; exact and case-variant TaskHubs; same-hub/different-connection behavior; external clients; missing, disabled, active, and deregistered functions; provider call suppression; replay safety; and an E2E assertion that a rejected disabled start is never persisted.

  • Focused current-app gate and disabled-path tests: 27 passed on net8.0; 27 passed on net10.0
  • Full relevant unit classes (DurableClientBaseTests, DisabledFunctionDispatchTests, HttpApiHandlerTests, LocalGrpcListenerTests): 183 passed on net8.0; 183 passed on net10.0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Copilot AI lite review requested due to automatic review settings August 6, 2026 03:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a gap where orchestrator functions that are indexed but startup-disabled could still be “started” via optimized HTTP/gRPC paths (and sometimes the durable client), leading to instances that appear accepted but can remain pending indefinitely. It introduces a consistent “current-app orchestrator availability” validation across entry points while preserving cross-app (remote hub/connection) starts and replay behavior.

Changes:

  • Add a dedicated exception + extension helper to detect missing/startup-disabled orchestrators and surface a consistent error message.
  • Enforce the validation for current-app starts/restarts across DurableClient, HTTP management API, and gRPC sidecar paths (with appropriate HTTP 400 / gRPC InvalidArgument mapping).
  • Make “current-app” detection match task hub names case-insensitively and ensure connection matching is based on the extension’s default provider.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
test/FunctionsV2/Tests/Unit/LocalGrpcListenerTests.cs Adds gRPC coverage for rejecting disabled orchestrator start/restart and for bypassing validation on external targets; updates helper wiring to configure extension and vary providers.
test/FunctionsV2/Tests/Unit/HttpApiHandlerTests.cs Adds HTTP coverage for rejecting disabled orchestrator start/restart (400) and for allowing external targets without local validation.
test/FunctionsV2/Tests/Unit/DurableClientBaseTests.cs Adds direct-client coverage for throwing on disabled orchestrator start/restart and for allowing deregistered listener drain behavior.
test/FunctionsV2/Tests/Unit/DisabledFunctionDispatchTests.cs Adds unit coverage ensuring the new “missing/disabled orchestrator” check doesn’t break active out-of-proc orchestrators or deterministic replay.
src/WebJobs.Extensions.DurableTask/TaskHubGrpcServer.cs Validates current-app orchestrator availability for gRPC starts and maps restart validation failures to InvalidArgument.
src/WebJobs.Extensions.DurableTask/HttpApiHandler.cs Validates current-app orchestrator availability for HTTP start path and maps restart validation failures to HTTP 400 with the correct message.
src/WebJobs.Extensions.DurableTask/Exceptions/OrchestratorFunctionUnavailableException.cs Introduces a dedicated ArgumentException-compatible exception to represent missing/startup-disabled/non-orchestrator availability failures.
src/WebJobs.Extensions.DurableTask/DurableTaskExtension.cs Adds ThrowIfOrchestratorFunctionIsMissingOrDisabled based on null orchestrator registration semantics.
src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs Applies the new validation to direct client start/restart when referencing the current app; fixes current-app detection to be case-insensitive for hub names and to compare against the default provider for connection matching.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Copilot AI review requested due to automatic review settings August 6, 2026 04:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs:380

  • ReferencesCurrentApp() now treats task hub names case-insensitively, but ConnectionNameMatchesCurrentApp uses DurabilityProvider.ConnectionNameMatches(), which is case-sensitive. This can misclassify a client that targets the current app but uses a different-case connection name (e.g., "Storage" vs "storage") as an external target, skipping the disabled-orchestrator validation that this PR is adding. Other connection-name comparisons in this codebase use OrdinalIgnoreCase (e.g., GetDurableClient and AzureStorageDurabilityProviderFactory).
        private bool ConnectionNameMatchesCurrentApp(DurableClient client)
        {
            return this.config != null &&
                this.config.DefaultDurabilityProvider.ConnectionNameMatches(client.DurabilityProvider);
        }

Copilot AI review requested due to automatic review settings August 14, 2026 02:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

@YunchuWang
wangbill (YunchuWang) marked this pull request as ready for review August 14, 2026 02:31
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356
Copilot AI review requested due to automatic review settings August 14, 2026 16:36
This reverts commit a455865.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/WebJobs.Extensions.DurableTask/TaskHubGrpcServer.cs:535

  • Typo in the comment: "Thrown when th instanceId is not found." should be "the instanceId".
                throw new RpcException(new Status(StatusCode.InvalidArgument, ex.Message));
            }
            catch (ArgumentException ex)
            {

Copilot AI review requested due to automatic review settings August 14, 2026 16:42
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 14, 2026 16:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs:392

  • ConnectionNameMatchesCurrentApp currently relies on DurabilityProvider.ConnectionNameMatches, which compares connection names case-sensitively. Elsewhere in the codebase (e.g., DurableClientAttribute equality and AzureStorageDurabilityProviderFactory) TaskHub/ConnectionName matching is treated as case-insensitive, so this can incorrectly treat a case-variant connection name as an external target and skip the disabled-orchestrator rejection for what is effectively the same app setting key.
        private bool ConnectionNameMatchesCurrentApp(DurableClient client)
        {
            return this.config != null &&
                this.config.DefaultDurabilityProvider.ConnectionNameMatches(client.DurabilityProvider);
        }

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356
Copilot AI review requested due to automatic review settings August 18, 2026 15:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/WebJobs.Extensions.DurableTask/TaskHubGrpcServer.cs:65

  • The disabled-orchestrator validation can throw OrchestratorFunctionUnavailableException here, but StartInstance currently only has a generic catch (ArgumentException) that wraps everything as an "Invalid argument for start instance request for instance ID ..." message. This is misleading because the failure is about the orchestrator name being disabled/unavailable (not the instance ID), and it’s inconsistent with RestartInstance which already maps OrchestratorFunctionUnavailableException directly to InvalidArgument.

Consider catching OrchestratorFunctionUnavailableException close to where it can be thrown (or adding a dedicated catch) and returning StatusCode.InvalidArgument with the original message.

                if (this.GetClient(context) is DurableClient durableClient && durableClient.ReferencesCurrentApp())
                {
                    this.extension.ThrowIfOrchestratorFunctionIsDisabled(request.Name);
                }

Copilot AI review requested due to automatic review settings August 18, 2026 17:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e84c5737-8808-4d03-901a-49afc08fbcf0
Copilot AI review requested due to automatic review settings August 19, 2026 13:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 19, 2026 16:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e84c5737-8808-4d03-901a-49afc08fbcf0
Copilot AI review requested due to automatic review settings August 19, 2026 21:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unexpected Behavior Starting Disabled Orchestrator Function

3 participants