Repository navigation
Reject starts for disabled orchestrators - #3514
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
There was a problem hiding this comment.
Pull request overview
This PR fixes a gap where orchestrator functions that are indexed but startup-disabled could still be “started” via optimized HTTP/gRPC paths (and sometimes the durable client), leading to instances that appear accepted but can remain pending indefinitely. It introduces a consistent “current-app orchestrator availability” validation across entry points while preserving cross-app (remote hub/connection) starts and replay behavior.
Changes:
- Add a dedicated exception + extension helper to detect missing/startup-disabled orchestrators and surface a consistent error message.
- Enforce the validation for current-app starts/restarts across DurableClient, HTTP management API, and gRPC sidecar paths (with appropriate HTTP 400 / gRPC
InvalidArgumentmapping). - Make “current-app” detection match task hub names case-insensitively and ensure connection matching is based on the extension’s default provider.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/FunctionsV2/Tests/Unit/LocalGrpcListenerTests.cs | Adds gRPC coverage for rejecting disabled orchestrator start/restart and for bypassing validation on external targets; updates helper wiring to configure extension and vary providers. |
| test/FunctionsV2/Tests/Unit/HttpApiHandlerTests.cs | Adds HTTP coverage for rejecting disabled orchestrator start/restart (400) and for allowing external targets without local validation. |
| test/FunctionsV2/Tests/Unit/DurableClientBaseTests.cs | Adds direct-client coverage for throwing on disabled orchestrator start/restart and for allowing deregistered listener drain behavior. |
| test/FunctionsV2/Tests/Unit/DisabledFunctionDispatchTests.cs | Adds unit coverage ensuring the new “missing/disabled orchestrator” check doesn’t break active out-of-proc orchestrators or deterministic replay. |
| src/WebJobs.Extensions.DurableTask/TaskHubGrpcServer.cs | Validates current-app orchestrator availability for gRPC starts and maps restart validation failures to InvalidArgument. |
| src/WebJobs.Extensions.DurableTask/HttpApiHandler.cs | Validates current-app orchestrator availability for HTTP start path and maps restart validation failures to HTTP 400 with the correct message. |
| src/WebJobs.Extensions.DurableTask/Exceptions/OrchestratorFunctionUnavailableException.cs | Introduces a dedicated ArgumentException-compatible exception to represent missing/startup-disabled/non-orchestrator availability failures. |
| src/WebJobs.Extensions.DurableTask/DurableTaskExtension.cs | Adds ThrowIfOrchestratorFunctionIsMissingOrDisabled based on null orchestrator registration semantics. |
| src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs | Applies the new validation to direct client start/restart when referencing the current app; fixes current-app detection to be case-insensitive for hub names and to compare against the default provider for connection matching. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 47efed8f-40ca-43b8-8564-de5406bafbc0
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs:380
- ReferencesCurrentApp() now treats task hub names case-insensitively, but ConnectionNameMatchesCurrentApp uses DurabilityProvider.ConnectionNameMatches(), which is case-sensitive. This can misclassify a client that targets the current app but uses a different-case connection name (e.g., "Storage" vs "storage") as an external target, skipping the disabled-orchestrator validation that this PR is adding. Other connection-name comparisons in this codebase use OrdinalIgnoreCase (e.g., GetDurableClient and AzureStorageDurabilityProviderFactory).
private bool ConnectionNameMatchesCurrentApp(DurableClient client)
{
return this.config != null &&
this.config.DefaultDurabilityProvider.ConnectionNameMatches(client.DurabilityProvider);
}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356
This reverts commit a455865. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/WebJobs.Extensions.DurableTask/TaskHubGrpcServer.cs:535
- Typo in the comment: "Thrown when th instanceId is not found." should be "the instanceId".
throw new RpcException(new Status(StatusCode.InvalidArgument, ex.Message));
}
catch (ArgumentException ex)
{
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs:392
- ConnectionNameMatchesCurrentApp currently relies on DurabilityProvider.ConnectionNameMatches, which compares connection names case-sensitively. Elsewhere in the codebase (e.g., DurableClientAttribute equality and AzureStorageDurabilityProviderFactory) TaskHub/ConnectionName matching is treated as case-insensitive, so this can incorrectly treat a case-variant connection name as an external target and skip the disabled-orchestrator rejection for what is effectively the same app setting key.
private bool ConnectionNameMatchesCurrentApp(DurableClient client)
{
return this.config != null &&
this.config.DefaultDurabilityProvider.ConnectionNameMatches(client.DurabilityProvider);
}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cf4a7ba-c821-42dd-ba14-0333dddc5356
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/WebJobs.Extensions.DurableTask/TaskHubGrpcServer.cs:65
- The disabled-orchestrator validation can throw OrchestratorFunctionUnavailableException here, but StartInstance currently only has a generic
catch (ArgumentException)that wraps everything as an "Invalid argument for start instance request for instance ID ..." message. This is misleading because the failure is about the orchestrator name being disabled/unavailable (not the instance ID), and it’s inconsistent with RestartInstance which already maps OrchestratorFunctionUnavailableException directly to InvalidArgument.
Consider catching OrchestratorFunctionUnavailableException close to where it can be thrown (or adding a dedicated catch) and returning StatusCode.InvalidArgument with the original message.
if (this.GetClient(context) is DurableClient durableClient && durableClient.ReferencesCurrentApp())
{
this.extension.ThrowIfOrchestratorFunctionIsDisabled(request.Name);
}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e84c5737-8808-4d03-901a-49afc08fbcf0
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e84c5737-8808-4d03-901a-49afc08fbcf0
Fixes #1933
Problem
A function disabled at host startup is still indexed by name, but its
RegisteredFunctionInforemainsnullbecause no listener is created. Starts could therefore appear successful and remain pending indefinitely.Change
InvalidArgumentwhile retainingArgumentExceptioncompatibility for direct clients.Compatibility intentionally preserved
Disabled validation deliberately reuses the existing
ClientReferencesCurrentApppredicate used by direct missing-function validation. This PR does not change current-app classification:ExternalClient=truebypasses local validation.Tests
Coverage includes direct, HTTP, and gRPC starts and restarts; exact and case-variant TaskHubs; same-hub/different-connection behavior; external clients; missing, disabled, active, and deregistered functions; provider call suppression; replay safety; and an E2E assertion that a rejected disabled start is never persisted.
DurableClientBaseTests,DisabledFunctionDispatchTests,HttpApiHandlerTests,LocalGrpcListenerTests): 183 passed on net8.0; 183 passed on net10.0