Skip to content

Fix V2 Orchestrate Ask an agent steps refused before the agent runs - #1701

Merged
Paul Lizer (paullizer) merged 3 commits into
paullizer-react-v2-uifrom
paullizer-orchestrate-agent-unavailable-investigat
Oct 7, 2026
Merged

Paul Lizer (paullizer) merged 3 commits into
paullizer-react-v2-uifrom
paullizer-orchestrate-agent-unavailable-investigat

Conversation

@paullizer

Copy link
Copy Markdown
Collaborator

Summary

  • Problem: with Orchestrate on, every Ask an agent step was refused just before the agent ran, with "This step's agent or action isn't available to you right now". It happened in personal and shared conversations, while the same agent answered with Orchestrate off. 0.261.289 removed an earlier refusal, so these steps then failed at this later check.
  • Root cause: OrchestrationExternalSourceProvider._resolve_integration required the agent returned by resolve_delegation_agent to be exactly a dict. azure-cosmos 4.9.0 point reads return CosmosDict, a dict subclass that deepcopy keeps, so every agent was refused with result_external_source_unavailable. Action steps were unaffected because that branch already used isinstance.
  • Fix, at both layers: _canonical_agent now returns a plain dict, which fixes all 8 callers of resolve_delegation_agent. The provider also accepts any dict from its agent resolver, copies it into a plain one, and still requires the exact scoped reference (ID, scope type and scope ID) to match. The settings check that 0.261.237 deliberately kept exact is unchanged.
  • Diagnostics: one refusal code came from nine places, and finding this needed request-level Azure SDK logs. Each provider refusal now logs [ORCHESTRATION_EXTERNAL_SOURCES] A step's source was refused. once, with sc_stage, sc_authority_reason and sc_reason (the check that failed). It carries hashed identifiers only and joins the executor's failure event on the run and step hashes. The user's message and the executor's event are unchanged.
  • Test seam: the shared fake (DelegationServices) now returns point reads as CosmosDictLike, so a check that requires exactly dict on a stored record fails in tests as it did in production. New regression suite: test_orchestration_agent_document_type_fix.py.

Linked issue

Fixes #1699

Refs #1661 (agents in Orchestrate; its expected behavior also depends on this fix)

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

VERSION is 0.261.291. The base is at 0.261.290 after #1700, and this branch has merged the base. A parallel fix for Orchestrate chart delivery is meant to merge first. If it takes 0.261.291, this branch renumbers to the next patch in config.py, the fix doc, the test header and version check, the docs rows, the module docstring and the release notes.

Testing / validation

  • python -m pytest functional_tests/test_orchestration_agent_document_type_fix.py -q: 31 passed. On the previous code, 27 of the 31 fail.
  • python -m pytest over the 18 suites that use DelegationServices: 98 failed, 968 passed, 232 subtests passed, identical to the base. The suites are the 6 test_agent_delegation_* suites, test_workspace_authoring_backend.py, test_app_settings_store_plain_dict_reads.py and the 10 orchestration external-source suites. Results are also identical when point reads return the real azure-cosmos 4.9.0 CosmosDict. Before the fix, SDK-typed point reads made 35 tests and 9 subtests fail, all at the same agent check. The 98 failures already fail on the base: 79 in test_workspace_authoring_backend.py ("Unable to complete this workspace request"), and 19 NameErrors in test_agent_delegation_action.py and test_agent_delegation_workflow_actor.py.
  • python -m pytest functional_tests/test_orchestration_agent_selection.py functional_tests/test_orchestration_v2_plan_backend.py functional_tests/test_orchestration_external_identity.py functional_tests/test_orchestration_external_bootstrap.py functional_tests/test_orchestration_external_pre_effect.py -q: 202 passed, 180 subtests passed.
  • After merging the base, python -m pytest functional_tests/test_orchestration_agent_document_type_fix.py functional_tests/test_orchestration_seeded_agent_preference_fix.py functional_tests/test_orchestration_session_trusted_sources.py functional_tests/test_orchestration_external_sources.py functional_tests/test_v2_sidebar_primary_nav_stays_in_v2.py -q: 113 passed, 102 subtests passed.
  • python functional_tests/test_docs_site_quality.py: 6/6 passed. python functional_tests/test_docs_app_surface_coverage.py: 7/7 passed.
  • python -m pytest functional_tests/test_logging_tag_standardization.py: fails, as it does on the base, on non-standard tags in other files (EditorAssist, Workflow*). The new [ORCHESTRATION_EXTERNAL_SOURCES] tag is detected in source and listed in the inventory.
  • Environment: Python 3.12 in a venv with cryptography==46.0.7 and azure-cosmos==4.9.0 (the production pin). With cryptography 50, pyOpenSSL 25.3.0 doesn't import, so five of the suites can't be collected.
  • Not yet run on a deployment: V2 with Orchestrate and a picked agent that uses Microsoft 365 actions, in a personal and a shared conversation, with a KQL check for agent_invoke failures. Such an agent may next stop at the existing Microsoft 365 sign-in or approval checks, which have their own messages.

Documentation

  • Release notes updated, or not needed: a v0.261.291 entry, plus the missing v0.261.289 entry for the selected-agent fix.
  • Feature documentation updated, or not needed: ORCHESTRATION_EXTERNAL_SOURCE_ACCESS.md (resolver contract and refusal logging), docs/admin/orchestration.md (troubleshooting rows, Actions and agents) and docs/reference/logging-tags.md (new tag, event, properties and a KQL example).
  • Fix documentation updated, or not needed: new ORCHESTRATION_AGENT_DOCUMENT_TYPE_FIX.md, and a follow-up note in ORCHESTRATION_SELECTED_AGENT_DISABLED_PREFERENCE_FIX.md.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security()) (no new routes)
  • Settings sent to non-admin frontends use sanitize_settings_for_user() (no settings reach a frontend)
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS (no browser changes)
  • No secrets, keys, connection strings, or local-only artifacts are included. Refusal events log only codes and SHA-256 hashes, and tests check that no user, agent, group or selector identifiers reach them.

Paul Lizer (paullizer) and others added 3 commits October 7, 2026 15:13
Every orchestrated Ask an agent step was refused just before the agent ran,
with "This step's agent or action isn't available to you right now". The
external-source provider required the agent returned by
resolve_delegation_agent to be exactly a dict, and azure-cosmos 4.9.0 point
reads return CosmosDict, a dict subclass that deepcopy keeps.

- resolve_delegation_agent returns the stored agent as a plain dict, for
  every caller.
- The provider accepts any dict from its agent resolver, copies it and still
  requires the exact scoped reference to match the catalog selection.
- Each provider refusal is logged once as
  "[ORCHESTRATION_EXTERNAL_SOURCES] A step's source was refused." with
  sc_stage, sc_authority_reason and sc_reason (the check that failed), using
  hashed identifiers only.
- The shared test fake returns point reads as CosmosDictLike, so checks that
  require exactly dict on stored records fail in tests as they did in
  production.
- Version 0.261.291. Adds the fix doc and updates the admin troubleshooting
  table, the external source access feature doc and the logging tags reference.

Fixes #1699

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brings in #1700 (V2 sidebar links, 0.261.290). The only conflict was VERSION;
this branch keeps 0.261.291, the next free patch number.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds the v0.261.291 entry for orchestrated Ask an agent steps (#1699) and the
missing v0.261.289 entry for picked agents and the agents setting.

Refs #1699

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit be2a353 into paullizer-react-v2-ui Oct 7, 2026
11 checks passed
Paul Lizer (paullizer) added a commit that referenced this pull request Oct 7, 2026
…lizer-control-center-v2-fixes

The base took 0.261.291 for the orchestrated Ask an agent fix (#1701), so
this change moves to 0.261.292. The release notes keep every section.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 7, 2026
PR microsoft#1701 merged into paullizer-react-v2-ui with 0.261.291, so this fix
takes the next free patch number.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 7, 2026
Brings in microsoft#1700 (V2 sidebar links, 0.261.290) and microsoft#1701 (Ask an agent
steps, 0.261.291). VERSION keeps 0.261.292, the next free patch number.
The admin troubleshooting table keeps the base's 0.261.289 and 0.261.291
rows, followed by this fix's chart rows.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 7, 2026
Brings in V2 at 36f0af1 (microsoft#1701, VERSION 0.261.291, and microsoft#1702,
VERSION 0.261.292).

Conflicts:
- application/single_app/config.py: V2's file byte-for-byte (VERSION
  0.261.292). This branch changes only that line.
- docs/explanation/release_notes.md: V2's file byte-for-byte, with this
  branch's own (v0.261.291) section inserted at the very top, above V2's
  first section, its (v0.261.292). Every V2 section is unchanged,
  including microsoft#1701's own (v0.261.291).

docs/admin/orchestration.md merged cleanly: microsoft#1701's notes and
troubleshooting rows and this branch's settings and hand-off pause rows
are separate hunks.

V2's other changes, to the Ask an agent step's access check, the V2
Control Center and their docs and tests, don't touch this branch's
files.

The next commit renumbers this branch's section and VERSION to
0.261.293, above V2's 0.261.292.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 7, 2026
V2 now uses 0.261.291 (microsoft#1701) and 0.261.292 (microsoft#1702), with their own
release notes sections, so this branch's version moves above them.
Renumbers the same 20 lines in 15 files as the previous bump: config.py,
this branch's release-notes section, the documentation version notes and
the new tests' headers. V2's own (v0.261.292) and (v0.261.291)
release-notes sections and microsoft#1701's 0.261.291 notes in orchestration.md
are untouched. The tests' version floor stays 0.261.253.

Refs microsoft#1549

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant