Azure Files Search action with live file permission checks, and File Sync Azure Files fix

[Paul Lizer (paullizer)]

Summary

Customers already index file shares into Azure AI Search with the Azure Files indexer and don't want to ingest them into SimpleChat again. Those indexes store no permissions, so this action enforces them live.

• Azure Files Search action (global only). Admins point a global action at an existing Azure Files indexer index (document-per-file or chunked, with keyword, semantic, or hybrid queries). For each candidate file, SimpleChat reads the NTFS DACL through Azure Files OAuth with backup intent. It also checks share access: the default share permission plus the user's role assignments, read from ARM. Both are evaluated against the signed-in user's Entra SIDs and transitive group SIDs from Graph. Only allowed files reach the model, and anything that can't be verified fails closed.
• Withheld results are visible to admins only. Users are never told results were withheld. Each search that withholds files writes an azure_files_search_access activity-log entry: agent, conversation, counts, reason codes, and up to 25 paths, never content or the question. Unverified results notify admins at most once per action per day. Both are shown in the V1 Control Center.
• Global-only enforcement. User and group type lists exclude the type. Personal and group saves return 403, in the classic and V2 editors and for type aliases the loader would match. The runtime executes only manifests bound to a global origin. Configuration lives in the V1 admin modal and the V2 global editor, each with an admin connection test.
• File Sync Azure Files fix.
  • Managed identity and service principal sources failed every connection test and run because the SDK requires token_intent="backup". That's fixed, with classified errors, run-failure notifications, and Storage File Data Privileged Reader guidance in the V1 and V2 editors.
  • These sources now send a storage token, so the file service URL must be an Azure Files host.
  • The deployers gain optional role grants on existing storage accounts and search services (Bicep/azd, CLI, Terraform; deployer 1.0.34).

Notes for reviewers:

• The security-critical code is functions_azure_files_acl.py, which parses SDDL and evaluates the DACL in order for FILE_READ_DATA. Every ambiguous case resolves to unverified: an unknown SID, conditional or object ACEs, generic-only rights, a missing DACL, or a parse error. BUILTIN\Users counts as a member only when the admin opts in.

• The app identity's Storage File Data Privileged Reader role bypasses NTFS, so SimpleChat's evaluation is the only enforcement.

• CodeQL flagged a partial SSRF (critical) on the first push. A File Sync Azure Files source accepted any HTTPS host, and once the token path worked, it would have sent the app's storage token there. It's fixed in 2ac3afa:

  • _normalize_azure_file_url accepts only <account>.file.<Azure storage suffix> through validate_azure_file_endpoint, the same validate-and-rebuild pattern the Blob connector uses.
  • _get_azure_files_service_client re-checks stored sources before creating a credential.
  • The search action already rebuilt each file endpoint from a validated host and an allowlisted account.

  CodeQL passes on that commit, with no open alerts on the PR.

• Background runs without a user session return nothing (identity_unavailable).

• The Search API is pinned to 2024-07-01, so index aliases can't be addressed. SimpleChat's own indexes are refused.

• A V1 regression introduced during this work was fixed before merge: populateSummary read an undeclared variable, which broke the summary step for most action types. The new static-harness UI test caught it. It never shipped, so it has no release note.

• Test harness fixes:

  • group_file_source_harness.py and public_file_source_harness.py now stub the two azure-core exceptions and the group helpers that File Sync imports.
  • test_workspace_authoring_backend.py stubs validate_mcp_tool_pinning_for_save. It was missing on the base, so 80 of the file's tests returned 503 there. It also treats azure_files_index as global only.
  • test_file_sync_azure_blob_storage.py supplies the new category messages.

• The version is 0.261.294 because Fix orchestrated action charts that were drawn but never shown #1703 took 0.261.293.

Linked issue

Refs #1697. It stays open until the in-app checks listed under "Not run" are done.

Release Notes & Latest Features

• New Feature
• Bug Fix
• UI Enhancement
• Breaking Change
• Internal only

Is this visible to end users?

• Yes
• No

Is this admin-facing (Admin Settings, governance, deployment, config)?

• Yes
• No

Should this become a Latest Feature card?

• Yes
• No
• Already added

Screenshot needed for the card?

• Yes
• No
• Attached

Version bump

• application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only (0.261.294)
• deployers/version.txt bumped, or not needed because deployers/ was not changed (1.0.34)

Testing / validation

Python runs use the repo .venv, on Windows. Results are on 42af198 unless marked with the final commit, 2ac3afa, which adds only the File Sync host check, two CodeQL cleanups, and docs.

• python -m pytest functional_tests/test_azure_files_acl_evaluator.py -q: 14 passed
• python -m pytest functional_tests/test_azure_files_search_pipeline.py -q: 21 passed
• python -m pytest functional_tests/test_azure_files_search_integration.py -q: 14 passed
• python -m pytest functional_tests/test_file_sync_azure_files_token_intent.py -q: 12 passed (2ac3afa)
  • Behavioral tests against the pinned azure-storage-file-share SDK, which rejects token credentials without an intent.
  • Saving refuses non-Azure Files hosts, and a stored source with another host is refused before any credential is created. With the host check reverted, both new tests fail.
• On 2ac3afa, 178 passed across these files:
  • test_file_sync_azure_files_token_intent.py, test_file_sync_azure_files_identity.py, test_deployers_external_role_grants.py, and test_action_app_identity_endpoint_hardening.py
  • the group and public file-source API and sync-field tests, plus test_file_source_credential_round_trip_fix.py
  • ui_tests/test_admin_azure_files_index_action_modal_static.py
• python -m pytest functional_tests/test_file_sync_azure_files_identity.py -q: 5 passed
• python -m pytest functional_tests/test_deployers_external_role_grants.py -q: 6 passed. This includes a PowerShell run of the CLI resource-ID validator with the empty defaults.
• python -m pytest functional_tests/test_action_test_connection_modal_wiring.py -q: 5 passed
• python -m pytest functional_tests/test_control_center_activity_logs_hardening.py -q: 4 passed
• python -m pytest on the six group and public file-source harness files: 191 passed
• python -m pytest functional_tests/test_file_sync_azure_blob_storage.py -q: 20 passed, 1 failed. test_file_sync_routes_do_not_disclose_exception_details fails identically on the base.
• python -m pytest functional_tests/test_workspace_authoring_backend.py -q: 141 passed, 1 failed. test_changed_assigned_knowledge_uses_real_personal_scope_policy fails identically on the base.
• python -m pytest ui_tests/test_admin_azure_files_index_action_modal_static.py -q: 3 passed. It runs the real V1 modal and stepper in Chromium on the static harness, covering create, edit, and connection-test pass, warn, and fail states.
• npm --prefix application/v2_ui run build, then python -m pytest ui_tests/test_v2_admin_azure_files_index_action.py -q: 1 passed
• ui_tests/test_admin_azure_files_index_action_modal.py: skipped, because it needs SIMPLECHAT_UI_BASE_URL and a signed-in storage state
• node functional_tests/test_v2_azure_files_index_action_logic.mjs: 4 checks passed
• npm run typecheck in application/v2_ui: passed
• Route policy tests: test_route_blueprint_policy_inventory.py 12/12, test_route_unauthenticated_policy_contract.py 7/7, test_route_policy_test_coverage.py 3/3
• test_docs_app_surface_coverage.py 7/7 and test_docs_site_quality.py 6/6 (2ac3afa). scripts/build_docs_inventory.py output is committed.
• python scripts/check_xss_sinks.py --base-sha <merge-base> --head-sha HEAD <changed files>: passed for 48 files
• python scripts/check_broken_access_control.py --base-sha <merge-base> <changed .py files>: passed for 30 files (2ac3afa)
• az bicep build --file deployers/bicep/main.bicep: succeeded, and main.json is regenerated
• Related-test sweep: 423 functional test files were run on this branch and on the base, in parallel chunks. These are the files that use the touched modules or the shared test harnesses. Every failure that remains also fails on the base.
  • Docs: test_docs_link_integrity.py (3), with the same broken counts as the base, and test_docs_release_notes_integrity.py (1, the generated release-notes pages are stale).
  • test_personal_action_save_helper_regression.py (1) and test_admin_key_vault_reminders_ui.py (1).
  • A test_plugin_logging.py collection error.
  • Others that only fail inside a shared pytest process.
• Live Azure check without an app deployment. I ran the real functions_azure_files_search_runtime and functions_file_sync modules against a test environment:
  • The share held eight files whose ACLs were set through SDDL: user allow, group allow, other group only, explicit deny, unknown AD SID, Everyone, and BUILTIN\Users.
  • The Basic AI Search service had a document-per-file index and a chunked integrated-vectorization index.
  • Results:
    • Keyword, semantic, and hybrid searches returned only the files the user can open.
    • Denied and unverified files were logged with the expected reason codes and never appeared in the model response.
    • With the BUILTIN\Users opt-in, that file was returned.
    • Unbound and personal manifests were refused.
    • Both connection checks passed, and File Sync listed the share over OAuth.
  • One cold first run, using Azure CLI credentials, exceeded the 25-second permission budget and failed closed as designed. Warm runs passed.
• Live Semantic Kernel check (2ac3afa):
  • I loaded the plugin the way semantic_kernel_loader does, with PluginHealthChecker.create_plugin_safely and KernelPlugin.from_object, from a manifest bound to the global origin.
  • The health check passed. The tool schema is query (required string) and top_n (optional integer).
  • Called through kernel.invoke inside a Flask request context with a signed-in session, against the same environment, it returned only the four files the user can open.
  • The access review recorded the agent and conversation without copying agent configuration.
  • A signed-out call returned nothing.
• Not run:
  • No deployment yet, so these are untested against a live app: V1 and V2 configuration, model-driven agent chat, the Control Center views of the entries, and the notifications.
  • The Graph lookup of on-premises SIDs with the app's delegated scopes is unverified. It fails closed if those scopes aren't enough.

Documentation

• Release notes updated, or not needed (v0.261.294 in docs/explanation/release_notes.md)
• Feature documentation updated, or not needed:
  • docs/explanation/features/AZURE_FILES_SEARCH_ACTION.md
  • docs/reference/actions/azure-files-index.md
  • The Azure Files access section in docs/admin/knowledge.md
  • The deployer READMEs
• Fix documentation updated, or not needed (docs/explanation/fixes/AZURE_FILES_FILE_SYNC_MANAGED_IDENTITY_FIX.md, including the host check, with the role guidance in AZURE_FILES_FILE_SYNC.md corrected)

Security checklist

• New Flask routes include @swagger_route(security=get_auth_security()). The one new route is POST /api/plugins/test-azure-files-index-connection, which also has @login_required and @admin_required and accepts global scope only.
• Settings sent to non-admin frontends use sanitize_settings_for_user(). No settings are sent; the connection test returns only reviewed check messages.
• Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
• No secrets, keys, connection strings, or local-only artifacts are included. Test fixtures use placeholder IDs and a fake base64 key.