Port durable batched search metadata sync to React v2

[Paul Lizer (paullizer)]

Summary

Ports the durable, batched document search metadata sync from #1658 (merged into Development as 900a3e431, including hardening commit 9c156a429) and its follow-up #1673 (df3879c07) to the React V2 branch. It is re-implemented against V2's update_document() (strict mode, etag and operation guards, content screening), the group projection fence, and the native group and public document APIs; it is not a cherry-pick.

• Azure load (headline): a metadata or tag change no longer re-reads every chunk with its embedding and writes it back one chunk at a time. A durable background sync merges only the changed fields in key-paged batches of 500, so a 10,000-chunk document goes from about 20,000 AI Search calls (plus about 40,000 Cosmos write-fence operations) to about 40 Search calls.
• What users will notice:
  • Metadata and tag saves on large documents return immediately instead of timing out with "invalid JSON". Bulk tag, tag rename, and tag delete no longer sync every chunk twice.
  • A short info notice says search and chat results will reflect the change once the index finishes updating; failed updates retry automatically with backoff.
• V2 bug fixed along the way: native group metadata edits on any document with more than one chunk failed with document_propagation_incomplete. Strict mode compared the document etag before every chunk, and the projection fence's claim changes that etag.
• Sharing: classic-route group sharing changes now reach Search. Revocations are projected through the fence before the save and fail closed; grants are projected after it. Approving an already-approved share repairs its search access, on the legacy group, native group, and personal approve paths.

V2-specific adaptations (beyond #1658)

• Strict mode:
  • The sync request is written before the save, and its token rides in the same strict _upsert_document_and_sync_access_index(..., strict=True) save.
  • A strict caller re-requests every mirrored field it sends, so a retry always re-projects.
  • Etag, operation-guard, screening, and origin-field semantics are unchanged. The worker never writes the document.
• Held screened documents: these report search_sync: not_required, and the worker treats them as nothing to sync, because publication rebuilds their chunks.
• Native APIs:
  • Receipts keep status: 'updated', updated_fields, document_id, and the scope id, plus success/errors/documents_updated/vocabulary_retained. They add search_sync.
  • Metadata projection failures no longer produce document_propagation_incomplete, because they retry durably. Access-index failures, and group grants that don't reach Search, still do.
• Group ACL with the projection fence:
  • Revocations project the reduced list before the save. The fence claim changes the etag, so the document is re-read and compared without system fields or the fence claim. If anything else changed, Search is restored to the Cosmos list and the request answers 409.
  • Grants project after the save and fail with DocumentMutationPropagationError: native APIs report document_propagation_incomplete, and legacy routes answer 503 with Retry-After when frozen, otherwise 500.
• Blob tag metadata:
  • Strict inline propagation moved to the worker for all scopes. It is best effort and conditioned on the blob ETag, and it never touches a screened release's pinned blob.
  • Justification: nothing in V2 reads blob document_tags. An inline failure could only fail a request after the document had already saved.
• Search cache (matches Development Clear cached search results after a background metadata sync #1673, df3879c07): after a pass that changed chunks, the worker clears cached search results for the document's workspace and for every user or group with an approved share. Without that, a search during the sync window could keep serving pre-sync values for the cache TTL (5 minutes by default).
• React V2 UI:
  • editMetadata now returns { searchSyncPending }, and only an updated 200 confirms an edit.
  • Batch and tag outcomes carry searchSyncPending. The explorer and tag manager show an info toast with SEARCH_SYNC_PENDING_NOTICE.
  • The classic personal, group, and public dialogs show the same notice.

Follow-ups (not in this PR)

• The native _search_acl (collaboration) and set_document_chunk_visibility() (publication) still write a document's chunks as a single batch, which AI Search caps at 1,000 documents. Moving them onto project_fields_to_document_chunks() removes that limit.
• Chunk access lists are not backfilled. Group shares approved through the classic routes before this change become searchable when sharing changes again, or on re-approval.
• Nothing reconciles projection-fence claims left uncertain after ambiguous Search errors. This predates the PR.
• Chunks saved by an in-flight upload carry the metadata read when they were saved. This predates the PR.

Linked issue

Refs #1657. Port of #1658 (merged as 900a3e431) and #1673 (df3879c07, search cache refresh after a sync).

Release Notes & Latest Features

• New Feature
• Bug Fix
• UI Enhancement
• Breaking Change
• Internal only

Is this visible to end users?

• Yes
• No

Is this admin-facing (Admin Settings, governance, deployment, config)?

• Yes
• No

Should this become a Latest Feature card?

• Yes
• No
• Already added

Screenshot needed for the card?

• Yes
• No
• Attached

Version bump

• application/single_app/config.py VERSION third segment bumped: 0.261.267 → 0.261.268. Rebased on the latest V2 (10fbad92e, Add collapsible category rail to V2 Admin Settings #1671) and renumbered.
• deployers/version.txt bumped, or not needed because deployers/ was not changed

Testing / validation

Python runs use a Python 3.12 venv with Flask 3.1.3 and Werkzeug 3.1.6 (this machine's global Flask 2.2.5 does not work with its Werkzeug). Each pytest file runs in its own process, because combined runs cross-talk through module stubs. "Base" means the same command on V2 without this change.

Final tree (10fbad92e + this PR)

• python -m pytest functional_tests/test_document_search_metadata_sync.py: 35 passed (new suite, including Clear cached search results after a background metadata sync #1673's owner/approved-recipient cache test)
• python -m pytest functional_tests/test_group_document_management.py -k "receipt_preserves or search_failure_after or access_index_failure or blob_tag_refresh or screened_tag_edit or storage_cas_rejection or tag_finalization_conflict or successful_operations_use_the_path": 22 passed
• python -m pytest "functional_tests/test_content_screening_upload_only_intake.py::test_model_generated_metadata_applies_without_a_hold": passed
• python -m pytest functional_tests/test_image_description_indexing_without_ocr_text.py: 9 passed (the offline real-import probe, which covers the new utils_cache import)
• Route policy tests, all passing:
  • python functional_tests/route_tests/test_route_blueprint_policy_inventory.py: 12/12
  • test_route_unauthenticated_policy_contract.py: 7/7
  • test_route_policy_test_coverage.py: 2/2
• Static checks, all passing:
  • python scripts/check_swagger_routes.py <39 changed files>: 5 route files
  • python scripts/check_xss_sinks.py --base-sha 10fbad92e --head-sha HEAD <changed files>: 35 files
  • python scripts/check_broken_access_control.py --base-sha 10fbad92e --head-sha HEAD <changed files>: 27 files
  • git diff --check 10fbad92e HEAD: clean
• Docs tests: python functional_tests/test_docs_app_surface_coverage.py 7/7; python functional_tests/test_docs_site_quality.py 6/6
• application/v2_ui: npm run typecheck passed; npm run build passed, with the existing chunk-size warning.
• package.json has no test script; the V2 UI logic tests are the node .mjs suites. node functional_tests/test_v2_group_document_operations.mjs: 20 checks passed. It failed on base because of a stale queued metadata case, which this PR updates.

Full per-file suites

These ran on e66f8f475 + this change. The later V2 merges (#1669, #1670, #1671) touch only admin settings, settings derivation, and workflow code.

Suite	This PR	Base
test_group_document_management.py	239 passed	238 passed
test_group_document_fixture_parity.py	103 passed	103 passed
test_public_document_management.py	206 passed	205 passed
test_public_document_fixture_parity.py	63 passed	63 passed
test_public_context_fixture_parity.py	252 passed	252 passed
test_public_document_read_apis.py	131 passed	131 passed
test_group_document_read_apis.py	208 passed	1 failed, 207 passed
test_group_document_collaboration.py	321 passed	321 passed
test_group_document_publication.py	127 passed	127 passed
test_public_document_publication.py	59 passed	59 passed
test_group_document_projection_fence.py	31 passed	31 passed
test_data_management_search_write_fence_authorization.py	6 passed	4 passed
test_group_tag_definition_writers.py	38 passed	38 passed
test_public_workspace_tag_writer_safety.py	33 passed	33 passed
test_content_screening_upload_only_intake.py	127 passed	127 passed
test_content_screening_jobs.py	112 passed	112 passed
test_content_screening_upload_routes.py	3 passed, 16 subtests	same
test_document_provenance_versions_and_guards.py	161 passed	161 passed
test_xsd_ingestion_generation_integration.py	25 passed	25 passed
test_group_document_blob_upload_etag.py	3 passed	3 passed
test_public_document_call_boundary.py	8 passed, 1 skipped	same

These failures predate this PR and are unchanged by it:

• test_document_auto_metadata_extraction_consistency.py: 1 failed, 3 passed
• test_content_screening_pipeline.py: 6 failed, 26 passed
• test_workspace_upload_conversation_fix.py: 4 failed
• each *_swagger_integration.py suite: 3 failed, 3 passed
• test_multiselect_metadata_extraction.py: 1 failed (an exact-version check)
• test_logging_tag_standardization.py: nonstandard Workflow* tags in files this PR does not touch. The new [DOCUMENT_METADATA], [DOCUMENT_SEARCH_SYNC], and [DOCUMENT_SHARING] tags are used and documented.

Regression sweep

The 108 other functional test files that reference the changed routes, JS, templates, or background_tasks.py were run per file on c968d5875 with and without this change: 0 new failures. 54 files fail identically on both, for environment-dependent or pre-existing reasons.

.mjs suites

Of the 75 functional_tests/*.mjs files, 73 pass. The 2 failures are unrelated and pre-existing:

• test_v2_admin_nested_setting_reads.mjs is a companion script that needs its Python driver.
• test_v2_workspace_action_authoring_logic.mjs finds no V2 action-registry entry for m365_calendar_plugin.py.

Not run

• ui_tests/test_workspace_metadata_search_sync_toast.py (new, env-gated Playwright): needs SIMPLECHAT_UI_BASE_URL and a storage state.
• The browser suites that use the updated UI fixtures.

Documentation

• Release notes updated, or not needed: ### **(v0.261.268)** in docs/explanation/release_notes.md
• Feature documentation updated, or not needed: not needed
• Fix documentation updated, or not needed: docs/explanation/fixes/DOCUMENT_SEARCH_METADATA_SYNC_FIX.md
• Also updated: docs/reference/logging-tags.md ([DOCUMENT_METADATA], [DOCUMENT_SEARCH_SYNC], [DOCUMENT_SHARING]) and docs/guides/use-tags-in-chat.md (troubleshooting row)

Security checklist

• New Flask routes include @swagger_route(security=get_auth_security()): no new routes; check_swagger_routes.py passes on the changed route files
• Settings sent to non-admin frontends use sanitize_settings_for_user(): no settings are sent
• Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
• No secrets, keys, connection strings, or local-only artifacts are included
• Route error responses now carry fixed messages instead of exception text.