feat: integrate agent forensics into Azure benchmark reporting

[Sai Koumudi Kaluvakolanu (saikoumudi)]

Summary

• Replace the bespoke copilot -p "analyze msbench run..." report generation and benchmark-repository clone with msbench-agent-forensics 0.4.0.
• Extract every run for a queued date into one shared data root, then run one exact azure.skill analysis across all models/runs with formatter mode both.
• Run root-cause classification from the generated snapshot using the external Copilot CLI and HMAC provider authentication.
• Preserve the daily schedule, Azure service connection, ToBeProcessed queue, per-date run_ids.json, Azure DevOps artifact publishing, and existing per-task eval report blob contract.

Design and operational contract

• Python remains 3.12 (the package requires Python 3.11+).
• PipAuthenticate@1 continues to authenticate internal/MicrosoftSweBench; the script installs msbench-cli and pins msbench-agent-forensics==0.4.0 from the configured feed.
• Each run uses agent-forensics extract --run_id ... --run_date ... --data-dir <shared-root>.
• Each date uses one combined analyze --benchmark azure.skill --exact_match --formatter-mode both --provider hmac, followed by classify --use-copilot --provider hmac --source snapshot-only.
• No Copilot SDK invocation is used by the production command path. The existing external copilot binary installation remains the provider execution path.
• A date enters the processed set only after extraction, analysis, classification, compatibility enrichment, and all date uploads succeed. Failed dates remain queued; successful dates can still be removed before the task reports the failed-date summary.

Authentication setup

The azure_msbench_eval service connection must retain get access to these existing secrets in kv-msbench-eval-azuremcp:

• azure-mcp-eval-capi-hmac: CAPI HMAC signing key
• azure-mcp-eval-capi-id: matching Copilot integration ID

The script exports them only as COPILOT_HMAC_KEY and COPILOT_HMAC_INTEGRATION_ID, then removes those environment variables during cleanup. Agent Forensics generates COPILOT_PROVIDER_HEADERS per Copilot subprocess. The report-generation GitHub PAT path is removed.

Blob and artifact compatibility

New Agent Forensics artifacts use a non-colliding date prefix:

• <date>/agent-forensics/reports/**
• <date>/agent-forensics/snapshots/**

The Azure DevOps artifact also retains ToBeProcessed, run_ids_<date>.json, date-scoped Agent Forensics reports/snapshots, and enriched compatibility JSON. Existing dashboard consumers continue to receive:

• <date>/<instance>/<model>_<instance>_eval_report.json

Only source tasks that emit eval_report.json are uploaded, matching the current behavior.

Validation

• PowerShell AST parse of Invoke-GenerateBenchmarkReport.ps1
• YAML parse of azure-benchmark-report.yml
• git diff --check
• Contract assertions for required Agent Forensics/HMAC flags and removal of legacy PAT/generic prompt paths
• Compatibility enrichment exercised against the extracted 2026-08-17 Azure fixture: 50 task directories, 16 available eval reports, 16 enriched compatibility files generated

Rollout notes and residual risk

1. Confirm msbench-agent-forensics 0.4.0 is published in internal/MicrosoftSweBench before enabling the scheduled run; the pipeline fails clearly if the required CLI contract is unavailable.
2. Confirm the service connection can read both HMAC secrets and access msbenchnightlydata/msbench-reports.
3. Manually queue the report pipeline once and verify the new date-level report/snapshot prefix plus legacy eval-report ingestion before relying on the daily schedule.

A live end-to-end run was not performed locally because it requires the Azure DevOps feed identity, Key Vault secrets, storage RBAC, completed MSBench runs, and production Copilot HMAC routing.