Add workspace Key Vault secrets retrieval API for workspace services and user resources

[Copilot]

Resolves #2402

What is being addressed

Researchers need to access secrets that resources place in the workspace Key Vault (e.g. VM credentials, storage account keys, database connection strings). Previously there was no way to retrieve these values through the API — resources could only surface an opaque Key Vault reference.

How is this addressed

Adopts the convention that any resource property whose name contains keyvault_secret_id holds a Key Vault secret identifier (the full secret URI). New endpoints resolve that reference to the underlying value on demand.

• API endpoints (api_app/api/routes/workspaces.py):
  • GET /workspaces/{workspace_id}/workspace-services/{service_id}/secrets/{secret_name}
  • GET /workspaces/{workspace_id}/workspace-services/{service_id}/user-resources/{resource_id}/secrets/{secret_name}
  • secret_name is the property key. Requests are rejected with 400 if the property is not a secret reference, and 404 if the property or secret is absent. The user-resource endpoint keeps the existing owner/researcher ownership check.
• Secret retrieval (services/secrets.py): async SecretClient fetch, parsing vault + name from the stored identifier. Retrieval is constrained to the workspace's own Key Vault (keyvault_uri) as defence-in-depth against cross-workspace access; failures surface as a typed SecretRetrievalError.
• Base workspace template (templates/workspaces/base): grants the API managed identity Key Vault Secrets User on the workspace Key Vault and outputs keyvault_uri as a property.
• Adds azure-keyvault-secrets dependency and SecretInResponse schema.

Only the Key Vault reference is stored in resource properties; the secret value is fetched on demand and is not persisted or masked in standard resource responses.

The UI "reveal secret" affordance described in the issue is out of scope here (API and below only).

• Documentation: added the keyvault_secret_id convention to docs/tre-workspace-authors/authoring-workspace-templates.md.
• CHANGELOG.md updated.
• Versions incremented: API 0.26.0 → 0.27.0, tre-workspace-base 2.10.0 → 2.11.0.

• Fixes As a researcher I need to be able to access Azure blob storage (and other secrets) from within the workspace. #2401

[Jonny Rylands (jonnyry)]

Should the general direction be moving away from secrets - e.g. AAD authentication for databases and storage accounts?
Are the VM credentials necessary?

[Marcus Robinson (marrobi)]

Should the general direction be moving away from secrets - e.g. AAD authentication for databases and storage accounts? Are the VM credentials necessary?

Great question. Agree should be moving to AAD/Entra. I was trying something, and didn't mean to open a PR. I was looking if we can have a single central Guacamole service, they key blocker is a shared service needs to retrieve the workspace VM passwords/secrets. I'm not sure is a way around this with Linux VMs? Also if a machine locks, no way to unlock without the password.

Also we have scenarios where AI model auth is via API key. But completely agree Entra auth should be the default.

Welcome thoughts.

[Jonny Rylands (jonnyry)]

Should the general direction be moving away from secrets - e.g. AAD authentication for databases and storage accounts? Are the VM credentials necessary?

Great question. Agree should be moving to AAD/Entra. I was trying something, and didn't mean to open a PR. I was looking if we can have a single central Guacamole service, they key blocker is a shared service needs to retrieve the workspace VM passwords/secrets. I'm not sure is a way around this with Linux VMs? Also if a machine locks, no way to unlock without the password.

Also we have scenarios where AI model auth is via API key. But completely agree Entra auth should be the default.

Welcome thoughts.

Ah yes of course - the VM passwords. I was looking at the AAD auth VM extensions but it doesn't look easy to get working with Guacamole - plus I couldn't see one that would work for Linux RDP.