From 17101d86282f85b31b45ac024eb780e1c23d0ae2 Mon Sep 17 00:00:00 2001 From: Jurie Smit Date: Sun, 23 Aug 2026 12:08:31 +0200 Subject: [PATCH 1/6] docs: define PII and credential authority boundaries --- ...acy-and-credential-authority-boundaries.md | 119 ++++++++ docs/prds/00-README.md | 4 +- .../pii-tokenization-and-credential-sync.md | 218 ++++++++++++++ ...ential-rotation-and-token-leak-response.md | 198 +++++++++++++ .../pii-tokenization-and-credential-sync.md | 266 ++++++++++++++++++ 5 files changed, 804 insertions(+), 1 deletion(-) create mode 100644 docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md create mode 100644 docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md create mode 100644 docs/runbooks/credential-rotation-and-token-leak-response.md create mode 100644 docs/spec/pii-tokenization-and-credential-sync.md diff --git a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md new file mode 100644 index 00000000..d8a88e0f --- /dev/null +++ b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md @@ -0,0 +1,119 @@ +# ADR-005: Privacy Processing and Credential Authority Boundaries + +**Status:** Accepted + +**Date:** 2026-08-23 + +**Decision owners:** Foundation Security and Architecture + +**Baton:** `deacf6eb` + +## Context + +`IPIITokenizationPort` exists in Foundation Security, while `GDPRComplianceAdapter` lives in Business Applications and handles consent, legal basis, retention, and data-subject rights. Current LLM clients accept prompt, chat, and embedding content without a repository-wide tokenization boundary. The architecture must decide whether PII protection belongs inside the GDPR adapter, at persistence, at LLM ingress, or at more than one controlled boundary. + +`ICredentialSyncPort` is provider-neutral and names Azure Key Vault, HashiCorp Vault, and AWS Secrets Manager. The repository also has a simulated in-memory `SecretsManagementEngine`, but it implements `ISecretsManagementPort` and does not provide external credential synchronization, dependent validation, or authoritative rotation. The architecture must decide whether Azure Key Vault is the only possible authority or the production default within a provider-neutral design. + +## Decision + +### 1. Tokenization is a Foundation privacy boundary + +The planned `PIITokenizationEngine` implements the existing `IPIITokenizationPort`. Authenticated composition hosts must invoke it before external LLM, embedding, reranking, or content-bearing tool calls and before writing raw content to general-purpose memory, vector, analytics, telemetry, log, or evidence stores. + +The boundary does not require all lawful systems of record to discard originals. Raw PII may remain only in an explicitly authorized source of record or encrypted token-map store with documented purpose, retention, and access policy. External/provider and general-purpose paths receive scoped tokens. + +### 2. GDPR remains the policy and data-rights orchestrator + +`GDPRComplianceAdapter` does not absorb token detection, token creation, or token-map persistence. It remains responsible for consent, lawful basis, retention, access, rectification, portability, and erasure. It may call `IPIITokenizationPort` to enumerate or delete a verified subject's mappings as one step in a broader data-rights workflow. + +This preserves dependency direction: Business Applications can consume Foundation ports, while Foundation does not depend on Business Applications. + +### 3. Tokens are scoped pseudonyms, not anonymous data + +Production tokens are opaque and scoped to tenant, internal subject identifier, processing purpose, policy, and key version. Cross-tenant or cross-purpose deterministic token reuse is forbidden. Token mappings and detokenization are protected data processing and remain subject to privacy policy. + +### 4. Credential authority is single-writer per namespace/environment + +Each credential namespace and environment has exactly one authoritative vault selected by reviewed configuration. Azure Key Vault is the production default for Cognitive Mesh. The existing provider-neutral contract remains valid so an explicitly approved namespace may use another provider adapter. + +This is not unrestricted multi-vault synchronization. Callers cannot select `VaultProvider` or `VaultUrl` for production operations, and two providers cannot write the same credential identity. Authority migration is a separate, explicit cutover with reconciliation and rollback. + +### 5. Credential synchronization is distinct from generic secret CRUD + +The planned `CredentialSyncEngine` implements `ICredentialSyncPort` and owns authority resolution, authorization, metadata reconciliation, rotation state, dependent validation, and revocation. Provider adapters own vault-specific API calls and native error/version mapping. + +`SecretsManagementEngine` remains a simulated implementation of `ISecretsManagementPort`; it is neither a production vault nor evidence that credential sync exists. A future production secrets adapter may share lower-level provider infrastructure, but the two public capability contracts are not merged or substituted. + +### 6. Values are used just in time + +Production consumers obtain credential values only through an authenticated, authorized, bounded use path. Persistent application state retains references and metadata, never plaintext. Managed/workload identity is preferred over reusable secrets where supported. + +## Alternatives Considered + +### Put tokenization only in `GDPRComplianceAdapter` + +Rejected. LLM, embedding, memory, telemetry, and tool paths can bypass that Business-layer adapter. It would also invert responsibility by placing a reusable Foundation security mechanism inside a business compliance component. + +### Tokenize only immediately before persistence + +Rejected. Raw PII could still leave the trust boundary through an external model or tool call before persistence occurs. + +### Tokenize only at each provider adapter + +Rejected. Repeating policy at every adapter creates inconsistent coverage and makes new providers unsafe by default. Provider adapters still enforce that they receive a privacy-qualified payload, but the composition boundary owns orchestration. + +### Make Azure Key Vault the globally hard-coded sole provider + +Rejected. Azure Key Vault is the production default, but hard-coding it contradicts the existing provider-neutral port and prevents bounded migrations or sovereign deployments. Single authority per namespace prevents split brain without sacrificing adapter portability. + +### Permit multi-vault fan-in or last-write-wins sync + +Rejected. It creates ambiguous provenance, rollback, revocation, and conflict semantics for sensitive credentials. + +### Reuse `SecretsManagementEngine` as credential sync + +Rejected. It is an in-memory demo implementation with a different contract and no external authority, dependent registry, or production rotation proof. + +## Consequences + +### Positive + +- New provider integrations are safe only when routed through one privacy boundary. +- Data-rights policy and token mechanics remain decoupled in the correct layer direction. +- Azure Key Vault can be the production default without turning provider strings into caller-controlled routing. +- Credential provenance and rotation have one authoritative writer. +- Existing public capability ports remain stable. + +### Costs and constraints + +- Composition hosts must carry tenant, subject, purpose, policy, and authenticated caller context. +- Existing port models require additive versioned contracts before production enforcement. +- Token-map and vault-provider adapters need independent threat modeling, authorization, retention, and recovery design. +- Model outputs can still infer or introduce PII and require post-response detection policy where applicable. +- Provider-specific rotations may require different overlap or atomic-cutover procedures. + +## Guardrails + +- No silent raw-content fallback when tokenization is required. +- No raw PII, restored values, tokens with mapping value, or credentials in logs, telemetry, evidence, chat, Git, or Baton. +- No raw email/name/national identifier as `DataSubjectId`. +- No caller-selected production vault URL/provider. +- No plaintext durable credential cache. +- No detokenization or credential read without authenticated identity and explicit scope. +- No claim of production enforcement based on these documents. + +## Capability Status + +- **Implemented:** port contracts, GDPR workflows, direct LLM clients, and demo in-memory secret management. +- **Experimental:** none accepted. +- **Planned:** the engines, adapters, protected stores, composition boundary, authority registry, and rehearsal. +- **Blocked:** enforcement and provider activation pending separately authorized implementation and security acceptance. + +## References + +- [`pii-tokenization-and-credential-sync.md`](../prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md) +- [`pii-tokenization-and-credential-sync.md`](../spec/pii-tokenization-and-credential-sync.md) +- [`credential-rotation-and-token-leak-response.md`](../runbooks/credential-rotation-and-token-leak-response.md) +- `src/FoundationLayer/Security/Ports/IPIITokenizationPort.cs` +- `src/FoundationLayer/Security/Ports/ICredentialSyncPort.cs` +- `src/BusinessApplications/Compliance/Adapters/GDPRComplianceAdapter.cs` diff --git a/docs/prds/00-README.md b/docs/prds/00-README.md index 4afaeb76..bab67352 100644 --- a/docs/prds/00-README.md +++ b/docs/prds/00-README.md @@ -17,7 +17,8 @@ docs/prds/ ├── 00-README.md ← This file ├── global-nfr.md ← Global Non-Functional Requirements Appendix ├── 01-foundational-infrastructure/ -│ └── deterministic-security-scanning-build-loop.md +│ ├── deterministic-security-scanning-build-loop.md +│ └── pii-tokenization-and-credential-sync.md ├── plugin-dashboard.md ← Plugin-Based Dashboard Architecture PRD ├── convener-backend.md ← Convener Backend Architecture PRD └── convener-widget.md ← Convener Widget/Plugin PRD @@ -30,6 +31,7 @@ docs/prds/ | **00-README.md** | Orientation & conventions for all PRDs. | | **global-nfr.md** | Platform-wide security, privacy, SLA, compliance, and quality gates. All other PRDs inherit these requirements. | | **01-foundational-infrastructure/deterministic-security-scanning-build-loop.md** | Exact-input security qualification for pull requests and protected releases. | +| **01-foundational-infrastructure/pii-tokenization-and-credential-sync.md** | Privacy-qualified external processing and authoritative credential lifecycle for agent tools. | | **plugin-dashboard.md** | Defines the core shell & extensible plugin/widget model. | | **convener-backend.md** | Specifies APIs/services that power Convener features (champion discovery, pulse, etc.). | | **convener-widget.md** | UI plugin that consumes Convener backend APIs & surfaces insights in the dashboard. | diff --git a/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md b/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md new file mode 100644 index 00000000..22b9b9a5 --- /dev/null +++ b/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md @@ -0,0 +1,218 @@ +# PRD: PII Tokenization and Credential Sync + +**Status:** Proposed + +**Owner:** Foundation Security + +**Baton:** `deacf6eb` + +**Scope:** Product and architecture documentation only + +## TL;DR + +Cognitive Mesh needs two related Foundation security capabilities: a mandatory privacy boundary that de-identifies sensitive text before external AI processing or general-purpose persistence, and an authoritative credential lifecycle that supplies agent tools with current credentials without copying secrets into configuration, logs, or durable caches. The existing `IPIITokenizationPort` and `ICredentialSyncPort` are the public capability contracts; this PRD does not introduce replacements. Production credential authority is configured per credential namespace and environment, with Azure Key Vault as the default provider, while other approved providers remain possible. No engine, adapter, vault, credential, deployment, or runtime behavior is created or changed by this document. + +## Problem + +The repository currently has: + +- an `IPIITokenizationPort` contract but no tokenization engine, detector adapter, or protected token-map store; +- an `ICredentialSyncPort` contract but no credential-sync engine or external-vault adapter; +- a `GDPRComplianceAdapter` that handles consent and data-subject rights but does not de-identify content in transit; +- direct LLM clients that can receive raw prompt and embedding content; +- a simulated in-memory `SecretsManagementEngine` implementing `ISecretsManagementPort`, which is not an external credential synchronization implementation; and +- no production proof that Azure Key Vault or another provider is currently authoritative for agent-tool credentials. + +Without an enforced boundary, raw PII can reach model providers, embeddings, memory, telemetry, or logs. Without a single credential authority per namespace, consumers can use stale values, rotations can diverge, and operators cannot prove which version was active. + +## Goals + +- De-identify policy-covered PII before external LLM, embedding, or tool-provider calls. +- Prevent raw PII from entering general-purpose memory, analytics, telemetry, logs, or evidence stores. +- Preserve explicitly authorized systems of record and a protected token-map store where lawful processing requires original data. +- Support narrowly authorized detokenization and data-subject deletion with auditable, metadata-only records. +- Make one configured vault authoritative for each credential namespace and environment. +- Default production namespaces to Azure Key Vault while retaining the existing provider-neutral port. +- Rotate, validate, activate, and revoke credentials without exposing values to logs, chat, Git, Baton, or durable application caches. +- Keep `IPIITokenizationPort` and `ICredentialSyncPort` as the sole public capability ports. + +## Non-Goals / Out of Scope + +- Implementing either engine, detector, token store, vault adapter, or composition-host integration. +- Creating or changing Azure Key Vault resources, secrets, access policies, RBAC, managed identities, or credentials. +- Claiming that tokenization eliminates all privacy risk or that tokens are anonymous data. +- Replacing consent, legal-basis, retention, access, rectification, portability, or erasure workflows in `GDPRComplianceAdapter`. +- Copying all authorized system-of-record PII into a token vault. +- Supporting caller-selected arbitrary vault URLs or multi-writer synchronization. +- Treating the simulated `SecretsManagementEngine` as a production credential vault. +- Deploying, enforcing, or rehearsing the runbook. + +## Users and Stories + +- As a privacy owner, I want external AI requests to contain scoped tokens instead of raw PII so provider processing is minimized. +- As a data-subject-rights operator, I want token mappings included in verified access and erasure workflows without exposing unrelated subjects. +- As an agent-tool operator, I want tools to consume the current credential version just in time so rotations do not require plaintext configuration changes. +- As a security responder, I want a two-phase rotation and immediate-revocation path with evidence that never records secret material. +- As an auditor, I want to prove the privacy policy, detector version, token scope, credential authority, and active version used for an operation. +- As a developer, I want local/test fakes that are visibly non-production and cannot qualify as enforcement evidence. + +## Functional Requirements + +### FR-1: Mandatory privacy boundary + +Policy-covered content must pass through `IPIITokenizationPort` before: + +- any external LLM completion, chat, embedding, or reranking call; +- any external tool/provider call whose payload can contain user content; and +- persistence to general-purpose memory, vector stores, analytics, telemetry, logs, or evidence stores. + +Raw content may exist only transiently in the authenticated request boundary, in an explicitly authorized system of record with a documented legal basis, or in the encrypted token-map store. Provider failure, unavailable policy, or failed required detection/tokenization must block the external call and non-authoritative persistence. There is no silent raw-data fallback. + +### FR-2: GDPR responsibility boundary + +`GDPRComplianceAdapter` remains responsible for consent, lawful-basis assessment, retention, and data-subject workflows. It may orchestrate `DeleteTokensForSubjectAsync` and token statistics through `IPIITokenizationPort`, but it does not implement detection, token generation, or token-map persistence. Erasure closeout must distinguish deletion from token mappings, authorized systems of record, caches, model/provider retention, backups, and audit records. + +### FR-3: Scoped tokenization + +Qualifying tokens must be opaque, non-semantic, and scoped by tenant, internal data-subject identifier, processing purpose, policy version, and tokenization key/version. The raw email, name, or other PII value must not be used as `DataSubjectId`. Cross-tenant or cross-purpose deterministic tokens are prohibited because they enable linkability. + +Detection must define deterministic overlap resolution, confidence behavior, detector/rule versions, and handling for unsupported content. `OriginalValue` is sensitive compatibility data: production results omit it by default and must never persist it in logs, telemetry, or general evidence. + +Policy must set content-size and entity-count limits. Oversized input is rejected or split by a deterministic, policy-versioned chunking algorithm that preserves span offsets and cannot separate an entity silently. Token-map writes are idempotent and atomic with the returned tokenized result: a reversible token is never released unless its encrypted mapping is durably committed. + +### FR-4: Detokenization authorization + +Detokenization is deny-by-default. A request must carry authenticated workload/user identity, tenant, subject, purpose, exact token scope, reason, and correlation identifier. Authorization must validate subject and purpose scope; bulk or cross-subject restoration requires separately approved elevated policy. Audit records contain token identifiers or digests and decision metadata, never restored values. + +### FR-5: Single credential authority + +Each credential namespace and environment must map to exactly one configured authoritative vault. Production defaults to Azure Key Vault. Another provider may be selected only through reviewed configuration and a compatible adapter; two providers cannot simultaneously act as writers for the same credential identity. + +The composition host resolves the authority from trusted configuration. Caller-provided `VaultProvider` and `VaultUrl` cannot redirect a production sync request. Unknown, unapproved, or mismatched authority fails closed. + +### FR-6: Just-in-time credential use + +Credential values must be retrieved only for an authenticated, authorized consumer and held for the shortest practical lifetime. Durable caches retain metadata, vault reference, version, status, and expiry—not plaintext. Values must not appear in exception messages, logs, telemetry, HTTP responses, screenshots, evidence, or Baton. + +Managed identity or workload identity is preferred when the provider supports it; it is recorded as identity metadata, not synchronized as a reusable secret. + +### FR-7: Rotation state machine + +Rotation must follow an auditable state machine: + +1. resolve authority and current immutable version; +2. create or obtain a candidate version; +3. validate the candidate against each declared dependent without exposing it; +4. activate the candidate for new retrievals; +5. observe a policy-bounded overlap period when the provider supports dual validity; +6. revoke the old version; and +7. verify dependents and record metadata-only evidence. + +Providers that cannot support overlap require an explicitly approved atomic-cutover plan, rollback point, and maintenance boundary. Partial validation, unknown dependents, or failed revocation cannot be reported as successful rotation. + +### FR-8: Credential sync and drift + +Sync reconciles credential metadata and approved version references from the authority. It must not replicate plaintext into an application database. Production requests use an explicit policy-approved credential set or namespace registry; caller `null = all` compatibility behavior cannot enumerate or sync an entire vault. Metadata listing is access-controlled, filtered, bounded, and paginated. Drift states include missing authority, unexpected version, stale consumer, expired credential, unregistered dependent, and authority mismatch. A forced refresh does not override authorization or authority selection. + +### FR-9: Evidence and audit + +Privacy evidence records request/correlation ID, tenant/subject pseudonymous identifiers, policy/detector/key versions, entity counts, token-map operation, caller identity, decision, and timestamps. Credential evidence records authority ID, credential ID, version identifiers, consumer identities, state transitions, validation outcomes, and timestamps. Neither evidence type contains raw PII, token-map values, secrets, authentication material, or provider response bodies. + +### FR-10: Compatibility and enforcement + +Existing public method signatures remain available. Production enforcement must use additive versioned request/result contracts that carry missing tenant, purpose, authority, caller, policy, and version context. Legacy calls that cannot prove this context are restricted to trusted compatibility paths and cannot independently qualify a protected privacy or credential operation. + +## Non-Functional Requirements + +This capability inherits [`global-nfr.md`](../global-nfr.md). + +| Area | Requirement | +| --- | --- | +| Privacy | Raw PII is minimized; tokens remain personal data when re-identification or linkability is possible. | +| Security | Detokenization and credential retrieval are deny-by-default, least privilege, and fully audited without sensitive values. | +| Isolation | Tenant, subject, purpose, environment, and credential namespace boundaries are enforced. | +| Availability | External calls fail closed when required privacy processing or credential authority is unavailable; bounded retry cannot weaken policy. | +| Performance | Initial planning target: tokenization adds no more than 250 ms p95 for text payloads up to the approved size; validate by baseline before enforcement. | +| Recoverability | Token-map keys and vault versions follow separately approved backup, retention, and break-glass policies. | +| Observability | Metrics use counts, reason codes, versions, and durations only; high-cardinality identifiers and sensitive values are excluded. | + +## UX and Operational Flow + +### Privacy flow + +1. An authenticated composition host resolves tenant, subject, purpose, and privacy policy. +2. The tokenization engine detects and tokenizes required entities. +3. The host sends only tokenized content to external providers and general-purpose stores. +4. Results remain tokenized unless an authorized workflow explicitly detokenizes them. +5. Audit and telemetry record metadata-only evidence. + +### Credential flow + +1. The composition host resolves the credential namespace to its configured authority. +2. The credential engine authorizes the consumer and retrieves metadata/current version. +3. The value is exposed through a bounded just-in-time use path. +4. The consumer releases the value and records only version/outcome metadata. +5. Rotation uses the staged state machine and the linked runbook. + +## Acceptance Criteria + +- No parallel public PII-tokenization or credential-sync capability port is introduced. +- Tests prove raw PII cannot reach external LLM/embedding/tool adapters or general-purpose persistence when policy requires tokenization. +- Tests prove token scopes do not correlate across tenants, subjects, or purposes. +- Production tokenization results omit `OriginalValue` and audit/log tests reject raw PII. +- Oversized input follows deterministic policy limits, and reversible results cannot escape without committed mappings. +- Detokenization rejects missing identity, tenant, subject, purpose, reason, or scope and does not log restored content. +- GDPR erasure coordinates token-map deletion and reports every other retention surface separately. +- Credential requests cannot select arbitrary provider names or vault URLs. +- Credential sync/list operations cannot use caller-controlled unbounded vault enumeration. +- Exactly one configured authority exists per credential namespace/environment. +- Credential plaintext is absent from configuration, durable caches, logs, telemetry, HTTP responses, and evidence. +- Rotation tests cover dual-version success, validation failure, partial dependent failure, non-overlap atomic cutover, rollback, and failed old-version revocation. +- Existing public method signatures remain compatible while qualifying paths use versioned context-rich contracts. +- The runbook is rehearsed in a non-production environment before enforcement. + +## Risks and Mitigations + +| Risk | Mitigation | +| --- | --- | +| Detector misses PII | Versioned policy, conservative thresholds, targeted fixtures, provider-boundary blocking, and periodic evaluation. | +| Deterministic tokens become correlators | Scope tokens to tenant, subject, purpose, and key version; prohibit cross-scope token reuse. | +| Token-map compromise enables re-identification | Encrypt, isolate, minimize, rotate keys, restrict detokenization, and audit every mapping operation. | +| Existing `OriginalValue` leaks | Omit it in production results and test logs/serialization for sensitive-field rejection. | +| Caller redirects sync to a malicious vault | Resolve authority only from trusted configuration; reject caller authority mismatch. | +| Rotation breaks consumers | Declare dependents, validate candidate, use bounded overlap or approved atomic cutover, retain rollback until verification. | +| Multi-vault portability creates split brain | Permit multiple adapter types but exactly one writer authority per namespace/environment. | +| Simulated secrets engine creates false confidence | Label it demo-only and exclude it from production acceptance evidence. | + +## Milestones and Timeline + +Dates are unset until implementation is funded and owned. + +1. **Contract refinement:** additive versioned requests, authorization context, policy schemas, and redaction tests. +2. **Privacy boundary:** tokenization engine, detector/token-store adapters, composition-host enforcement, and data-rights integration. +3. **Credential authority:** authority registry, Azure Key Vault adapter, just-in-time use, metadata reconciliation, and dependent registry. +4. **Rotation rehearsal:** non-production staged rotation, token-leak exercise, rollback, and evidence review. +5. **Enforcement:** separately approved deployment and policy activation. + +## OpenAPI / Interface Links + +No public HTTP API is introduced. The implementation starts from: + +- [`IPIITokenizationPort`](../../../src/FoundationLayer/Security/Ports/IPIITokenizationPort.cs) +- [`ICredentialSyncPort`](../../../src/FoundationLayer/Security/Ports/ICredentialSyncPort.cs) +- [`pii-tokenization-and-credential-sync.md`](../../spec/pii-tokenization-and-credential-sync.md) + +## Capability Status + +- **Implemented:** the two public port contracts; GDPR data-subject/consent workflows; direct LLM clients; simulated in-memory secrets management for demo use. +- **Experimental:** none established by this PRD. +- **Planned:** tokenization and credential-sync engines, provider adapters, token-map store, privacy composition boundary, authority registry, just-in-time credential use, and staged rotation. +- **Blocked:** production enforcement until implementation, provider configuration, security review, non-production rehearsal, and separately authorized deployment are complete. + +## References + +- [`ADR-005-privacy-and-credential-authority-boundaries.md`](../../planning/ADR-005-privacy-and-credential-authority-boundaries.md) +- [`pii-tokenization-and-credential-sync.md`](../../spec/pii-tokenization-and-credential-sync.md) +- [`credential-rotation-and-token-leak-response.md`](../../runbooks/credential-rotation-and-token-leak-response.md) +- `src/BusinessApplications/Compliance/Adapters/GDPRComplianceAdapter.cs` +- `src/FoundationLayer/Security/Engines/SecretsManagementEngine.cs` diff --git a/docs/runbooks/credential-rotation-and-token-leak-response.md b/docs/runbooks/credential-rotation-and-token-leak-response.md new file mode 100644 index 00000000..700838e8 --- /dev/null +++ b/docs/runbooks/credential-rotation-and-token-leak-response.md @@ -0,0 +1,198 @@ +# Runbook: Credential Rotation and Token/PII Leak Response + +**Status:** Proposed; not yet rehearsed + +**Baton:** `deacf6eb` + +**Applies to:** planned PII tokenization and credential-sync capabilities + +## Capability Status + +- **Implemented:** current port contracts and adjacent GDPR/demo-secret components only. +- **Experimental:** none established. +- **Planned:** the engines, adapters, authority registry, protected token store, alerts, and procedures below. +- **Blocked:** operational use until implementation, synthetic rehearsal, owner assignment, and separately approved activation. + +## Safety Rules + +- Never paste PII, token mappings, credential values, authorization headers, vault responses, or secrets into chat, tickets, GitHub, Baton, logs, screenshots, or terminal transcripts. +- Use credential IDs, version IDs, token digests, controlled reason codes, and access-controlled evidence references. +- Do not fetch a value merely to prove it exists; prefer metadata/version/health validation. +- Do not rotate or revoke a production credential unless the incident scope and authority are verified and the action is authorized. +- Do not detokenize leaked content during triage. Treat tokens as personal data until proven otherwise. +- Do not weaken privacy policy, redirect to another vault, or enable a raw-data fallback as incident response. + +## Roles + +| Role | Responsibility | +| --- | --- | +| Incident commander | Scope, authorization, timeline, decision log, and closeout. | +| Privacy owner | PII classification, subject/tenant impact, legal/notification workflow, and retention decisions. | +| Credential owner | Authority/version verification, dependent inventory, rotation or revocation approval. | +| Platform operator | Execute approved metadata, rotation, rollback, and health operations. | +| Service owner | Validate dependents and remediate source/persistence/provider paths. | +| Security reviewer | Confirm evidence, containment, redaction, and closeout criteria. | + +## Initial Classification + +Classify without retrieving sensitive values: + +| Event | Initial class | +| --- | --- | +| Raw PII reached an external model/tool, log, telemetry, or general store | PII exposure | +| Scoped token appeared outside its approved tenant/subject/purpose path | Token exposure/linkability | +| Token-map store or detokenization scope was accessed unexpectedly | Re-identification risk | +| Credential metadata drift or expiry without value exposure | Credential lifecycle incident | +| Credential value, header, or provider response was exposed | Credential compromise | +| Rotation candidate fails a dependent before activation | Rotation failure; current version remains active | +| Active version breaks dependents or old version cannot be revoked | Post-activation incident / rollback required | + +## Common First Response + +1. Open an access-controlled incident using only identifiers and evidence links. +2. Verify environment, tenant/namespace, authenticated actor/workload, authority or privacy policy, exact version, and time window. +3. Stop further exposure through the narrowest reversible control: pause the affected provider call, consumer, persistence path, or rotation workflow. +4. Preserve metadata-only evidence and access logs. Do not copy payloads into the incident record. +5. Notify the privacy owner for any raw PII, token-map, or cross-scope token event; notify the credential owner for any value or lifecycle event. +6. Choose the applicable procedure below. + +## Procedure A: Raw PII or Token Leak + +### A1. Contain + +1. Disable or quarantine the exact unqualified outbound/persistence path; keep unrelated qualified paths running when safe. +2. Record policy version, detector version, token key version, tenant/subject pseudonymous scope, provider/store identity, and first/last observed timestamps. +3. If a token-map access path is implicated, revoke the caller/session and disable detokenization for that scope pending review. +4. If external provider retention may apply, preserve the provider request/evidence identifier and invoke the approved privacy/provider process; do not re-send the content. + +### A2. Assess + +1. Determine whether the exposed material was raw PII, a scoped token, an encrypted mapping, or metadata. +2. Identify PII types and affected subject counts from protected analysis tooling, not from copied payloads. +3. Check tenant, subject, purpose, policy, and key-version boundaries for cross-scope linkability. +4. Determine every surface reached: provider, source of record, general persistence, vector/embedding store, cache, telemetry, log, evidence, backup, or export. +5. Privacy/legal owners decide notification and regulatory steps outside this runbook. + +### A3. Remediate + +1. Fix the missing/incorrect privacy qualification or unauthorized detokenization path. +2. Delete or quarantine unauthorized general-store copies under approved retention policy. +3. For compromised token mappings or cross-scope tokens, rotate the token key/version and re-tokenize affected authorized content through a separately approved migration. +4. Coordinate `DeleteTokensForSubjectAsync` only through a verified data-rights or incident workflow; do not use deletion to conceal evidence. +5. Add seeded regression fixtures that prove raw/cross-scope content cannot reach the affected adapter. + +### A4. Verify + +1. Re-run the affected path with synthetic canary PII. +2. Confirm provider/general-store test adapters receive only properly scoped tokens. +3. Confirm logs, telemetry, errors, and evidence contain no canary originals or restorable mappings. +4. Confirm authorized systems of record and data-rights paths still behave as policy requires. + +## Procedure B: Planned Credential Rotation + +### B1. Preflight + +1. Verify the credential ID, namespace/environment, configured authority, current immutable version, owner, and dependent-set version. +2. Confirm caller authorization and change/incident approval. +3. Confirm every dependent has a validation owner and rollback signal. +4. Determine whether the provider supports overlapping versions. If not, use the approved atomic-cutover plan and maintenance boundary. +5. Confirm the current version remains recoverable until candidate validation and activation evidence are complete. + +### B2. Rotate + +1. Start one idempotent rotation record bound to the current version and dependent set. +2. Create or obtain the candidate at the authoritative provider. +3. Validate the candidate against every declared dependent without logging or persisting its value. +4. If any validation fails, stop before activation, revoke the candidate when safe, and leave the current version active. +5. Activate the candidate for new just-in-time retrievals. +6. Observe the bounded overlap period or atomic-cutover health window. +7. Revoke the previous version. +8. Verify every dependent uses the active version and record metadata-only evidence. + +### B3. Rotation rollback + +Rollback is available only while the previous version remains valid or the provider supports reactivation: + +1. stop new candidate retrievals; +2. restore the previous version reference for authorized consumers; +3. validate every dependent; +4. revoke/quarantine the failed candidate; and +5. record the incomplete state and cause. + +If rollback is impossible, escalate as a post-activation incident and restore service through the provider-specific, separately approved recovery plan. + +## Procedure C: Suspected Credential Compromise + +1. Verify the credential and authority from metadata; do not retrieve the exposed value. +2. Disable the compromised consumer/session and block new use where supported. +3. Inventory dependents and exposure window from access/audit logs. +4. For confirmed or high-confidence compromise, use emergency rotation/revocation policy. Do not wait for the normal grace period when overlap extends attacker access. +5. Create and validate a replacement version, activate it, and revoke the compromised version as an authorized atomic sequence. +6. Rotate derived sessions/tokens and inspect adjacent credentials only when evidence supports that scope. +7. Validate every dependent and monitor for continued use of the revoked version. +8. Remediate the exposure source and add canary/redaction regression tests. + +## Procedure D: Sync Drift or Provider Failure + +1. Verify configured authority and provider health using metadata-only operations. +2. Compare authority version, registry version, cached metadata, and dependent-observed version. +3. Reject caller-supplied alternate providers or URLs; do not fail over authority implicitly. +4. Retry bounded transient failures with the same authority and idempotency key. +5. If exhaustion occurs, block new rotations and unsafe credential-dependent actions; preserve current known-good use only when policy explicitly permits it. +6. Authority migration or provider failover requires a separate approved cutover with reconciliation and rollback. + +## Escalation + +| Condition | Escalate to | Timing | +| --- | --- | --- | +| Raw PII reached an unauthorized external or durable surface | Privacy owner, security lead, legal/compliance | Immediate | +| Cross-tenant/subject/purpose token reuse or token-map access | Privacy owner and security lead | Immediate | +| Credential value or auth header exposed | Credential owner, security lead, affected service owners | Immediate | +| Candidate validation fails before activation | Credential and service owners | Same change window | +| Active rotation causes outage or old version cannot be revoked | Incident commander and provider owner | Immediate | +| Authority mismatch or two active writers detected | Security and architecture owners | Immediate; freeze sync/rotation | +| Provider outage exhausts bounded retry | Platform and credential owners | Per service severity | + +## Evidence Checklist + +- Incident/change identifier and controlled evidence links. +- Environment, tenant/namespace, authority/policy IDs, and affected time window. +- Pseudonymous subject count or credential IDs; no raw values. +- Detector, token key, provider adapter, registry, dependent-set, and credential version identifiers. +- Authorization and approval decision metadata. +- Containment, candidate validation, activation, rollback, revocation, and verification outcomes. +- Synthetic regression/rehearsal results. +- Privacy/legal notification decision reference when applicable. + +## Closeout Criteria + +- The exposure or rotation source is identified and remediated. +- Unauthorized paths are disabled or now enforce the approved policy. +- No raw PII, mappings, or credential values remain in ordinary incident artifacts. +- Every affected store/provider/consumer surface has an explicit disposition. +- Active credential authority and version are unambiguous; the old/compromised version is revoked or an exception is actively tracked. +- All registered dependents validate the intended version. +- Synthetic privacy and credential canaries pass without sensitive log/evidence output. +- Privacy, security, credential, and service owners approve closeout as applicable. + +## Rehearsal Requirement + +Before production activation, rehearse with synthetic PII and non-production credentials: + +- a required-detector outage; +- raw-content blocking at an LLM and memory adapter; +- cross-purpose token replay; +- unauthorized detokenization; +- dual-version rotation success; +- candidate validation failure before activation; +- non-overlap atomic cutover and rollback; +- compromised-version emergency revocation; and +- authority mismatch with no outbound call to the supplied URL. + +Record metadata-only results and update this runbook before enforcement. + +## References + +- [`pii-tokenization-and-credential-sync.md`](../prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md) +- [`ADR-005-privacy-and-credential-authority-boundaries.md`](../planning/ADR-005-privacy-and-credential-authority-boundaries.md) +- [`pii-tokenization-and-credential-sync.md`](../spec/pii-tokenization-and-credential-sync.md) diff --git a/docs/spec/pii-tokenization-and-credential-sync.md b/docs/spec/pii-tokenization-and-credential-sync.md new file mode 100644 index 00000000..e0851fd0 --- /dev/null +++ b/docs/spec/pii-tokenization-and-credential-sync.md @@ -0,0 +1,266 @@ +# Technical Specification: PII Tokenization and Credential Sync + +**Status:** Proposed; implementation not started + +**Baton:** `deacf6eb` + +**Public ports:** `IPIITokenizationPort`, `ICredentialSyncPort` + +## 1. Scope and Current State + +This specification defines compatible future implementations of the two existing Foundation Security ports. It does not add code or a parallel public capability port. + +Current repository state: + +- `IPIITokenizationPort` defines detection, tokenization, detokenization, subject deletion, and statistics. +- `ICredentialSyncPort` defines value/metadata access, listing, provider sync, rotation, revocation, expiry queries, and validation. +- neither port has an engine or adapter implementation; +- `GDPRComplianceAdapter` has data-subject workflows but does not consume the tokenization port; +- `SecretsManagementEngine` is an in-memory demo implementing another port; and +- LLM clients currently accept raw text directly. + +## 2. Component Placement + +| Component | Layer/location | Responsibility | +| --- | --- | --- | +| `PIITokenizationEngine` | Foundation Security engine | Policy validation, deterministic span resolution, token scope, fail-closed result, and authorization orchestration. Implements `IPIITokenizationPort`. | +| PII detector adapter(s) | Foundation Security adapter | Provider/local detector invocation and normalized candidate spans. Never persist raw content. | +| Token-map store adapter | Foundation Security adapter | Encrypted mapping persistence, optimistic concurrency, expiry/deletion, and key-version metadata. | +| Privacy boundary decorator/service | Composition host | Invoke tokenization before external providers and general-purpose persistence; reject unqualified payloads. | +| `GDPRComplianceAdapter` | Business Applications | Consent/legal basis/data rights; coordinate verified token-map operations through the Foundation port. | +| `CredentialSyncEngine` | Foundation Security engine | Authority resolution, consumer authorization, metadata reconciliation, lifecycle state, rotation, and revocation. Implements `ICredentialSyncPort`. | +| Vault provider adapter(s) | Foundation Security adapter | Provider-native version retrieval, candidate creation, activation, revocation, and health mapping. | +| Credential authority registry | Composition/configuration boundary | Signed/reviewed mapping from namespace/environment to one authority and adapter. | +| Dependent validator adapter(s) | Foundation Security adapter | Validate a candidate version against a declared consumer without logging the value. | + +Provider-facing outbound interfaces may be added as internal Foundation implementation ports, such as token-store, detector, vault-provider, and dependent-validator ports. They are not replacements for the two public capability ports and must not be exposed as alternate application entrypoints. + +## 3. Versioned Privacy Contract + +The existing methods remain supported for source compatibility. Production enforcement requires an additive structured request, under the same port boundary, carrying at least: + +| Field | Requirement | +| --- | --- | +| `RequestId` / `CorrelationId` | Stable non-secret identifiers. | +| `TenantId` | Required authenticated tenant scope. | +| `DataSubjectId` | Internal pseudonymous subject identifier; never raw PII. | +| `ProcessingPurpose` | Approved purpose code, not free-form sensitive text. | +| `PolicyId` / `PolicyVersion` | Authoritative privacy policy selected by the host. | +| `DetectorVersions` | Immutable detector/rule/model identifiers. | +| `TokenKeyVersion` | Active tokenization key/version resolved by policy. | +| `TargetTypes` | Policy-approved set; callers cannot silently omit required types. | +| `ConfidenceThresholds` | Per-type policy thresholds. | +| `Reversible` | Policy decision; caller cannot promote irreversible to reversible. | +| `CallerIdentity` / `Scopes` | Authenticated authorization context. | +| `Content` | Sensitive transient input; excluded from logs and serialization outside the request path. | +| `ContentLimit` / `ChunkingPolicyVersion` | Host-resolved limits and deterministic chunking rules; caller cannot increase them. | +| `IdempotencyKey` | Binds scope, policy, content digest, and operation without exposing content. | + +Legacy `TokenizationRequest` lacks tenant, purpose, policy, caller, and key context. It may remain for trusted compatibility/test use, but cannot independently qualify an enforced provider or persistence boundary. + +### 3.1 Detection normalization + +Detector adapters return candidate spans with type, confidence, detector version, and optional rule identifier. The engine: + +1. rejects invalid indexes, non-finite confidence, and candidates outside content bounds; +2. normalizes indexes to the agreed .NET string convention; +3. filters against policy-selected types and per-type thresholds; +4. resolves overlaps by longest span, then higher confidence, then stricter policy type order, then lower start index; +5. orders accepted spans by start index; +6. replaces content from the end toward the start; and +7. records counts and decision metadata without original values. + +If policy requires a detector and it fails, times out, or returns malformed data, the qualifying boundary fails. Optional advisory detectors may report a non-qualifying warning but cannot override required results. + +Input above policy limits is rejected or divided by a versioned deterministic chunker. Chunk boundaries must preserve global offsets and overlap enough to detect entities crossing a boundary; duplicate overlap findings are resolved by the same normalization rules. Entity-count and expansion limits prevent resource exhaustion. + +### 3.2 Token construction and mapping + +A token identity is derived from a cryptographically protected opaque identifier plus tenant, subject, purpose, type, policy, and key-version scope. It must not embed original data. Format-preserving output may preserve a syntactic category but cannot retain identifying substrings. + +Mapping records contain encrypted original value, token digest/identifier, tenant, subject, purpose, PII type, key version, creation/expiry, and concurrency version. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. + +For reversible tokenization, mapping writes and result publication form one idempotent operation: retrying the same scoped key/content/policy tuple returns the same committed mapping, a conflicting tuple fails, and tokenized output is not returned until all required mappings commit. Partial writes are rolled back or remain unreachable and are reconciled by a metadata-only recovery job. + +`PIIEntity.OriginalValue` remains for compatibility but is null in production-facing results. A privileged detector-to-engine in-memory handoff may carry the value only until the encrypted mapping is committed, after which buffers/references are released. + +### 3.3 Detokenization + +Add a versioned detokenization request carrying tenant, subject, purpose, authenticated caller, scopes, correlation ID, and target-token set. The engine checks every token against every scope and returns no partial cross-scope values. Default behavior is all-or-nothing; a separately named administrative batch operation may provide per-token failures without restored values in its audit result. + +The current `RequestedBy` and free-form `Reason` are audit attributes, not proof of identity or authorization. Production hosts populate them from authenticated context and use a controlled reason code plus optional non-sensitive note. + +### 3.4 Data-subject deletion + +`DeleteTokensForSubjectAsync` deletes or cryptographically erases token mappings for the authenticated tenant/subject and returns metadata-only counts. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. + +## 4. Privacy Boundary Integration + +The composition host creates a privacy-qualified payload before calling any external LLM, embedding, reranker, or content-bearing tool adapter. The payload carries tokenized content plus a qualification manifest: + +- tenant/subject/purpose scope; +- policy and detector versions; +- token key version; +- entity counts by type; +- keyed, scope-bound original-content digest computed inside the protected boundary when policy permits (never a plain hash of low-entropy PII); +- tokenized-content digest; and +- success/failure reason codes. + +Provider adapters accept only a qualified payload in enforced mode. General-purpose persistence accepts tokenized content plus the manifest. Raw authoritative records use a separate explicitly authorized path and cannot be inferred from a generic save method. + +Model/provider output may contain newly inferred PII. Policy determines whether response detection/tokenization must run before display or persistence. Required response processing also fails closed. + +## 5. Versioned Credential Contract + +Existing `ICredentialSyncPort` signatures remain source-compatible. Qualifying production operations use additive versioned requests with: + +| Field | Requirement | +| --- | --- | +| `CredentialId` | Stable logical identifier, not a secret or raw provider URL. | +| `Namespace` / `Environment` | Authority lookup key. | +| `AuthorityId` | Host-resolved expected authority; engine verifies registry match. | +| `ExpectedVersion` | Optional immutable concurrency guard. | +| `ConsumerId` | Authenticated workload identity. | +| `Scopes` | Retrieve, metadata, sync, rotate, revoke, or validate. | +| `ReasonCode` / `CorrelationId` | Controlled audit metadata. | +| `DependentSetVersion` | Immutable registry version for rotation validation. | + +`CredentialSyncRequest.VaultProvider` and `VaultUrl` remain compatibility fields. In production, the engine resolves authority from the registry and rejects any caller value that does not exactly match; arbitrary URLs are never dereferenced. `ForceRefresh` bypasses metadata freshness only, not authorization, authority, version, or dependent checks. + +### 5.1 Authority registry + +Registry entries are uniquely keyed by namespace/environment and contain authority ID, provider adapter ID/version, approved endpoint identifier, tenant/subscription/account scope, authentication mode, credential-name mapping, and policy version. Exactly one active writer entry is permitted. Authority changes use a separately approved migration record with source/target reconciliation, cutover instant, and rollback. + +Production sync selectors come from the registry or an explicit policy-approved credential set. Legacy `CredentialNames = null` cannot mean unbounded vault enumeration in a qualifying operation. Provider adapters must use bounded pagination and stop at the registered namespace; discovery outside that boundary fails rather than expanding scope. + +### 5.2 Value retrieval + +The legacy `GetCredentialValueAsync` returns an immutable .NET `string`, which cannot be reliably zeroized. It remains supported for compatibility but is not the preferred production path. Add a bounded-use overload or callback under `ICredentialSyncPort` that: + +- authorizes consumer, credential, namespace, and purpose; +- retrieves the exact provider version just in time; +- exposes the value only within the callback/lease lifetime; +- prevents serialization and diagnostics capture; +- releases disposable buffers where possible; and +- records only authority/version/outcome metadata. + +No application HTTP endpoint exposes credential values. + +### 5.3 Metadata reconciliation + +Permitted durable metadata includes credential ID, type, status, authority, provider reference without embedded secrets, version ID, created/expiry/rotation times, tags approved by schema, and declared consumers. Provider error strings are normalized and redacted before storage. + +Add a versioned paginated metadata query under the existing port boundary with a policy-bounded page size and opaque continuation token bound to caller, namespace, filter, and registry version. The legacy list method remains available to trusted compatibility callers but cannot expose an unbounded production namespace. + +Sync calculates one of: `InSync`, `MissingAtAuthority`, `UnexpectedVersion`, `StaleConsumer`, `Expired`, `Revoked`, `AuthorityMismatch`, `UnregisteredDependent`, or `ProviderUnavailable`. Plaintext never participates in drift reports. + +### 5.4 Rotation state machine + +```text +Requested -> CandidateCreated -> CandidateValidated -> Activated + -> OverlapObserved -> PreviousRevoked -> Verified -> Completed +``` + +Any state may enter `Failed`; pre-activation failures leave the current version active. Post-activation failures enter `RollbackRequired` or `Incident` according to provider capability. `Completed` requires every registered dependent to validate the active version and the previous version to be revoked or covered by an explicitly tracked provider limitation. + +Idempotency keys bind credential ID, authority, current version, dependent-set version, and request ID. A retry resumes the recorded state; it does not create another candidate blindly. + +Providers without overlapping versions use a documented atomic cutover. The engine must not pretend a grace period exists when the provider cannot honor it. + +## 6. Authorization + +Every operation is deny-by-default and authenticates the caller outside the request payload. Minimum scopes: + +| Operation | Required scope | +| --- | --- | +| Tokenize/detect | tenant + purpose + `pii:tokenize` or `pii:detect` | +| Detokenize | tenant + subject/purpose + `pii:detokenize` | +| Delete/statistics | verified data-rights/admin scope | +| Credential metadata/list | namespace + `credential:metadata` | +| Credential use | credential + consumer + `credential:use` | +| Sync | authority/namespace + `credential:sync` | +| Rotate | credential + `credential:rotate`; elevated approval policy | +| Revoke | credential + `credential:revoke`; emergency policy where applicable | + +Caller-supplied identity strings such as `RequestedBy` are never trusted as authentication evidence. + +## 7. Failure and Redaction Semantics + +Use stable reason codes. Examples: + +- `privacy_context_missing` +- `privacy_policy_mismatch` +- `required_detector_unavailable` +- `detector_output_invalid` +- `token_scope_mismatch` +- `detokenization_forbidden` +- `token_mapping_not_found` +- `credential_authority_missing_or_mismatch` +- `credential_access_forbidden` +- `credential_version_conflict` +- `candidate_validation_failed` +- `previous_version_revocation_failed` +- `provider_unavailable` + +Exceptions and result objects must not contain source content, original/restored PII, tokens that enable lookup, credential values, authorization headers, provider response bodies, or sensitive vault URLs. Redaction happens before logging, retry classification, and evidence persistence. + +## 8. Audit and Observability + +Emit metadata-only events for privacy request accepted/rejected, detection completed, mapping stored/deleted, detokenization allowed/denied, credential authority resolved/rejected, candidate created/validated, activation, rollback, revocation, and final verification. + +Metrics use bounded labels: policy version, detector/adapter ID, PII type, credential type, provider type, operation, result class, and duration bucket. Tenant, subject, credential, token, vault, correlation, and version identifiers stay in access-controlled audit evidence rather than metric labels. + +Audit read paths require explicit scope and return deny-by-default. Retention is configured by privacy/security policy and must not be silently extended by observability systems. + +## 9. Test Strategy + +### Unit and contract tests + +- invalid/overlapping detector spans normalize deterministically; +- deterministic chunking preserves cross-boundary entities and enforces size/entity limits; +- reversible tokenization never returns tokens for uncommitted mappings and retries idempotently; +- per-type thresholds and required detectors come from policy; +- tokens differ across tenant, subject, purpose, and key version; +- production results and serializers omit `OriginalValue`; +- token/detokenization rejects missing or mismatched scope; +- raw PII and restored values never reach logs, metrics, or evidence fixtures; +- caller-selected vault provider/URL is rejected on authority mismatch; +- null/all selectors and pagination tokens cannot escape the registered credential namespace; +- credential access rejects unregistered consumers and insufficient scopes; +- legacy signatures remain source-compatible while qualifying paths require structured context; +- rotation state transitions, idempotent resume, rollback, and non-overlap cutover behave as specified; and +- provider-native errors are redacted before normalization. + +### Integration tests + +- composition-host tests prove raw fixtures cannot reach LLM, embedding, tool, memory, vector, telemetry, or evidence test adapters; +- authorized source-of-record paths remain separate from general persistence; +- GDPR erasure coordinates token-map deletion and reports other retention surfaces; +- a fake vault proves single authority, metadata-only sync, just-in-time use, dependent validation, activation, and old-version revocation; +- stale consumers and missing dependents prevent successful rotation; +- simulated `SecretsManagementEngine` cannot be selected as production acceptance evidence; and +- fail-closed behavior survives cancellation, timeout, retry, and provider outage. + +### Security tests + +- cross-tenant, cross-subject, and cross-purpose token replay fails; +- arbitrary vault URL/authority injection fails without outbound access; +- logs and artifacts pass seeded PII/credential canary scans; +- concurrency tests prevent two active writers or two independent rotation candidates; and +- break-glass actions require the documented elevated policy and produce metadata-only evidence. + +## 10. Rollout Gates + +1. Contract/security review of additive models and authorization context. +2. Detector evaluation with representative approved fixtures and measured false-negative/false-positive rates. +3. Token-store threat model, key lifecycle, recovery, and data-rights rehearsal. +4. Vault adapter, authority registry, managed identity, and dependent registry review. +5. Non-production privacy-boundary and rotation exercises using synthetic data and test credentials. +6. Separately authorized deployment and enforcement. + +## Capability Status + +- **Implemented:** contracts and adjacent GDPR/LLM/demo-secret code described in section 1. +- **Experimental:** none accepted. +- **Planned:** all engines, adapters, stores, decorators, versioned contracts, policies, and tests in this specification. +- **Blocked:** production rollout until implementation and every rollout gate is satisfied. From c36f2a270e53a4e8564b944364cff7be74011611 Mon Sep 17 00:00:00 2001 From: Jurie Smit Date: Sun, 23 Aug 2026 12:18:05 +0200 Subject: [PATCH 2/6] docs: harden identity and rotation contracts --- ...acy-and-credential-authority-boundaries.md | 4 ++- .../pii-tokenization-and-credential-sync.md | 12 ++++--- ...ential-rotation-and-token-leak-response.md | 23 +++++++------- .../pii-tokenization-and-credential-sync.md | 31 ++++++++++++------- 4 files changed, 41 insertions(+), 29 deletions(-) diff --git a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md index d8a88e0f..e61efade 100644 --- a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md +++ b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md @@ -32,6 +32,8 @@ This preserves dependency direction: Business Applications can consume Foundatio Production tokens are opaque and scoped to tenant, internal subject identifier, processing purpose, policy, and key version. Cross-tenant or cross-purpose deterministic token reuse is forbidden. Token mappings and detokenization are protected data processing and remain subject to privacy policy. +A request-level subject identifier describes the authenticated request context; it does not prove that every detected entity belongs to that person. Each mapping carries an ownership state (`Verified`, `Shared`, or `Unassigned`) and zero or more verified internal subject references. Detector adapters never infer identity ownership. Unassigned entities remain bound to protected source-record lineage, while shared mappings require a verified multi-subject data-rights decision rather than deletion under whichever person submitted the text. + ### 4. Credential authority is single-writer per namespace/environment Each credential namespace and environment has exactly one authoritative vault selected by reviewed configuration. Azure Key Vault is the production default for Cognitive Mesh. The existing provider-neutral contract remains valid so an explicitly approved namespace may use another provider adapter. @@ -95,7 +97,7 @@ Rejected. It is an in-memory demo implementation with a different contract and n ## Guardrails - No silent raw-content fallback when tokenization is required. -- No raw PII, restored values, tokens with mapping value, or credentials in logs, telemetry, evidence, chat, Git, or Baton. +- No raw PII, restored values, token-map values, lookup-capable token identifiers, cross-scope-linkable tokens, or credentials in logs, telemetry, evidence, chat, Git, or Baton. - No raw email/name/national identifier as `DataSubjectId`. - No caller-selected production vault URL/provider. - No plaintext durable credential cache. diff --git a/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md b/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md index 22b9b9a5..ace8dbf5 100644 --- a/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md +++ b/docs/prds/01-foundational-infrastructure/pii-tokenization-and-credential-sync.md @@ -70,11 +70,11 @@ Raw content may exist only transiently in the authenticated request boundary, in ### FR-2: GDPR responsibility boundary -`GDPRComplianceAdapter` remains responsible for consent, lawful-basis assessment, retention, and data-subject workflows. It may orchestrate `DeleteTokensForSubjectAsync` and token statistics through `IPIITokenizationPort`, but it does not implement detection, token generation, or token-map persistence. Erasure closeout must distinguish deletion from token mappings, authorized systems of record, caches, model/provider retention, backups, and audit records. +`GDPRComplianceAdapter` remains responsible for consent, lawful-basis assessment, retention, and data-subject workflows. It may orchestrate `DeleteTokensForSubjectAsync` and token statistics through `IPIITokenizationPort`, but it does not implement detection, token generation, or token-map persistence. Erasure closeout must distinguish deletion from token mappings, authorized systems of record, caches, model/provider retention, backups, and audit records. A submitter's request-level subject ID does not make every person mentioned in the payload that submitter's data: mappings must retain verified, shared, or unassigned subject ownership and protected source-record lineage. ### FR-3: Scoped tokenization -Qualifying tokens must be opaque, non-semantic, and scoped by tenant, internal data-subject identifier, processing purpose, policy version, and tokenization key/version. The raw email, name, or other PII value must not be used as `DataSubjectId`. Cross-tenant or cross-purpose deterministic tokens are prohibited because they enable linkability. +Qualifying tokens must be opaque, non-semantic, and scoped by tenant, verified subject or protected record ownership scope, processing purpose, policy version, and tokenization key/version. The raw email, name, or other PII value must not be used as `DataSubjectId`. Detector adapters cannot infer which person owns an entity. Unknown ownership remains `Unassigned`; a value verified as belonging to multiple subjects is `Shared`. Cross-tenant or cross-purpose deterministic tokens are prohibited because they enable linkability. Detection must define deterministic overlap resolution, confidence behavior, detector/rule versions, and handling for unsupported content. `OriginalValue` is sensitive compatibility data: production results omit it by default and must never persist it in logs, telemetry, or general evidence. @@ -105,8 +105,9 @@ Rotation must follow an auditable state machine: 3. validate the candidate against each declared dependent without exposing it; 4. activate the candidate for new retrievals; 5. observe a policy-bounded overlap period when the provider supports dual validity; -6. revoke the old version; and -7. verify dependents and record metadata-only evidence. +6. prove every registered dependent has adopted the active version; +7. revoke the old version; and +8. verify dependents again and record metadata-only evidence. Providers that cannot support overlap require an explicitly approved atomic-cutover plan, rollback point, and maintenance boundary. Partial validation, unknown dependents, or failed revocation cannot be reported as successful rotation. @@ -159,6 +160,7 @@ This capability inherits [`global-nfr.md`](../global-nfr.md). - No parallel public PII-tokenization or credential-sync capability port is introduced. - Tests prove raw PII cannot reach external LLM/embedding/tool adapters or general-purpose persistence when policy requires tokenization. - Tests prove token scopes do not correlate across tenants, subjects, or purposes. +- Tests prove request submitters do not automatically own other subjects' entities and that verified, shared, and unassigned mappings follow distinct deletion rules. - Production tokenization results omit `OriginalValue` and audit/log tests reject raw PII. - Oversized input follows deterministic policy limits, and reversible results cannot escape without committed mappings. - Detokenization rejects missing identity, tenant, subject, purpose, reason, or scope and does not log restored content. @@ -167,7 +169,7 @@ This capability inherits [`global-nfr.md`](../global-nfr.md). - Credential sync/list operations cannot use caller-controlled unbounded vault enumeration. - Exactly one configured authority exists per credential namespace/environment. - Credential plaintext is absent from configuration, durable caches, logs, telemetry, HTTP responses, and evidence. -- Rotation tests cover dual-version success, validation failure, partial dependent failure, non-overlap atomic cutover, rollback, and failed old-version revocation. +- Rotation tests cover dual-version success, active-adoption proof before old-version revocation, validation failure, partial dependent failure, non-overlap atomic cutover, rollback, emergency compromise, and failed old-version revocation. - Existing public method signatures remain compatible while qualifying paths use versioned context-rich contracts. - The runbook is rehearsed in a non-production environment before enforcement. diff --git a/docs/runbooks/credential-rotation-and-token-leak-response.md b/docs/runbooks/credential-rotation-and-token-leak-response.md index 700838e8..083c9ec7 100644 --- a/docs/runbooks/credential-rotation-and-token-leak-response.md +++ b/docs/runbooks/credential-rotation-and-token-leak-response.md @@ -68,8 +68,8 @@ Classify without retrieving sensitive values: ### A2. Assess 1. Determine whether the exposed material was raw PII, a scoped token, an encrypted mapping, or metadata. -2. Identify PII types and affected subject counts from protected analysis tooling, not from copied payloads. -3. Check tenant, subject, purpose, policy, and key-version boundaries for cross-scope linkability. +2. Identify PII types, ownership state (`Verified`, `Shared`, or `Unassigned`), and affected subject counts from protected analysis tooling, not from copied payloads. +3. Check tenant, verified subject set or source-record scope, purpose, policy, and key-version boundaries for cross-scope linkability; do not assign every entity to the submitter. 4. Determine every surface reached: provider, source of record, general persistence, vector/embedding store, cache, telemetry, log, evidence, backup, or export. 5. Privacy/legal owners decide notification and regulatory steps outside this runbook. @@ -100,14 +100,14 @@ Classify without retrieving sensitive values: ### B2. Rotate -1. Start one idempotent rotation record bound to the current version and dependent set. +1. Start one idempotent rotation record bound to credential ID, authority ID, current version, dependent-set version, and request ID. 2. Create or obtain the candidate at the authoritative provider. 3. Validate the candidate against every declared dependent without logging or persisting its value. 4. If any validation fails, stop before activation, revoke the candidate when safe, and leave the current version active. 5. Activate the candidate for new just-in-time retrievals. -6. Observe the bounded overlap period or atomic-cutover health window. -7. Revoke the previous version. -8. Verify every dependent uses the active version and record metadata-only evidence. +6. For a dual-validity provider, observe the bounded overlap period, verify every registered dependent actually uses the active version rather than a cached previous value, revoke the previous version, then verify again that no old-version use continues. +7. For a non-overlap provider, execute the approved atomic activation/revocation cutover and immediately validate every dependent during the health window; follow the provider-specific rollback/recovery plan on failure. +8. Record metadata-only adoption, revocation, and final verification evidence. ### B3. Rotation rollback @@ -126,11 +126,12 @@ If rollback is impossible, escalate as a post-activation incident and restore se 1. Verify the credential and authority from metadata; do not retrieve the exposed value. 2. Disable the compromised consumer/session and block new use where supported. 3. Inventory dependents and exposure window from access/audit logs. -4. For confirmed or high-confidence compromise, use emergency rotation/revocation policy. Do not wait for the normal grace period when overlap extends attacker access. -5. Create and validate a replacement version, activate it, and revoke the compromised version as an authorized atomic sequence. -6. Rotate derived sessions/tokens and inspect adjacent credentials only when evidence supports that scope. -7. Validate every dependent and monitor for continued use of the revoked version. -8. Remediate the exposure source and add canary/redaction regression tests. +4. For confirmed or high-confidence compromise, use emergency rotation/revocation policy. Do not use the normal grace period when it extends attacker access. +5. Create and validate a replacement. When the provider supports overlapping versions, activate it with an incident-approved bounded overlap only long enough for dependent cutover, then revoke the compromised version; when overlap is unsupported, use the approved atomic replacement/activation/revocation sequence. +6. If the replacement fails after activation, do not restore the compromised version. Isolate failing dependents and roll forward to another replacement or provider-specific recovery under incident command. Revoke-first containment is allowed when continued attacker access outweighs availability. +7. Rotate derived sessions/tokens and inspect adjacent credentials only when evidence supports that scope. +8. Validate every dependent and monitor for continued use of the revoked version. +9. Remediate the exposure source and add canary/redaction regression tests. ## Procedure D: Sync Drift or Provider Failure diff --git a/docs/spec/pii-tokenization-and-credential-sync.md b/docs/spec/pii-tokenization-and-credential-sync.md index e0851fd0..a9851a54 100644 --- a/docs/spec/pii-tokenization-and-credential-sync.md +++ b/docs/spec/pii-tokenization-and-credential-sync.md @@ -43,12 +43,14 @@ The existing methods remain supported for source compatibility. Production enfor | --- | --- | | `RequestId` / `CorrelationId` | Stable non-secret identifiers. | | `TenantId` | Required authenticated tenant scope. | -| `DataSubjectId` | Internal pseudonymous subject identifier; never raw PII. | +| `DataSubjectId` | Compatibility/request-context subject identifier; never raw PII and never automatic ownership proof for every entity. | +| `EntitySubjectAssociations` | Trusted source-record/span associations with ownership state (`Verified`, `Shared`, or `Unassigned`) and zero or more internal subject references. Detector output cannot populate this authority. | | `ProcessingPurpose` | Approved purpose code, not free-form sensitive text. | | `PolicyId` / `PolicyVersion` | Authoritative privacy policy selected by the host. | | `DetectorVersions` | Immutable detector/rule/model identifiers. | | `TokenKeyVersion` | Active tokenization key/version resolved by policy. | | `TargetTypes` | Policy-approved set; callers cannot silently omit required types. | +| `TypePrecedence` | Immutable ordered list containing every targeted type exactly once; earlier entries win equal-length/equal-confidence overlap ties. | | `ConfidenceThresholds` | Per-type policy thresholds. | | `Reversible` | Policy decision; caller cannot promote irreversible to reversible. | | `CallerIdentity` / `Scopes` | Authenticated authorization context. | @@ -65,10 +67,11 @@ Detector adapters return candidate spans with type, confidence, detector version 1. rejects invalid indexes, non-finite confidence, and candidates outside content bounds; 2. normalizes indexes to the agreed .NET string convention; 3. filters against policy-selected types and per-type thresholds; -4. resolves overlaps by longest span, then higher confidence, then stricter policy type order, then lower start index; -5. orders accepted spans by start index; -6. replaces content from the end toward the start; and -7. records counts and decision metadata without original values. +4. validates that immutable `TypePrecedence` contains every `TargetType` exactly once with no extras or duplicates; +5. resolves overlaps by longest span, then higher confidence, then earlier `TypePrecedence` position, then lower start index; +6. orders accepted spans by start index; +7. replaces content from the end toward the start; and +8. records counts and decision metadata without original values. If policy requires a detector and it fails, times out, or returns malformed data, the qualifying boundary fails. Optional advisory detectors may report a non-qualifying warning but cannot override required results. @@ -76,9 +79,9 @@ Input above policy limits is rejected or divided by a versioned deterministic ch ### 3.2 Token construction and mapping -A token identity is derived from a cryptographically protected opaque identifier plus tenant, subject, purpose, type, policy, and key-version scope. It must not embed original data. Format-preserving output may preserve a syntactic category but cannot retain identifying substrings. +A token identity is derived from a cryptographically protected opaque identifier plus tenant, ownership scope, purpose, type, policy, and key-version scope. Ownership scope is a verified internal subject, a verified shared-subject set, or a protected source-record identifier for unassigned content. It must not embed original data. Format-preserving output may preserve a syntactic category but cannot retain identifying substrings. -Mapping records contain encrypted original value, token digest/identifier, tenant, subject, purpose, PII type, key version, creation/expiry, and concurrency version. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. +Mapping records contain encrypted original value, token digest/identifier, tenant, ownership state, zero or more verified subject references, protected source-record lineage, purpose, PII type, key version, creation/expiry, and concurrency version. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. For reversible tokenization, mapping writes and result publication form one idempotent operation: retrying the same scoped key/content/policy tuple returns the same committed mapping, a conflicting tuple fails, and tokenized output is not returned until all required mappings commit. Partial writes are rolled back or remain unreachable and are reconciled by a metadata-only recovery job. @@ -92,7 +95,7 @@ The current `RequestedBy` and free-form `Reason` are audit attributes, not proof ### 3.4 Data-subject deletion -`DeleteTokensForSubjectAsync` deletes or cryptographically erases token mappings for the authenticated tenant/subject and returns metadata-only counts. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. +`DeleteTokensForSubjectAsync` deletes or cryptographically erases mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, the verified workflow removes the subject association and re-tokenizes or deletes only when the remaining subjects' rights and retention policy permit it. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. ## 4. Privacy Boundary Integration @@ -158,14 +161,15 @@ Sync calculates one of: `InSync`, `MissingAtAuthority`, `UnexpectedVersion`, `St ```text Requested -> CandidateCreated -> CandidateValidated -> Activated - -> OverlapObserved -> PreviousRevoked -> Verified -> Completed + -> OverlapObserved -> ActiveAdoptionVerified + -> PreviousRevoked -> PostRevokeVerified -> Completed ``` -Any state may enter `Failed`; pre-activation failures leave the current version active. Post-activation failures enter `RollbackRequired` or `Incident` according to provider capability. `Completed` requires every registered dependent to validate the active version and the previous version to be revoked or covered by an explicitly tracked provider limitation. +Any state may enter `Failed`; pre-activation failures leave the current version active. In a normal rotation, every registered dependent must prove actual use of the active version before the previous version is revoked, then validate again afterward. Post-activation failures enter `RollbackRequired` or `Incident` according to provider capability. `Completed` requires both adoption and post-revocation verification and the previous version to be revoked or covered by an explicitly tracked provider limitation. Emergency compromise may revoke first under incident policy; it never rolls back to a compromised version. Idempotency keys bind credential ID, authority, current version, dependent-set version, and request ID. A retry resumes the recorded state; it does not create another candidate blindly. -Providers without overlapping versions use a documented atomic cutover. The engine must not pretend a grace period exists when the provider cannot honor it. +The state sequence above is the dual-validity path. Providers without overlapping versions use a documented atomic cutover that collapses activation and previous-version revocation, followed by immediate dependent validation and provider-specific rollback/recovery. The engine must not pretend a grace period or pre-revocation adoption proof exists when the provider cannot honor it. ## 6. Authorization @@ -217,18 +221,21 @@ Audit read paths require explicit scope and return deny-by-default. Retention is ### Unit and contract tests - invalid/overlapping detector spans normalize deterministically; +- equal-length, equal-confidence overlap fixtures choose the type earlier in immutable `TypePrecedence`; - deterministic chunking preserves cross-boundary entities and enforces size/entity limits; - reversible tokenization never returns tokens for uncommitted mappings and retries idempotently; - per-type thresholds and required detectors come from policy; - tokens differ across tenant, subject, purpose, and key version; - production results and serializers omit `OriginalValue`; - token/detokenization rejects missing or mismatched scope; +- multi-person content preserves verified, shared, and unassigned ownership without assigning every entity to the submitter; +- subject deletion handles exclusive, shared, and unassigned mappings without erasing another subject's mapping; - raw PII and restored values never reach logs, metrics, or evidence fixtures; - caller-selected vault provider/URL is rejected on authority mismatch; - null/all selectors and pagination tokens cannot escape the registered credential namespace; - credential access rejects unregistered consumers and insufficient scopes; - legacy signatures remain source-compatible while qualifying paths require structured context; -- rotation state transitions, idempotent resume, rollback, and non-overlap cutover behave as specified; and +- rotation state transitions, idempotent resume, active-adoption proof before revocation, rollback, emergency no-rollback-to-compromised behavior, and non-overlap cutover behave as specified; and - provider-native errors are redacted before normalization. ### Integration tests From 73f72a25154fef9c1db5e836616bd7407355d264 Mon Sep 17 00:00:00 2001 From: Jurie Smit Date: Sun, 23 Aug 2026 12:28:29 +0200 Subject: [PATCH 3/6] docs: close privacy and rotation review gaps --- ...ivacy-and-credential-authority-boundaries.md | 2 +- ...edential-rotation-and-token-leak-response.md | 2 +- .../pii-tokenization-and-credential-sync.md | 17 +++++++++++------ 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md index e61efade..61e375da 100644 --- a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md +++ b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md @@ -32,7 +32,7 @@ This preserves dependency direction: Business Applications can consume Foundatio Production tokens are opaque and scoped to tenant, internal subject identifier, processing purpose, policy, and key version. Cross-tenant or cross-purpose deterministic token reuse is forbidden. Token mappings and detokenization are protected data processing and remain subject to privacy policy. -A request-level subject identifier describes the authenticated request context; it does not prove that every detected entity belongs to that person. Each mapping carries an ownership state (`Verified`, `Shared`, or `Unassigned`) and zero or more verified internal subject references. Detector adapters never infer identity ownership. Unassigned entities remain bound to protected source-record lineage, while shared mappings require a verified multi-subject data-rights decision rather than deletion under whichever person submitted the text. +A request-level subject identifier describes the authenticated request context; it does not prove that every detected entity belongs to that person. Each mapping carries exactly one valid ownership shape: `Verified` has exactly one verified internal subject reference, `Shared` has at least two distinct verified internal subject references, and `Unassigned` has no verified subject references plus protected source-record lineage. Detector adapters never infer identity ownership. Invalid state/reference/lineage combinations are rejected. Unassigned entities remain bound to protected source-record lineage, while shared mappings require a verified multi-subject data-rights decision rather than deletion under whichever person submitted the text. ### 4. Credential authority is single-writer per namespace/environment diff --git a/docs/runbooks/credential-rotation-and-token-leak-response.md b/docs/runbooks/credential-rotation-and-token-leak-response.md index 083c9ec7..a2b1d575 100644 --- a/docs/runbooks/credential-rotation-and-token-leak-response.md +++ b/docs/runbooks/credential-rotation-and-token-leak-response.md @@ -127,7 +127,7 @@ If rollback is impossible, escalate as a post-activation incident and restore se 2. Disable the compromised consumer/session and block new use where supported. 3. Inventory dependents and exposure window from access/audit logs. 4. For confirmed or high-confidence compromise, use emergency rotation/revocation policy. Do not use the normal grace period when it extends attacker access. -5. Create and validate a replacement. When the provider supports overlapping versions, activate it with an incident-approved bounded overlap only long enough for dependent cutover, then revoke the compromised version; when overlap is unsupported, use the approved atomic replacement/activation/revocation sequence. +5. Create and validate a replacement. Permit overlap with a compromised version only when the provider can block untrusted use of that version while authorized dependents cut over. Record the exception owner, maximum duration, and compensating controls before activation; keep the dependent-cutover sequence bounded and revoke the compromised version before the exception expires. If untrusted use cannot be blocked, revoke first and follow the approved provider-specific recovery path instead of creating an overlap window. 6. If the replacement fails after activation, do not restore the compromised version. Isolate failing dependents and roll forward to another replacement or provider-specific recovery under incident command. Revoke-first containment is allowed when continued attacker access outweighs availability. 7. Rotate derived sessions/tokens and inspect adjacent credentials only when evidence supports that scope. 8. Validate every dependent and monitor for continued use of the revoked version. diff --git a/docs/spec/pii-tokenization-and-credential-sync.md b/docs/spec/pii-tokenization-and-credential-sync.md index a9851a54..40b2a068 100644 --- a/docs/spec/pii-tokenization-and-credential-sync.md +++ b/docs/spec/pii-tokenization-and-credential-sync.md @@ -44,7 +44,7 @@ The existing methods remain supported for source compatibility. Production enfor | `RequestId` / `CorrelationId` | Stable non-secret identifiers. | | `TenantId` | Required authenticated tenant scope. | | `DataSubjectId` | Compatibility/request-context subject identifier; never raw PII and never automatic ownership proof for every entity. | -| `EntitySubjectAssociations` | Trusted source-record/span associations with ownership state (`Verified`, `Shared`, or `Unassigned`) and zero or more internal subject references. Detector output cannot populate this authority. | +| `EntitySubjectAssociations` | Trusted source-record/span associations with exactly one valid ownership shape: `Verified` has exactly one verified internal subject reference; `Shared` has at least two distinct verified internal subject references; `Unassigned` has no verified subject references and requires protected source-record lineage. Reject every other state/reference/lineage combination. Detector output cannot populate this authority. | | `ProcessingPurpose` | Approved purpose code, not free-form sensitive text. | | `PolicyId` / `PolicyVersion` | Authoritative privacy policy selected by the host. | | `DetectorVersions` | Immutable detector/rule/model identifiers. | @@ -81,7 +81,7 @@ Input above policy limits is rejected or divided by a versioned deterministic ch A token identity is derived from a cryptographically protected opaque identifier plus tenant, ownership scope, purpose, type, policy, and key-version scope. Ownership scope is a verified internal subject, a verified shared-subject set, or a protected source-record identifier for unassigned content. It must not embed original data. Format-preserving output may preserve a syntactic category but cannot retain identifying substrings. -Mapping records contain encrypted original value, token digest/identifier, tenant, ownership state, zero or more verified subject references, protected source-record lineage, purpose, PII type, key version, creation/expiry, and concurrency version. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. +Mapping records contain encrypted original value, token digest/identifier, tenant, validated ownership state and its required subject-reference/source-lineage shape, purpose, PII type, key version, creation/expiry, and concurrency version. The engine rejects a mapping unless `Verified` has exactly one verified internal subject reference, `Shared` has at least two distinct verified internal subject references, or `Unassigned` has no verified subject references and has protected source-record lineage. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. For reversible tokenization, mapping writes and result publication form one idempotent operation: retrying the same scoped key/content/policy tuple returns the same committed mapping, a conflicting tuple fails, and tokenized output is not returned until all required mappings commit. Partial writes are rolled back or remain unreachable and are reconciled by a metadata-only recovery job. @@ -95,7 +95,7 @@ The current `RequestedBy` and free-form `Reason` are audit attributes, not proof ### 3.4 Data-subject deletion -`DeleteTokensForSubjectAsync` deletes or cryptographically erases mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, the verified workflow removes the subject association and re-tokenizes or deletes only when the remaining subjects' rights and retention policy permit it. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. +`DeleteTokensForSubjectAsync` acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, the verified workflow removes the subject association and re-tokenizes or deletes only when the remaining subjects' rights and retention policy permit it. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. ## 4. Privacy Boundary Integration @@ -162,10 +162,14 @@ Sync calculates one of: `InSync`, `MissingAtAuthority`, `UnexpectedVersion`, `St ```text Requested -> CandidateCreated -> CandidateValidated -> Activated -> OverlapObserved -> ActiveAdoptionVerified - -> PreviousRevoked -> PostRevokeVerified -> Completed +ActiveAdoptionVerified -> PreviousRevoked -> PostRevokeVerified -> Completed +ActiveAdoptionVerified -> RevocationExceptionActive +RevocationExceptionActive -> PreviousRevoked -> PostRevokeVerified -> Completed ``` -Any state may enter `Failed`; pre-activation failures leave the current version active. In a normal rotation, every registered dependent must prove actual use of the active version before the previous version is revoked, then validate again afterward. Post-activation failures enter `RollbackRequired` or `Incident` according to provider capability. `Completed` requires both adoption and post-revocation verification and the previous version to be revoked or covered by an explicitly tracked provider limitation. Emergency compromise may revoke first under incident policy; it never rolls back to a compromised version. +Any state may enter `Failed`; pre-activation failures leave the current version active. In a normal rotation, every registered dependent must prove actual use of the active version before the previous version is revoked, then validate again afterward. Post-activation failures enter `RollbackRequired` or `Incident` according to provider capability. `Completed` is reserved for rotations with verified adoption, successful previous-version revocation, and successful post-revocation verification. + +If a provider limitation prevents revoking a previous version after adoption, the rotation enters non-terminal `RevocationExceptionActive`, not `Completed`. That state requires an accountable owner, a policy-bounded expiry, documented compensating controls, continued old-version-use monitoring, and scheduled revocation retry or authority-migration action. It can reach `Completed` only through `PreviousRevoked` and `PostRevokeVerified`. Emergency compromise may revoke first under incident policy; it never rolls back to a compromised version. Idempotency keys bind credential ID, authority, current version, dependent-set version, and request ID. A retry resumes the recorded state; it does not create another candidate blindly. @@ -222,6 +226,7 @@ Audit read paths require explicit scope and return deny-by-default. Retention is - invalid/overlapping detector spans normalize deterministically; - equal-length, equal-confidence overlap fixtures choose the type earlier in immutable `TypePrecedence`; +- invalid `Verified`, `Shared`, or `Unassigned` subject-reference/source-lineage combinations are rejected before mapping publication or deletion decisions; - deterministic chunking preserves cross-boundary entities and enforces size/entity limits; - reversible tokenization never returns tokens for uncommitted mappings and retries idempotently; - per-type thresholds and required detectors come from policy; @@ -235,7 +240,7 @@ Audit read paths require explicit scope and return deny-by-default. Retention is - null/all selectors and pagination tokens cannot escape the registered credential namespace; - credential access rejects unregistered consumers and insufficient scopes; - legacy signatures remain source-compatible while qualifying paths require structured context; -- rotation state transitions, idempotent resume, active-adoption proof before revocation, rollback, emergency no-rollback-to-compromised behavior, and non-overlap cutover behave as specified; and +- rotation state transitions, idempotent resume, active-adoption proof before revocation, non-terminal revocation-exception monitoring, rollback, emergency no-rollback-to-compromised behavior, and non-overlap cutover behave as specified; and - provider-native errors are redacted before normalization. ### Integration tests From 379a3ed6d60a54b9eccfffacbf58aa1adf9d2af1 Mon Sep 17 00:00:00 2001 From: Jurie Smit Date: Sun, 23 Aug 2026 12:36:30 +0200 Subject: [PATCH 4/6] docs: bind tenant and migration invariants --- .../ADR-005-privacy-and-credential-authority-boundaries.md | 2 +- .../runbooks/credential-rotation-and-token-leak-response.md | 2 +- docs/spec/pii-tokenization-and-credential-sync.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md index 61e375da..5d708c28 100644 --- a/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md +++ b/docs/planning/ADR-005-privacy-and-credential-authority-boundaries.md @@ -32,7 +32,7 @@ This preserves dependency direction: Business Applications can consume Foundatio Production tokens are opaque and scoped to tenant, internal subject identifier, processing purpose, policy, and key version. Cross-tenant or cross-purpose deterministic token reuse is forbidden. Token mappings and detokenization are protected data processing and remain subject to privacy policy. -A request-level subject identifier describes the authenticated request context; it does not prove that every detected entity belongs to that person. Each mapping carries exactly one valid ownership shape: `Verified` has exactly one verified internal subject reference, `Shared` has at least two distinct verified internal subject references, and `Unassigned` has no verified subject references plus protected source-record lineage. Detector adapters never infer identity ownership. Invalid state/reference/lineage combinations are rejected. Unassigned entities remain bound to protected source-record lineage, while shared mappings require a verified multi-subject data-rights decision rather than deletion under whichever person submitted the text. +A request-level subject identifier describes the authenticated request context; it does not prove that every detected entity belongs to that person. Each mapping carries exactly one valid ownership shape: `Verified` has exactly one verified internal subject reference, `Shared` has at least two distinct verified internal subject references, and `Unassigned` has no verified subject references plus protected source-record lineage. Every subject reference must resolve within the authenticated tenant and its authorized subject namespace; cross-tenant associations are rejected. Detector adapters never infer identity ownership. Invalid state/reference/lineage/tenant combinations are rejected. Unassigned entities remain bound to protected source-record lineage, while shared mappings require a verified multi-subject data-rights decision rather than deletion under whichever person submitted the text. ### 4. Credential authority is single-writer per namespace/environment diff --git a/docs/runbooks/credential-rotation-and-token-leak-response.md b/docs/runbooks/credential-rotation-and-token-leak-response.md index a2b1d575..95a3f5b7 100644 --- a/docs/runbooks/credential-rotation-and-token-leak-response.md +++ b/docs/runbooks/credential-rotation-and-token-leak-response.md @@ -127,7 +127,7 @@ If rollback is impossible, escalate as a post-activation incident and restore se 2. Disable the compromised consumer/session and block new use where supported. 3. Inventory dependents and exposure window from access/audit logs. 4. For confirmed or high-confidence compromise, use emergency rotation/revocation policy. Do not use the normal grace period when it extends attacker access. -5. Create and validate a replacement. Permit overlap with a compromised version only when the provider can block untrusted use of that version while authorized dependents cut over. Record the exception owner, maximum duration, and compensating controls before activation; keep the dependent-cutover sequence bounded and revoke the compromised version before the exception expires. If untrusted use cannot be blocked, revoke first and follow the approved provider-specific recovery path instead of creating an overlap window. +5. Resolve the registered `AuthorityId`, credential namespace, and environment before creating the replacement; reject caller-supplied alternatives and reuse Procedure B's scoped idempotency key and registered-dependent validation. Permit overlap with a compromised version only when that registered authority can block untrusted use of the version while authorized dependents cut over. Record the exception owner, maximum duration, and compensating controls before activation; keep the dependent-cutover sequence bounded and revoke the compromised version before the exception expires. If untrusted use cannot be blocked, revoke first and follow the approved provider-specific recovery path instead of creating an overlap window. 6. If the replacement fails after activation, do not restore the compromised version. Isolate failing dependents and roll forward to another replacement or provider-specific recovery under incident command. Revoke-first containment is allowed when continued attacker access outweighs availability. 7. Rotate derived sessions/tokens and inspect adjacent credentials only when evidence supports that scope. 8. Validate every dependent and monitor for continued use of the revoked version. diff --git a/docs/spec/pii-tokenization-and-credential-sync.md b/docs/spec/pii-tokenization-and-credential-sync.md index 40b2a068..2aecc74a 100644 --- a/docs/spec/pii-tokenization-and-credential-sync.md +++ b/docs/spec/pii-tokenization-and-credential-sync.md @@ -44,7 +44,7 @@ The existing methods remain supported for source compatibility. Production enfor | `RequestId` / `CorrelationId` | Stable non-secret identifiers. | | `TenantId` | Required authenticated tenant scope. | | `DataSubjectId` | Compatibility/request-context subject identifier; never raw PII and never automatic ownership proof for every entity. | -| `EntitySubjectAssociations` | Trusted source-record/span associations with exactly one valid ownership shape: `Verified` has exactly one verified internal subject reference; `Shared` has at least two distinct verified internal subject references; `Unassigned` has no verified subject references and requires protected source-record lineage. Reject every other state/reference/lineage combination. Detector output cannot populate this authority. | +| `EntitySubjectAssociations` | Trusted source-record/span associations with exactly one valid ownership shape: `Verified` has exactly one verified internal subject reference; `Shared` has at least two distinct verified internal subject references; `Unassigned` has no verified subject references and requires protected source-record lineage. Every reference must resolve within `TenantId` and its authorized subject namespace. Reject every other state/reference/lineage/tenant combination, including mixed-tenant associations, before mapping publication. Detector output cannot populate this authority. | | `ProcessingPurpose` | Approved purpose code, not free-form sensitive text. | | `PolicyId` / `PolicyVersion` | Authoritative privacy policy selected by the host. | | `DetectorVersions` | Immutable detector/rule/model identifiers. | @@ -81,7 +81,7 @@ Input above policy limits is rejected or divided by a versioned deterministic ch A token identity is derived from a cryptographically protected opaque identifier plus tenant, ownership scope, purpose, type, policy, and key-version scope. Ownership scope is a verified internal subject, a verified shared-subject set, or a protected source-record identifier for unassigned content. It must not embed original data. Format-preserving output may preserve a syntactic category but cannot retain identifying substrings. -Mapping records contain encrypted original value, token digest/identifier, tenant, validated ownership state and its required subject-reference/source-lineage shape, purpose, PII type, key version, creation/expiry, and concurrency version. The engine rejects a mapping unless `Verified` has exactly one verified internal subject reference, `Shared` has at least two distinct verified internal subject references, or `Unassigned` has no verified subject references and has protected source-record lineage. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. +Mapping records contain encrypted original value, token digest/identifier, tenant, validated ownership state and its required subject-reference/source-lineage shape, purpose, PII type, key version, creation/expiry, and concurrency version. The engine rejects a mapping unless every subject reference resolves within the mapping's authenticated tenant and authorized subject namespace and the mapping has one valid shape: `Verified` with exactly one verified internal subject reference, `Shared` with at least two distinct verified internal subject references, or `Unassigned` with no verified subject references and protected source-record lineage. Mixed-tenant associations cannot be persisted. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. For reversible tokenization, mapping writes and result publication form one idempotent operation: retrying the same scoped key/content/policy tuple returns the same committed mapping, a conflicting tuple fails, and tokenized output is not returned until all required mappings commit. Partial writes are rolled back or remain unreachable and are reconciled by a metadata-only recovery job. @@ -95,7 +95,7 @@ The current `RequestedBy` and free-form `Reason` are audit attributes, not proof ### 3.4 Data-subject deletion -`DeleteTokensForSubjectAsync` acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, the verified workflow removes the subject association and re-tokenizes or deletes only when the remaining subjects' rights and retention policy permit it. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. +`DeleteTokensForSubjectAsync` acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, the verified workflow removes the subject association and re-tokenizes or deletes only when the remaining subjects' rights and retention policy permit it. Because that changes ownership scope and token identity, the migration is atomic from the caller's perspective: create the replacement mapping, update every registered authorized reference and persisted or provider copy of the tokenized content, verify each update, and only then retire the old mapping. If any required surface fails, keep the old mapping reachable for the remaining subjects, roll back or quarantine partial replacement references, and fail closed for the deletion request. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. ## 4. Privacy Boundary Integration From cd3bac8d97f3f705e358cb4d5556356014880d03 Mon Sep 17 00:00:00 2001 From: Jurie Smit Date: Sun, 23 Aug 2026 12:42:22 +0200 Subject: [PATCH 5/6] docs: canonicalize shared subject handling --- docs/spec/pii-tokenization-and-credential-sync.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/spec/pii-tokenization-and-credential-sync.md b/docs/spec/pii-tokenization-and-credential-sync.md index 2aecc74a..c8eb9302 100644 --- a/docs/spec/pii-tokenization-and-credential-sync.md +++ b/docs/spec/pii-tokenization-and-credential-sync.md @@ -81,7 +81,7 @@ Input above policy limits is rejected or divided by a versioned deterministic ch A token identity is derived from a cryptographically protected opaque identifier plus tenant, ownership scope, purpose, type, policy, and key-version scope. Ownership scope is a verified internal subject, a verified shared-subject set, or a protected source-record identifier for unassigned content. It must not embed original data. Format-preserving output may preserve a syntactic category but cannot retain identifying substrings. -Mapping records contain encrypted original value, token digest/identifier, tenant, validated ownership state and its required subject-reference/source-lineage shape, purpose, PII type, key version, creation/expiry, and concurrency version. The engine rejects a mapping unless every subject reference resolves within the mapping's authenticated tenant and authorized subject namespace and the mapping has one valid shape: `Verified` with exactly one verified internal subject reference, `Shared` with at least two distinct verified internal subject references, or `Unassigned` with no verified subject references and protected source-record lineage. Mixed-tenant associations cannot be persisted. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. +Mapping records contain encrypted original value, token digest/identifier, tenant, validated ownership state and its required subject-reference/source-lineage shape, purpose, PII type, key version, creation/expiry, and concurrency version. The engine rejects a mapping unless every subject reference resolves within the mapping's authenticated tenant and authorized subject namespace and the mapping has one valid shape: `Verified` with exactly one verified internal subject reference, `Shared` with at least two distinct verified internal subject references, or `Unassigned` with no verified subject references and protected source-record lineage. Mixed-tenant associations cannot be persisted. Before token derivation or idempotency comparison, a `Shared` subject set is de-duplicated, sorted by the tenant-scoped canonical subject-ID byte ordering, and encoded with a versioned length-prefixed serialization; equivalent input orders therefore produce the same ownership scope. Mapping storage is isolated from general application data. The token itself is treated as sensitive metadata. The request-level `DataSubjectId` is not copied into every mapping unless a trusted association proves ownership. For reversible tokenization, mapping writes and result publication form one idempotent operation: retrying the same scoped key/content/policy tuple returns the same committed mapping, a conflicting tuple fails, and tokenized output is not returned until all required mappings commit. Partial writes are rolled back or remain unreachable and are reconciled by a metadata-only recovery job. @@ -95,7 +95,7 @@ The current `RequestedBy` and free-form `Reason` are audit attributes, not proof ### 3.4 Data-subject deletion -`DeleteTokensForSubjectAsync` acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, the verified workflow removes the subject association and re-tokenizes or deletes only when the remaining subjects' rights and retention policy permit it. Because that changes ownership scope and token identity, the migration is atomic from the caller's perspective: create the replacement mapping, update every registered authorized reference and persisted or provider copy of the tokenized content, verify each update, and only then retire the old mapping. If any required surface fails, keep the old mapping reachable for the remaining subjects, roll back or quarantine partial replacement references, and fail closed for the deletion request. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. +`DeleteTokensForSubjectAsync` acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, a verified multi-subject data-rights decision must authorize removal of the subject association before the workflow re-tokenizes or deletes the mapping, subject to the remaining subjects' rights and retention policy. Because that changes ownership scope and token identity, the migration is atomic from the caller's perspective: create the replacement mapping, update every registered authorized reference and persisted or provider copy of the tokenized content, verify each update, and only then retire the old mapping. If any required surface fails, keep the old mapping reachable for the remaining subjects, roll back or quarantine partial replacement references, and fail closed for the deletion request. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. ## 4. Privacy Boundary Integration @@ -227,6 +227,7 @@ Audit read paths require explicit scope and return deny-by-default. Retention is - invalid/overlapping detector spans normalize deterministically; - equal-length, equal-confidence overlap fixtures choose the type earlier in immutable `TypePrecedence`; - invalid `Verified`, `Shared`, or `Unassigned` subject-reference/source-lineage combinations are rejected before mapping publication or deletion decisions; +- equivalent `Shared` subject sets in different input orders produce the same canonical ownership scope, token identity, and idempotency comparison; - deterministic chunking preserves cross-boundary entities and enforces size/entity limits; - reversible tokenization never returns tokens for uncommitted mappings and retries idempotently; - per-type thresholds and required detectors come from policy; From cf0d098c1147681f6388cf7367338d6b4e90d07c Mon Sep 17 00:00:00 2001 From: Jurie Smit Date: Sun, 23 Aug 2026 12:46:49 +0200 Subject: [PATCH 6/6] docs: authenticate and fence token deletion --- docs/spec/pii-tokenization-and-credential-sync.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/spec/pii-tokenization-and-credential-sync.md b/docs/spec/pii-tokenization-and-credential-sync.md index c8eb9302..24444328 100644 --- a/docs/spec/pii-tokenization-and-credential-sync.md +++ b/docs/spec/pii-tokenization-and-credential-sync.md @@ -95,7 +95,9 @@ The current `RequestedBy` and free-form `Reason` are audit attributes, not proof ### 3.4 Data-subject deletion -`DeleteTokensForSubjectAsync` acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, a verified multi-subject data-rights decision must authorize removal of the subject association before the workflow re-tokenizes or deletes the mapping, subject to the remaining subjects' rights and retention policy. Because that changes ownership scope and token identity, the migration is atomic from the caller's perspective: create the replacement mapping, update every registered authorized reference and persisted or provider copy of the tokenized content, verify each update, and only then retire the old mapping. If any required surface fails, keep the old mapping reachable for the remaining subjects, roll back or quarantine partial replacement references, and fail closed for the deletion request. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. +Add a versioned deletion request under `IPIITokenizationPort` that binds `TenantId`, target internal subject ID, authenticated `CallerIdentity`/`Scopes`, correlation and idempotency IDs, policy version, and expected mapping concurrency version. When a `Shared` mapping is affected, the request also carries the identifier and version of the verified multi-subject data-rights decision, which the engine resolves from its authoritative source rather than trusting request text. The existing string-based `DeleteTokensForSubjectAsync` overload remains available only to trusted compatibility/test hosts with equivalent context supplied out of band; it does not independently qualify a production deletion boundary. `reason` and `requestedBy` remain audit attributes, not authorization evidence. + +The deletion operation acts only on the validated ownership state: it deletes or cryptographically erases `Verified` mappings exclusively associated with the authenticated tenant/subject and returns metadata-only counts. For `Shared` mappings, a verified multi-subject data-rights decision must authorize removal of the subject association before the workflow re-tokenizes or deletes the mapping, subject to the remaining subjects' rights and retention policy. Because that changes ownership scope and token identity, the migration is atomic from the caller's perspective. The engine first acquires a compare-and-swap migration fence using the expected concurrency version; the durable migration state prevents concurrent tokenization, detokenization, deletion, or retries from reading or republishing the retiring mapping. While holding the fence, it creates the replacement mapping, updates every registered authorized reference and persisted or provider copy of the tokenized content, verifies each update, and only then retires the old mapping. A successful retirement leaves a versioned tombstone for the idempotency/retry retention period. If any required surface fails, the engine rolls back or quarantines partial replacement references, restores the old mapping under the same fenced state transition for the remaining subjects, and fails closed for the deletion request. `Unassigned` mappings are reconciled through protected source-record lineage and cannot be deleted merely because the requester submitted the document. A mapping with an invalid state/reference/lineage combination is quarantined for authorized reconciliation rather than used for a deletion decision. The data-rights orchestrator must separately reconcile authorized sources of record, caches, backups, external provider retention, and audit-policy exceptions. Destroying a mapping is not proof that every copy of the original was erased. ## 4. Privacy Boundary Integration @@ -228,6 +230,8 @@ Audit read paths require explicit scope and return deny-by-default. Retention is - equal-length, equal-confidence overlap fixtures choose the type earlier in immutable `TypePrecedence`; - invalid `Verified`, `Shared`, or `Unassigned` subject-reference/source-lineage combinations are rejected before mapping publication or deletion decisions; - equivalent `Shared` subject sets in different input orders produce the same canonical ownership scope, token identity, and idempotency comparison; +- legacy deletion calls cannot qualify without trusted host-supplied tenant, caller/scope, policy, and decision context; +- concurrent Shared deletion and retry fixtures honor the migration fence, never republish a retired mapping, and resume or fail idempotently; - deterministic chunking preserves cross-boundary entities and enforces size/entity limits; - reversible tokenization never returns tokens for uncommitted mappings and retries idempotently; - per-type thresholds and required detectors come from policy;