Skip to content

Commit c870d22

Browse files
authored
refactor(solmara): pair each Relay with PostgreSQL-backed Notary (#9)
Replace the shared Redis-backed Notary topology with one PostgreSQL-backed Notary per Relay authority, pin Registry Stack v0.10.0, preserve restart semantics, and make hosted deployment closures self-contained. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
1 parent 00cd51a commit c870d22

372 files changed

Lines changed: 22333 additions & 10118 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 86 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -2,76 +2,100 @@
22

33
COMPOSE_PROJECT_NAME=solmara-lab-replace-me
44

5-
REGISTRY_RELAY_AUDIT_HASH_SECRET=
6-
REGISTRY_NOTARY_AUDIT_HASH_SECRET=
7-
REGISTRY_NOTARY_REPLAY_REDIS_URL=redis://redis:6379/0
5+
SOLMARA_RELAY_KEY_ACTIVE_WRITE_DEADLINE_UNIX_MS=4102444800000
6+
SOLMARA_RELAY_AUDIT_EVENT_RETENTION_MS=2592000000
87
REGISTRY_ESIGNET_KYC_KEYSTORE_PASSWORD=
98
REGISTRY_ESIGNET_KYC_TOKEN_SECRET=
109
REGISTRY_ESIGNET_PSUT_SECRET=
1110

1211
SOLMARA_POSTGRES_USER=solmara_registry
1312
SOLMARA_POSTGRES_PASSWORD=
1413
SOLMARA_POSTGRES_DB=solmara_lab
15-
SOLMARA_NIA_DATABASE_URL=postgres://solmara_registry:replace-me@postgres:5432/solmara_lab?sslmode=require
14+
NIA_SOURCE_POSTGRES_READER_PASSWORD=
15+
SIPF_SOURCE_POSTGRES_READER_PASSWORD=
16+
SOLMARA_NIA_DATABASE_URL=postgres://solmara_source_nia_reader:replace-me@postgres:5432/solmara_lab?sslmode=require
17+
SOLMARA_SIPF_DATABASE_URL=postgres://solmara_source_sipf_reader:replace-me@postgres:5432/solmara_lab?sslmode=require
1618
SOLMARA_ESIGNET_POSTGRES_PASSWORD=
1719

18-
CRA_CHILD_BENEFIT_SOURCE_RAW=
19-
CRA_CHILD_BENEFIT_SOURCE_HASH=
20-
CRA_PENSION_SOURCE_RAW=
21-
CRA_PENSION_SOURCE_HASH=
22-
CRA_CITIZEN_SOURCE_RAW=
23-
CRA_CITIZEN_SOURCE_HASH=
20+
CRA_RELAY_AUDIT_HASH_SECRET=
21+
CRA_RELAY_AUDIT_PSEUDONYM_EPOCH_1=
22+
CRA_RELAY_POSTGRES_RUNTIME_PASSWORD=
23+
CRA_RELAY_POSTGRES_KEYRING_MAINTENANCE_PASSWORD=
24+
CRA_RELAY_POSTGRES_KEYRING_READER_PASSWORD=
25+
CRA_NOTARY_AUDIT_HASH_SECRET=
26+
CRA_NOTARY_POSTGRES_MIGRATOR_PASSWORD=
27+
CRA_NOTARY_POSTGRES_RUNTIME_PASSWORD=
28+
NIA_RELAY_AUDIT_HASH_SECRET=
29+
NIA_RELAY_AUDIT_PSEUDONYM_EPOCH_1=
30+
NIA_RELAY_POSTGRES_RUNTIME_PASSWORD=
31+
NIA_RELAY_POSTGRES_KEYRING_MAINTENANCE_PASSWORD=
32+
NIA_RELAY_POSTGRES_KEYRING_READER_PASSWORD=
33+
NIA_NOTARY_AUDIT_HASH_SECRET=
34+
NIA_NOTARY_POSTGRES_MIGRATOR_PASSWORD=
35+
NIA_NOTARY_POSTGRES_RUNTIME_PASSWORD=
36+
SRO_RELAY_AUDIT_HASH_SECRET=
37+
SRO_RELAY_AUDIT_PSEUDONYM_EPOCH_1=
38+
SRO_RELAY_POSTGRES_RUNTIME_PASSWORD=
39+
SRO_RELAY_POSTGRES_KEYRING_MAINTENANCE_PASSWORD=
40+
SRO_RELAY_POSTGRES_KEYRING_READER_PASSWORD=
41+
SRO_NOTARY_AUDIT_HASH_SECRET=
42+
SRO_NOTARY_POSTGRES_MIGRATOR_PASSWORD=
43+
SRO_NOTARY_POSTGRES_RUNTIME_PASSWORD=
44+
PROGRAMME_RELAY_AUDIT_HASH_SECRET=
45+
PROGRAMME_RELAY_AUDIT_PSEUDONYM_EPOCH_1=
46+
PROGRAMME_RELAY_POSTGRES_RUNTIME_PASSWORD=
47+
PROGRAMME_RELAY_POSTGRES_KEYRING_MAINTENANCE_PASSWORD=
48+
PROGRAMME_RELAY_POSTGRES_KEYRING_READER_PASSWORD=
49+
PROGRAMME_NOTARY_AUDIT_HASH_SECRET=
50+
PROGRAMME_NOTARY_POSTGRES_MIGRATOR_PASSWORD=
51+
PROGRAMME_NOTARY_POSTGRES_RUNTIME_PASSWORD=
52+
SIPF_RELAY_AUDIT_HASH_SECRET=
53+
SIPF_RELAY_AUDIT_PSEUDONYM_EPOCH_1=
54+
SIPF_RELAY_POSTGRES_RUNTIME_PASSWORD=
55+
SIPF_RELAY_POSTGRES_KEYRING_MAINTENANCE_PASSWORD=
56+
SIPF_RELAY_POSTGRES_KEYRING_READER_PASSWORD=
57+
SIPF_NOTARY_AUDIT_HASH_SECRET=
58+
SIPF_NOTARY_POSTGRES_MIGRATOR_PASSWORD=
59+
SIPF_NOTARY_POSTGRES_RUNTIME_PASSWORD=
60+
NAGDI_RELAY_AUDIT_HASH_SECRET=
61+
NAGDI_RELAY_AUDIT_PSEUDONYM_EPOCH_1=
62+
NAGDI_RELAY_POSTGRES_RUNTIME_PASSWORD=
63+
NAGDI_RELAY_POSTGRES_KEYRING_MAINTENANCE_PASSWORD=
64+
NAGDI_RELAY_POSTGRES_KEYRING_READER_PASSWORD=
65+
NAGDI_NOTARY_AUDIT_HASH_SECRET=
66+
NAGDI_NOTARY_POSTGRES_MIGRATOR_PASSWORD=
67+
NAGDI_NOTARY_POSTGRES_RUNTIME_PASSWORD=
2468

25-
NIA_CHILD_BENEFIT_SOURCE_RAW=
26-
NIA_CHILD_BENEFIT_SOURCE_HASH=
27-
NIA_PENSION_SOURCE_RAW=
28-
NIA_PENSION_SOURCE_HASH=
29-
NIA_CITIZEN_SOURCE_RAW=
30-
NIA_CITIZEN_SOURCE_HASH=
31-
SOLMARA_ESIGNET_IDENTITY_RELEASE_RAW=
32-
SOLMARA_ESIGNET_IDENTITY_RELEASE_HASH=
33-
34-
SRO_CHILD_BENEFIT_SOURCE_RAW=
35-
SRO_CHILD_BENEFIT_SOURCE_HASH=
36-
SRO_CITIZEN_SOURCE_RAW=
37-
SRO_CITIZEN_SOURCE_HASH=
38-
39-
PROGRAMME_CHILD_BENEFIT_SOURCE_RAW=
40-
PROGRAMME_CHILD_BENEFIT_SOURCE_HASH=
41-
PROGRAMME_PENSION_SOURCE_RAW=
42-
PROGRAMME_PENSION_SOURCE_HASH=
43-
PROGRAMME_CITIZEN_SOURCE_RAW=
44-
PROGRAMME_CITIZEN_SOURCE_HASH=
45-
46-
SIPF_PENSION_SOURCE_RAW=
47-
SIPF_PENSION_SOURCE_HASH=
48-
SIPF_CITIZEN_SOURCE_RAW=
49-
SIPF_CITIZEN_SOURCE_HASH=
50-
51-
NAGDI_NOTARY_SOURCE_RAW=
52-
NAGDI_NOTARY_SOURCE_HASH=
53-
NAGDI_CITIZEN_SOURCE_RAW=
54-
NAGDI_CITIZEN_SOURCE_HASH=
55-
56-
CHILD_BENEFIT_NOTARY_TOKEN=
57-
CHILD_BENEFIT_CLIENT_TOKEN_HASH=
58-
PENSION_NOTARY_TOKEN=
59-
PENSION_CLIENT_TOKEN_HASH=
69+
CHILD_BENEFIT_FEDERATOR_TOKEN=
70+
CRA_CHILD_BENEFIT_CLIENT_TOKEN=
71+
CRA_CHILD_BENEFIT_CLIENT_TOKEN_HASH=
72+
NIA_CHILD_BENEFIT_CLIENT_TOKEN=
73+
NIA_CHILD_BENEFIT_CLIENT_TOKEN_HASH=
74+
SRO_CHILD_BENEFIT_CLIENT_TOKEN=
75+
SRO_CHILD_BENEFIT_CLIENT_TOKEN_HASH=
76+
PROGRAMME_CHILD_BENEFIT_CLIENT_TOKEN=
77+
PROGRAMME_CHILD_BENEFIT_CLIENT_TOKEN_HASH=
78+
CRA_PENSION_CLIENT_TOKEN=
79+
CRA_PENSION_CLIENT_TOKEN_HASH=
80+
CRA_CITIZEN_CLIENT_TOKEN=
81+
CRA_CITIZEN_CLIENT_TOKEN_HASH=
82+
NIA_CITIZEN_CLIENT_TOKEN=
83+
NIA_CITIZEN_CLIENT_TOKEN_HASH=
84+
SIPF_PENSION_CLIENT_TOKEN=
85+
SIPF_PENSION_CLIENT_TOKEN_HASH=
6086
NAGDI_NOTARY_TOKEN=
6187
NAGDI_CLIENT_TOKEN_HASH=
62-
CITIZEN_PORTAL_BFF_TOKEN_HASH=
63-
PORTAL_CITIZEN_NOTARY_TOKEN=
64-
PORTAL_RELAY_TOKEN=
65-
PORTAL_RELAY_TOKEN_HASH=
6688

67-
CHILD_BENEFIT_NOTARY_ISSUER_JWK=
68-
PENSION_NOTARY_ISSUER_JWK=
89+
NIA_NOTARY_ISSUER_JWK=
90+
SIPF_NOTARY_ISSUER_JWK=
6991
NAGDI_NOTARY_ISSUER_JWK=
70-
CITIZEN_NOTARY_ISSUER_JWK=
71-
CITIZEN_ISSUER_NOTARY_ISSUER_JWK=
72-
CITIZEN_ISSUER_NOTARY_ACCESS_TOKEN_JWK=
73-
CITIZEN_ISSUER_ESIGNET_RP_JWK=
74-
CITIZEN_ISSUER_ESIGNET_CLIENT_PRIVATE_KEY_B64=
92+
CRA_RELAY_WORKLOAD_JWK=
93+
NIA_RELAY_WORKLOAD_JWK=
94+
NIA_ESIGNET_RELAY_WORKLOAD_JWK=
95+
SRO_RELAY_WORKLOAD_JWK=
96+
PROGRAMME_RELAY_WORKLOAD_JWK=
97+
SIPF_RELAY_WORKLOAD_JWK=
98+
NAGDI_RELAY_WORKLOAD_JWK=
7599
PORTAL_SESSION_SECRET=
76100
PORTAL_AUTH_PROVIDER=mock
77101
PORTAL_ESIGNET_CLIENT_ID=solmara-portal
@@ -86,15 +110,13 @@ PORTAL_ESIGNET_REDIRECT_URI=http://127.0.0.1:4300/auth/callback
86110
PORTAL_ESIGNET_SCOPE=openid profile
87111
PORTAL_ESIGNET_SUBJECT_CLAIM=individual_id
88112

89-
CHILD_BENEFIT_NOTARY_URL=http://127.0.0.1:4321
90-
PENSION_NOTARY_URL=http://127.0.0.1:4322
113+
CHILD_BENEFIT_FEDERATOR_URL=http://127.0.0.1:4321
114+
CRA_NOTARY_URL=http://127.0.0.1:4325
115+
NIA_NOTARY_URL=http://127.0.0.1:4326
116+
SRO_NOTARY_URL=http://127.0.0.1:4327
117+
PROGRAMME_NOTARY_URL=http://127.0.0.1:4328
118+
SIPF_NOTARY_URL=http://127.0.0.1:4322
91119
NAGDI_NOTARY_URL=http://127.0.0.1:4323
92-
PORTAL_CITIZEN_NOTARY_URL=http://127.0.0.1:4324
93-
SOLMARA_CITIZEN_ISSUER_NOTARY_PORT=4325
94-
PORTAL_CIVIL_RELAY_URL=http://127.0.0.1:4311
95-
PORTAL_SOCIAL_RELAY_URL=http://127.0.0.1:4313
96-
PORTAL_AGRI_RELAY_URL=http://127.0.0.1:4316
97-
PORTAL_CERTS_RELAY_URL=http://127.0.0.1:4311
98120
PORTAL_URL=http://127.0.0.1:4300
99121
SCENARIO_RUNNER_URL=http://127.0.0.1:4302
100122
STATIC_METADATA_URL=http://127.0.0.1:4331

‎.github/workflows/release-candidate.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,13 +62,13 @@ jobs:
6262
just metadata-publish
6363
- name: Validate
6464
run: |
65-
just release-pins v0.8.4
65+
just release-pins v0.10.0
6666
just lint
6767
just test
6868
just compose
6969
- name: Start stack
7070
run: just up
71-
- name: Smoke
71+
- name: Smoke and PostgreSQL restart persistence
7272
run: just smoke
7373
- name: Portal live e2e
7474
run: just portal-live-e2e

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ __pycache__/
2222
.ruff_cache/
2323
.mypy_cache/
2424
dist/
25+
projects/*/.registry-stack/
2526
*.log
2627
config/postgres/ssl/*
2728
!config/postgres/ssl/.gitkeep

‎README.md‎

Lines changed: 34 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,10 @@ The first wave covers three journeys:
4141
generated fixture checks.
4242
- `geo/` contains the hand-authored Solmara geometry source used by the
4343
generator.
44-
- `ministries/` contains authority-owned Relay configs, manifest fragments,
45-
crosswalks, and generated fixtures.
46-
- `notaries/` contains purpose-oriented Notary configs.
44+
- `ministries/` contains authority-owned source fixtures, manifest fragments,
45+
and crosswalks.
46+
- `projects/` contains the six authority-owned Registry project sources. Each
47+
project generates one Relay config and one Notary config under `runtime/`.
4748
- `metadata/` assembles the multi-authority metadata publication.
4849
- `portal/` contains the citizen portal and BFF.
4950
- `scenarios/`, `requests/`, and `perf/` carry guided scenarios, API examples,
@@ -62,10 +63,10 @@ just portal-compose-smoke # HTTP smoke against the Compose portal and live BFF
6263
just portal-live-e2e # browser e2e against the running local stack
6364
just hosted-smoke # public hosted health, endpoint, scenario, and portal checks
6465
just up-esignet # local stack with eSignet-backed portal login
65-
just smoke-esignet # eSignet discovery plus NIA UserInfo release smoke
66+
just smoke-esignet # eSignet public discovery smoke
6667
just down # stop the local Compose topology without deleting volumes
6768
just reset # stop the local Compose topology and delete its volumes
68-
just release-pins v0.8.4 # compare versions.env against published GHCR tags
69+
just release-pins v0.10.0 # compare versions.env against published GHCR tags
6970
just review # security and release-readiness checks
7071
```
7172

@@ -76,43 +77,48 @@ committed generated artifact.
7677

7778
`versions.env` is the root source for published image digests. The Compose
7879
fallbacks mirror it so direct `docker compose` runs still use pinned Registry
79-
Stack images. The current pins are the Registry Stack `v0.8.4` release digest
80+
Stack images. The current pins are the Registry Stack `v0.10.0` release digest
8081
assets. Because that release publishes amd64 images, Compose defaults
8182
`REGISTRY_STACK_PLATFORM` to `linux/amd64`; override it only when the release
8283
publishes an image for another platform.
8384

84-
The NIA population Relay is intentionally Postgres-backed. `just gen-secrets`
85-
also creates local Postgres TLS material under `config/postgres/ssl/`, and the
86-
generated NIA connection string includes `sslmode=require`, matching Registry
87-
Stack `v0.8.4` Relay requirements.
85+
Every authority runs one Relay and one Notary. Relay consultation state and all
86+
Notary correctness state are PostgreSQL-backed. `just gen-secrets` creates
87+
local PostgreSQL TLS material and distinct runtime and migrator passwords for
88+
each authority. See
89+
[`docs/notary-postgresql-state.md`](docs/notary-postgresql-state.md) for the
90+
database map, diagnosis, backup, recovery, and upgrade workflow.
8891

8992
## Hosted Deployment
9093

9194
See [`docs/hosted-deployment.md`](docs/hosted-deployment.md) for the full
92-
runbook. Coolify uses one hosted compose file for the lab edge plus one compose
93-
file per pseudo-government authority:
95+
runbook. Coolify uses one hosted Compose file for the lab edge plus four
96+
ministry-grouped authority applications:
9497

95-
- `compose.coolify.yaml` for the Visitor Center, portal, scenario runner, and
96-
static metadata.
97-
- `compose.coolify.interior.yaml` for CRA, NIA, and the NIA Postgres store.
98+
- `compose.coolify.yaml` for the Visitor Center, portal, scenario runner,
99+
child-benefit evidence composition, and static metadata.
100+
- `compose.coolify.interior.yaml` for the CRA and NIA Relay and Notary pairs
101+
and their PostgreSQL databases.
98102
- `compose.coolify.esignet.yaml` for eSignet, eSignet UI, and its backing
99103
Postgres/Redis/seed services.
100-
- `compose.coolify.social-development.yaml` for SRO, MoSD programme MIS, and
101-
the child benefit notary.
102-
- `compose.coolify.labour-pensions.yaml` for SIPF and the pension notary.
103-
- `compose.coolify.agriculture.yaml` for NAgDI and the agriculture notary.
104-
- `compose.coolify.citizen-services.yaml` for the citizen services notary.
104+
- `compose.coolify.social-development.yaml` for the SRO and Programme Relay
105+
and Notary pairs and their PostgreSQL databases.
106+
- `compose.coolify.labour-pensions.yaml` for the SIPF Relay and Notary pair and
107+
its PostgreSQL databases.
108+
- `compose.coolify.agriculture.yaml` for the NAgDI Relay and Notary pair and
109+
its PostgreSQL databases.
105110

106111
The hosted compose files remove host port bindings and avoid repo bind mounts
107112
because Coolify does not seed bind-mount sources from the Git checkout. They do
108113
not define custom Docker networks; cross-authority calls use the public
109114
`*.solmara.registrystack.org` TLS endpoints. Authority compose files preserve
110-
Relay audit/cache volumes and Notary audit state volumes, with a small
111-
`volume-permissions` sidecar that makes those volumes writable by the product
112-
runtime users.
115+
authority-owned PostgreSQL state, persistent Relay snapshot caches, and
116+
workload credentials. Notary containers do not use Redis or a writable state
117+
directory.
113118

114-
Run `uv run scripts/render-hosted-configs.py` after editing Relay or Notary
115-
configs so the hosted config overlays stay current.
119+
Run `just registry-projects-sync` after editing an authority project, then
120+
`just registry-projects-runtime-check` to verify the local and hosted Relay and
121+
Notary closures are deterministic.
116122

117123
Run `just hosted-smoke` after each hosted deploy from a trusted shell with the
118124
demo tokens available in `.env` or the process environment. It checks public
@@ -132,8 +138,9 @@ workflow summary for Coolify env vars:
132138

133139
For local eSignet testing, run `just up-esignet` instead of `just up`, then
134140
sign in through the portal with Elena's fixture `legacy_nid` and static OTP
135-
`111111`. Run `just smoke-esignet` to verify eSignet discovery and the NIA
136-
`solmara-nia-userinfo` attribute-release profile.
141+
`111111`. This sign-in is the end-to-end check of the NIA
142+
`solmara-nia-userinfo` attribute-release profile and its rotating eSignet
143+
workload identity. Run `just smoke-esignet` for the public discovery checks.
137144

138145
Set `UMAMI_WEBSITE_ID` in the hosted environment to enable analytics for the
139146
Visitor Center through the Registry Stack Umami instance.

0 commit comments

Comments
 (0)