Self-hosted Openship 0.7.2. Two related findings about how service-level environment is stored and applied. The first cost us a botched credential rotation that reported success; the second is a hardening point.
1. service.environment silently overrides the project env store, on every deploy path
A key present in the environment jsonb on a service row is applied at container creation and wins over the project-level env store. openship project env set on such a key:
- stores the new value correctly,
- reports success,
- is carried into the deployment snapshot,
…and the container still comes up with the old service-level value. No warning anywhere.
Keys absent from service.environment follow the project store normally, for both upserts and deletions — which is exactly what makes this hard to diagnose. Half your changes land and half do not, with no signal distinguishing them.
Every deploy path we tried is affected: deploy --refresh, deployment redeploy, and a full git rebuild producing a brand-new image. All three reported success and changed nothing.
Why it is worse than a normal precedence rule
Precedence between layers is fine and expected. The problem is that the losing layer is the one the CLI writes to and reports on, so the tool tells you the change landed. We hit this rotating an OAuth signing key: the CLI said the key was updated, the deploy went ready, and the application kept signing with the old key. The only way to find out was docker exec <container> printenv <KEY>.
Reproduce
# 1. Pin a key at the service layer
openship api "/projects/<projId>/services/<svcId>" -X PATCH -d '{"environment":{"FOO":"from-service"}}'
# 2. Change it at the project layer — reports success
openship project env set <projId> --set FOO=from-project
# 3. Deploy by any path, then look at what the container actually got
docker exec <container> printenv FOO # -> from-service
Suggested fixes, roughly in order of value
- Warn on the write. When
project env set targets a key that is pinned in some service.environment in that project, say so on stdout and name the service. This alone would have saved the incident.
- Surface it in the deployment output, listing keys whose effective value came from the service layer rather than the project store.
- Expose the effective environment — an
openship project env get --effective (or --resolved) that shows, per key, the value that will actually reach the container and which layer it came from.
- Document the precedence in the env docs. It is currently discoverable only by querying the database.
The supported workaround, once you know, is to write the service layer directly (merge semantics; omit a key to keep it, null to remove) and then update the project layer so the two agree:
openship api "/projects/<projId>/services/<svcId>" -X PATCH -d '{"environment":{"KEY":"value"}}'
2. service.environment and service.buildArgs store secrets in plaintext
The same jsonb columns hold values in the clear, while the same value in the project env store is encrypted at rest. So a secret that is encrypted before it gets pinned is unencrypted after. Two specific observations:
- Anyone with read access to the Openship database reads every pinned secret directly, with no key material required.
PATCH /projects/<projId>/services/<svcId> echoes buildArgs back in the response body in plaintext, so a routine environment edit prints unrelated build-time secrets to the operator's terminal (and into any log or transcript capturing it).
Encrypting these columns the way env_var is already encrypted, and masking them in API responses the way environment values already are, would close both. The response masking is the cheaper half and worth doing on its own.
Happy to test a fix against a self-hosted instance.
Self-hosted Openship 0.7.2. Two related findings about how service-level environment is stored and applied. The first cost us a botched credential rotation that reported success; the second is a hardening point.
1.
service.environmentsilently overrides the project env store, on every deploy pathA key present in the
environmentjsonb on aservicerow is applied at container creation and wins over the project-level env store.openship project env seton such a key:…and the container still comes up with the old service-level value. No warning anywhere.
Keys absent from
service.environmentfollow the project store normally, for both upserts and deletions — which is exactly what makes this hard to diagnose. Half your changes land and half do not, with no signal distinguishing them.Every deploy path we tried is affected:
deploy --refresh,deployment redeploy, and a full git rebuild producing a brand-new image. All three reported success and changed nothing.Why it is worse than a normal precedence rule
Precedence between layers is fine and expected. The problem is that the losing layer is the one the CLI writes to and reports on, so the tool tells you the change landed. We hit this rotating an OAuth signing key: the CLI said the key was updated, the deploy went
ready, and the application kept signing with the old key. The only way to find out wasdocker exec <container> printenv <KEY>.Reproduce
Suggested fixes, roughly in order of value
project env settargets a key that is pinned in someservice.environmentin that project, say so on stdout and name the service. This alone would have saved the incident.openship project env get --effective(or--resolved) that shows, per key, the value that will actually reach the container and which layer it came from.The supported workaround, once you know, is to write the service layer directly (merge semantics; omit a key to keep it,
nullto remove) and then update the project layer so the two agree:2.
service.environmentandservice.buildArgsstore secrets in plaintextThe same jsonb columns hold values in the clear, while the same value in the project env store is encrypted at rest. So a secret that is encrypted before it gets pinned is unencrypted after. Two specific observations:
PATCH /projects/<projId>/services/<svcId>echoesbuildArgsback in the response body in plaintext, so a routine environment edit prints unrelated build-time secrets to the operator's terminal (and into any log or transcript capturing it).Encrypting these columns the way
env_varis already encrypted, and masking them in API responses the wayenvironmentvalues already are, would close both. The response masking is the cheaper half and worth doing on its own.Happy to test a fix against a self-hosted instance.