Skip to content

[Bug] service.environment silently overrides project env on every deploy path (and stores secrets in plaintext) #844

Description

@greerso

Self-hosted Openship 0.7.2. Two related findings about how service-level environment is stored and applied. The first cost us a botched credential rotation that reported success; the second is a hardening point.

1. service.environment silently overrides the project env store, on every deploy path

A key present in the environment jsonb on a service row is applied at container creation and wins over the project-level env store. openship project env set on such a key:

  • stores the new value correctly,
  • reports success,
  • is carried into the deployment snapshot,

…and the container still comes up with the old service-level value. No warning anywhere.

Keys absent from service.environment follow the project store normally, for both upserts and deletions — which is exactly what makes this hard to diagnose. Half your changes land and half do not, with no signal distinguishing them.

Every deploy path we tried is affected: deploy --refresh, deployment redeploy, and a full git rebuild producing a brand-new image. All three reported success and changed nothing.

Why it is worse than a normal precedence rule

Precedence between layers is fine and expected. The problem is that the losing layer is the one the CLI writes to and reports on, so the tool tells you the change landed. We hit this rotating an OAuth signing key: the CLI said the key was updated, the deploy went ready, and the application kept signing with the old key. The only way to find out was docker exec <container> printenv <KEY>.

Reproduce

# 1. Pin a key at the service layer
openship api "/projects/<projId>/services/<svcId>" -X PATCH -d '{"environment":{"FOO":"from-service"}}'

# 2. Change it at the project layer — reports success
openship project env set <projId> --set FOO=from-project

# 3. Deploy by any path, then look at what the container actually got
docker exec <container> printenv FOO      # -> from-service

Suggested fixes, roughly in order of value

  1. Warn on the write. When project env set targets a key that is pinned in some service.environment in that project, say so on stdout and name the service. This alone would have saved the incident.
  2. Surface it in the deployment output, listing keys whose effective value came from the service layer rather than the project store.
  3. Expose the effective environment — an openship project env get --effective (or --resolved) that shows, per key, the value that will actually reach the container and which layer it came from.
  4. Document the precedence in the env docs. It is currently discoverable only by querying the database.

The supported workaround, once you know, is to write the service layer directly (merge semantics; omit a key to keep it, null to remove) and then update the project layer so the two agree:

openship api "/projects/<projId>/services/<svcId>" -X PATCH -d '{"environment":{"KEY":"value"}}'

2. service.environment and service.buildArgs store secrets in plaintext

The same jsonb columns hold values in the clear, while the same value in the project env store is encrypted at rest. So a secret that is encrypted before it gets pinned is unencrypted after. Two specific observations:

  • Anyone with read access to the Openship database reads every pinned secret directly, with no key material required.
  • PATCH /projects/<projId>/services/<svcId> echoes buildArgs back in the response body in plaintext, so a routine environment edit prints unrelated build-time secrets to the operator's terminal (and into any log or transcript capturing it).

Encrypting these columns the way env_var is already encrypted, and masking them in API responses the way environment values already are, would close both. The response masking is the cheaper half and worth doing on its own.

Happy to test a fix against a self-hosted instance.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions