Document Doco-CD deployment hardening, rollback, and operations runbook

[wax911]

Background

local-stack is moving toward Doco-CD-managed deployment from dev, using stackctl.sh as the canonical Swarm deployment interface. The hosted machine is currently a single-node Swarm, but the deployment model should be reusable for other projects and future hosts.

This issue captures the operational hardening, rollback, and runbook work required before automatic patch/minor dependency deployments should be trusted.

Goals

• Document a production-safe Doco-CD deployment runbook for local-stack.
• Clearly explain the privilege boundary around Docker socket access.
• Define rollback procedures for bad merges and bad image updates.
• Define backup expectations for stateful services before major updates.
• Define operational checks for Doco-CD polling and webhook modes.
• Define verification steps for stackctl and SOPS-based deployments.
• Make the process reusable for other projects.

Non-goals

• Do not implement the OpenCode gate here.
• Do not implement the Doco-CD deploy runner here.
• Do not change branch protection here.
• Do not replace stackctl.sh.

Required documentation updates

Update or add documentation in:

README.md
stacks/README.md
docs/Managing Secrets.md
doco-cd/README.md
deploy/doco/local-stack-deployer/README.md

Required runbook sections

1. Deployment model overview

Document the final desired flow:

Dependabot PR
  -> OpenCode risk gate
  -> CI validation
  -> auto-merge for patch/minor only
  -> merge to dev
  -> Doco-CD polling or webhook trigger
  -> Doco-CD deploys local-stack-deployer
  -> local-stack-deployer runs stackctl
  -> stackctl deploys Swarm stacks

Make it explicit that Doco-CD must not directly deploy unresolved stacks/*.yml.

2. Branch and promotion policy

Document:

• dev is the deployment branch.
• Patch/minor dependency updates may auto-deploy after merge.
• Major updates require manual promotion.
• Major stateful updates require backup and rollback planning.

3. Doco-CD trigger modes

Document both supported trigger modes.

Polling:

• Preferred initial mode.
• No inbound public endpoint required.
• Recommended interval: 5-15 minutes.
• Tracks dev only.

Webhook:

• Faster deployment.
• Requires Doco-CD webhook secret.
• Must filter to refs/heads/dev.
• Should be exposed only through a controlled route, private network, or tunnel.
• Must not accept arbitrary commands from webhook payloads.

4. Docker socket risk

Document bluntly:

• Any component with Docker socket access has high host impact.
• Doco-CD and the deploy runner must be treated as privileged infrastructure.
• This is accepted for this project, but must be minimized.
• Do not expose Doco-CD publicly except the minimal webhook path if needed.
• Use pinned images.
• Keep secrets mounted read-only where possible.

5. SOPS + age deployment model

Document:

• .env.enc is committed.
• .env is plaintext and must not be committed.
• stackctl.sh secrets deploy decrypts, renders, deploys, and cleans plaintext.
• The age key path mounted into the runner is read-only.
• Plaintext .env cleanup must be verified after deployment.

Expected verification:

find . -name .env -print

This should return no plaintext env files that correspond to encrypted .env.enc files after SOPS deploy mode completes.

6. Deployment verification commands

Document commands:

./stackctl.sh doctor --fix-network
./stackctl.sh sync
./stackctl.sh up --dry-run --no-logs
./stackctl.sh secrets deploy
./stackctl.sh status

docker stack services infrastructure
docker stack services observability
docker stack services platform

7. Rollback procedures

Document two rollback paths.

Git rollback:

git revert <bad_merge_commit>
git push origin dev

Expected result: Doco-CD detects the revert on dev and redeploys.

Manual host rollback:

cd /path/to/local-stack
git fetch --prune origin dev
git reset --hard <known_good_commit>
./stackctl.sh secrets deploy

Warn that image rollback is not the same as data rollback. Stateful services may migrate data formats or persistent volumes.

8. Stateful service warning list

Document that these services require extra review for major updates:

postgres
mongo
redis
apisix etcd
grafana
prometheus
loki
tempo
portainer
growthbook if schema/database migrations are involved

9. Backup expectations

Before manually promoting major updates, require:

• Relevant volume backup exists.
• Current image tag is known.
• Target image release notes have been reviewed.
• Rollback image tag is known.
• Service-specific migration notes are captured in the PR or issue.

10. Incident handling

Document:

• How to pause Doco-CD polling/webhooks.
• How to disable the webhook at GitHub side.
• How to stop Doco-CD.
• How to inspect runner logs.
• How to inspect Swarm service logs.
• How to manually redeploy using stackctl.

Example commands:

docker logs doco-cd
docker service ls
docker service ps <service>
docker service logs <stack>_<service>
./stackctl.sh logs infrastructure_traefik observability_prometheus observability_loki

11. Reuse pattern for other projects

Document a project-agnostic pattern:

Git branch as source of truth
Doco-CD polling/webhook trigger
Project-specific deploy runner
No arbitrary webhook commands
Pinned controller image
Read-only secret mounts
Docker socket treated as privileged
Schema-validated AI gate before auto-merge

Acceptance criteria

• README explains Doco-CD deployment at a high level.
• stacks/README.md warns not to deploy unresolved generated stacks directly.
• docs/Managing Secrets.md includes Doco-CD runner notes.
• doco-cd/README.md documents bootstrap, polling, webhook, secrets, and exposure model.
• Deploy runner README documents environment variables and modes.
• Rollback procedures are documented.
• Stateful major update risks are documented.
• Backup expectations are documented.
• Incident handling steps are documented.
• The runbook explicitly states that patch/minor can auto-deploy after merge, but major updates require manual promotion.

Test plan

• Follow the runbook on a non-production single-node Swarm.
• Confirm a dry-run deployment path works.
• Confirm a SOPS deployment path works.
• Confirm rollback via git revert works.
• Confirm manual rollback to a known-good commit works.
• Confirm Doco-CD can be paused/stopped without affecting already-running Swarm services.
• Confirm webhook mode can be disabled without breaking polling mode.

[wax911]

Closed: Documentation runbook complete. README.md, stacks/README.md, and docs/Managing Secrets.md updated with Doco-CD deployment overview. doco-cd/ and deploy/doco/local-stack-deployer/ READMEs cover env vars, modes, rollback, security, and SOPS key setup. Ref: 6a5538f