Skip to content

fix(chk): let healthy keeper reconciles complete - #2059

Open
KyriosGN0 wants to merge 1 commit into
Altinity:0.27.3from
KyriosGN0:agent/fix-chk-inprogress
Open

fix(chk): let healthy keeper reconciles complete#2059
KyriosGN0 wants to merge 1 commit into
Altinity:0.27.3from
KyriosGN0:agent/fix-chk-inprogress

Conversation

@KyriosGN0

Copy link
Copy Markdown

What changed

  • remove the unconditional 10-second wait from same-size CHK reconciles while preserving Raft settle waits for scale-up and scale-down
  • propagate reconcile-start and completion status update errors to controller-runtime
  • emit completion events and metrics only after Completed is persisted
  • add unit coverage for settle-delay selection and status persistence failures
  • add a three-node CHK regression that verifies status and task histories remain stable beyond the former reconcile window

Root cause

The CHK reconciler waited 10 seconds even when Keeper membership had not changed. More importantly, status persistence errors from both reconcile start and finalization were discarded. A failed final status write could therefore leave the resource at InProgress with taskIDsStarted growing and no taskIDsCompleted, while the controller still reported successful reconciliation.

Impact

Healthy Keeper installations can converge to and remain at Completed. Transient status update failures are returned to controller-runtime for retry instead of being silently treated as success. No CRD or public configuration changes are required.

Fixes #2035

Validation

  • go test -count=1 -vet=off -race ./pkg/controller/chk/... ./pkg/controller/common/statefulset/... ./pkg/apis/clickhouse-keeper.altinity.com/v1/...
  • go test -vet=off ./pkg/controller/...
  • python3 -m py_compile tests/e2e/test_operator.py
  • git diff --check upstream/0.27.3...HEAD

The cluster-dependent e2e scenario was added but not run locally. Standard Go 1.26 vet remains affected by pre-existing dynamic logging format warnings.

Signed-off-by: AvivGuiser <avivguiser@gmail.com>
@KyriosGN0
KyriosGN0 marked this pull request as ready for review August 7, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant