0.27.2 - #2043
Merged
Merged
Conversation
Pre-fix, a ClickHouse pod that regressed to Ready=False while its CHI
was Status=Completed could stay stuck indefinitely: the CHI generation
never changes, the operator's ReconcileUpdate path had no handler for
Ready→NotReady on Completed CHIs, and shouldForceRestartHost had no
case for sustained Ready=False. Field observation: 26-hour stalls of
otherwise-healthy clusters until an external trigger (operator pod
restart) bumped the reconcile loop.
Close both gaps:
Signal side - worker-pod-retry.go adds
recoverCompletedReconcileOnPodNotReady, wired into processReconcilePod
alongside the existing recoverAbortedReconcileOnPodReady. Schedules a
delayed re-enqueue (time.AfterFunc) so the decision-side gets to
evaluate sustain rather than fire on every flap.
Decision side - worker.go adds a new case to shouldForceRestartHost
gated on ShouldRecoverCompletedOnPodNotReady() and a freshly-fetched
isPoustainedNotReady() check that reads
Pod.Status.Conditions[PodReady].LastTransitionTime - the same signal
kube-proxy uses for EndpointSlice membership.
Config: reconcile.recovery.from.completed.{onPodNotReady,
onPodNotReadyThreshold}, default retry / 5m. Default-on; the cost of
not acting by default is multi-hour outage, the cost of acting is at
most one StatefulSet restart per host per threshold window.
Observability: two new EventReasons (StuckHostRecoveryTriggered,
HostStuckNotReady), split so a debounced flap leaves only the schedule
event behind while a real outage leaves both. No new Prometheus
counters - events on the CHI provide the observability hook without
expanding the operator's /metrics surface.
Both signal and decision sides short-circuit when the pod is in a
kubelet-driven failure mode (ImagePullBackOff, CrashLoopBackOff,
Pending, etc.) so the operator does not race kubelet on its own
recovery. The signal side also skips pods with a non-zero
DeletionTimestamp (graceful shutdown is not a regression).
Validated end-to-end on kind 1.30 with a 65% bidirectional iptables
drop on the pod's 8123/tcp: self-heal in ~10 min vs. indefinite stall
pre-fix. New unit tests for each new function (pure predicates and
accessors), all table-driven.
Signed-off-by: dashashutosh80 <dashashutosh80@gmail.com>
The configs.files.config.yaml.watch.namespaces.include field was hardcoded to [] in values.yaml and the ConfigMap template rendered it directly with no way to override it via a top-level Helm value. This patch introduces a top-level watchNamespaces value and a new configmap-files helper that deep-copies configs.files, patches the nested watch.namespaces.include in-place, then delegates to the existing configmap-data helper for rendering. Fixes #1919
The configs.files.config.yaml.watch.namespaces.include field was hardcoded to [] in values.yaml and the ConfigMap template rendered it directly with no way to override it via a top-level Helm value. This patch introduces a top-level watchNamespaces value and a new configmap-files helper that deep-copies configs.files, patches the nested watch.namespaces.include in-place, then delegates to the existing configmap-data helper for rendering. Fixes #1919
Address review: the previous change hand-edited the generated ConfigMap-etc-clickhouse-operator-files.yaml. Instead, special-case the etc-clickhouse-operator-files ConfigMap in update_configmap_resource() so generate_helm_chart.sh emits the configmap-files helper (which wires watchNamespaces into watch.namespaces.include). Re-running the generator now reproduces the committed template. README values table regenerated by helm-docs.
Add reconcileCHKsThreadsNumber config option to control the number of concurrent ClickHouseKeeperInstallation reconciliations. Previously hardcoded to 1 via controller-runtime defaults, this caused a significant bottleneck at scale when many CHK resources needed reconciliation simultaneously. The new setting mirrors the existing reconcileCHIsThreadsNumber for CHI resources. Defaults to 1 to preserve existing behavior. Closes #2032 Signed-off-by: miguel-signoz <miguel@signoz.io>
Integrates dentiny's PR #2028 (github.com//pull/2028). The change was already landed in 0.27.2 via our own commits plus follow-up fixes (nil-safe IsExcluded helper, unit test, manifest regen); this merge brings the PR's commit into the branch so it is recorded as merged. Conflicts resolved in favor of our code (-X ours).
Integrates jtomaszon's PR #2007 (github.com//pull/2007). Already landed in 0.27.2 via cherry-pick + follow-up polish; this merge records the PR's commits in the branch. Conflicts resolved in favor of our code (-X ours).
Integrates miguel-signoz's PR #2033 (github.com//pull/2033). Already landed in 0.27.2 via cherry-pick plus our follow-up race fix (NewController initializes namer/kube once instead of per-Reconcile); this merge records the PR's commit in the branch so it is marked merged. Conflicts resolved in favor of our code (-X ours).
writeValue injected element text verbatim, so passwords or values containing &, < or > produced invalid users.xml and ClickHouse failed to start. Escape those characters so the rendered document stays well-formed. Signed-off-by: Arunesh Dwivedi <arunesh.devops@gmail.com>
Integrates AruneshDwivedi's PR #2034 (github.com//pull/2034). Already landed in 0.27.2 via cherry-pick plus our follow-up fix (gate escaping so embedded SetEmbed XML fragments are emitted verbatim; Raw/Escape encoding enum). This merge records the PR's commit in the branch. Conflicts resolved in favor of our code (-X ours) — keeps the embed-gate fix, not the PR's unconditional escape.
sunsingerus
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.27.2