Summary
After upgrading from 2.2.6 to 2.2.8, an existing InnoDBCluster that is already OFFLINE when the operator restarts may remain OFFLINE indefinitely if no further pod status change occurs after operator startup.
Versions
- works as expected: 2.2.6 (
9.5.0-2.2.6)
- problematic: 2.2.8 (
9.7.0-2.2.8)
Test setup
The reproduction I used was with an InnoDBCluster configured with 1 instance.
Steps to reproduce
- Deploy MySQL Operator and create an
InnoDBCluster with 1 instance
- Put the cluster into a state where it is already OFFLINE when the operator comes back up
- for example, after a node reboot or a simultaneous operator/pod restart scenario
- Let the operator restart
- Ensure no additional pod status change occurs after startup
- Observe the cluster status
Expected behavior
After operator restart, an existing cluster that is already OFFLINE should still be re-evaluated for recovery.
Actual behavior
In 2.2.8, the cluster may remain OFFLINE indefinitely unless a later pod status change happens.
Additional note
I compared the relevant source between 2.2.6 and 2.2.8, and this looks related to a change in the pod status watch / restart detection behavior after operator restart. I can provide the exact code comparison if that would be helpful.
Summary
After upgrading from 2.2.6 to 2.2.8, an existing InnoDBCluster that is already OFFLINE when the operator restarts may remain OFFLINE indefinitely if no further pod status change occurs after operator startup.
Versions
9.5.0-2.2.6)9.7.0-2.2.8)Test setup
The reproduction I used was with an
InnoDBClusterconfigured with 1 instance.Steps to reproduce
InnoDBClusterwith 1 instanceExpected behavior
After operator restart, an existing cluster that is already OFFLINE should still be re-evaluated for recovery.
Actual behavior
In 2.2.8, the cluster may remain OFFLINE indefinitely unless a later pod status change happens.
Additional note
I compared the relevant source between 2.2.6 and 2.2.8, and this looks related to a change in the pod status watch / restart detection behavior after operator restart. I can provide the exact code comparison if that would be helpful.