Self-fence Kubernetes Lease leadership using monotonic time

[marcschier]

Failure

Lease takeover trusted another replica's wall-clock RenewTime, so clock skew could steal a live Lease. A hung renewal could also leave local leadership asserted, and a stale completion or notification could resurrect leadership after fencing or disposal.

Fix

• Measure unchanged foreign resourceVersion age with local monotonic time.
• Serialize and bound API attempts, and self-fence with a lease-duration watchdog.
• Reject stale completions and suppress out-of-order leadership notifications.

Standard

This is a Kubernetes coordination extension to OPC 10000-4 §6.6 redundancy. Kubernetes Lease ownership is now evaluated from stable resource observations rather than untrusted peer wall clocks.

Tests

• dotnet test tests\Opc.Ua.Redundancy.Kubernetes.Tests\Opc.Ua.Redundancy.Kubernetes.Tests.csproj -c Release --nologo -m:1 — 97 passed on net8.0 and 97 passed on net10.0

• Review follow-up: KubernetesLeaseLeaderElectionTests — 22 passed on net8.0.

Split

Split from #4021.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR hardens Kubernetes Lease leader election against clock skew and stale/late operations by switching takeover logic to local monotonic observations and adding fencing/serialization around API attempts and notifications.

Changes:

• Evaluate foreign lease expiry using unchanged resourceVersion duration measured with local monotonic time.
• Serialize and bound Kubernetes API calls; add a watchdog to self-fence on hung renewal attempts.
• Suppress stale/out-of-order leadership notifications and reject stale completions.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
tests/Opc.Ua.Redundancy.Kubernetes.Tests/LeaderElection/KubernetesLeaseLeaderElectionTests.cs	Updates/expands tests to validate monotonic takeover window, resourceVersion reset behavior, watchdog fencing, and notification ordering.
src/Opc.Ua.Redundancy.Kubernetes/LeaderElection/KubernetesLeaseLeaderElection.cs	Implements monotonic resourceVersion observation, API attempt gating/timeouts, watchdog self-fencing, and notification generation fencing.

[marcschier]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.53%. Comparing base (70c74e7) to head (f85d0b5).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4042      +/-   ##
==========================================
- Coverage   73.55%   73.53%   -0.03%     
==========================================
  Files        1345     1345              
  Lines      180036   180038       +2     
  Branches    31677    31678       +1     
==========================================
- Hits       132427   132390      -37     
- Misses      36911    36944      +33     
- Partials    10698    10704       +6     

see 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.