CMP-4485: [experiment] Parallelize destructive serial e2e tests across per-worker pools

[Vincent056]

Experimental — measuring whether the destructive serial e2e tests are faster when run in parallel. Opening this to let CI time it and to see whether concurrent node reboots are stable. Not necessarily for merge as-is.

Motivation

The e2e-aws-serial suite is ~97 min of test wall-clock, and the 5 reboot/remediation tests dominate it because they each apply a MachineConfig to the single shared e2e pool and wait for a node reboot — strictly one after another.

What this does

Give each destructive test its own isolated MachineConfigPool lane so they run concurrently:

• framework/testpools.go (new): TestPool lane + f.AcquireTestPool(t) allocator (buffered channel — blocks/queues when all lanes are busy, releases via t.Cleanup). setUpTestPools() carves one pool per existing worker node (no cluster scaling — reuses what's available) with matching per-lane ScanSettings.
• framework/common.go: split createMachineConfigPool into getWorkerNodes() + createMachineConfigPoolFromNode(); generalized ensureE2EScanSettings → …ForPool(name).
• The 5 remediation tests (TestAutoRemediate, TestUnapplyRemediation, TestUpdateRemediation, TestKubeletConfigRemediation, TestRuntimeSSHConfigWithRemediation) now t.Parallel() and target their acquired lane.
• Distinct MachineConfigPools reboot independently (MCO parallelizes across pools), so lanes proceed at the same time.

Also includes the CMP-4485 SSH remediation fix (label-based remediation discovery) as the first commit.

Safety / scoping

• Lane count defaults to 1 (legacy single e2e pool) so the other e2e suites that share SetUp are unaffected. The e2e-serial Makefile target raises it to 3 via E2E_PARALLEL_POOLS (+ -parallel 3).
• Teardown deletes per-lane ScanSettings but skips the node-label restore / pool deletion (those would reboot every lane node back to rendered-worker, wasted since the CI cluster is torn down right after).

Known risk this experiment is testing

With 3 lanes on a 3-worker cluster, every worker is a reboot target, so the compliance-operator pod has no idle worker to fall back to during simultaneous reboots. It should survive (MCO drives the reboots; tests poll with long timeouts), but if it flakes/wedges, set E2E_PARALLEL_POOLS=2 to keep one baseline worker (2 lanes).

Expected result

Reboot block ~37 min (post-SSH-fix) → ~13–15 min at 3 lanes. CI timing on this PR is the real measurement.

Jira: https://redhat.atlassian.net/browse/CMP-4485

🤖 Generated with Claude Code

[openshift-ci-robot]

@Vincent056: This pull request references CMP-4485 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "compliance-operator-1.10.0" instead.

Details

In response to this:

Experimental — measuring whether the destructive serial e2e tests are faster when run in parallel. Opening this to let CI time it and to see whether concurrent node reboots are stable. Not necessarily for merge as-is.

Motivation

The e2e-aws-serial suite is ~97 min of test wall-clock, and the 5 reboot/remediation tests dominate it because they each apply a MachineConfig to the single shared e2e pool and wait for a node reboot — strictly one after another.

What this does

Give each destructive test its own isolated MachineConfigPool lane so they run concurrently:

• framework/testpools.go (new): TestPool lane + f.AcquireTestPool(t) allocator (buffered channel — blocks/queues when all lanes are busy, releases via t.Cleanup). setUpTestPools() carves one pool per existing worker node (no cluster scaling — reuses what's available) with matching per-lane ScanSettings.
• framework/common.go: split createMachineConfigPool into getWorkerNodes() + createMachineConfigPoolFromNode(); generalized ensureE2EScanSettings → …ForPool(name).
• The 5 remediation tests (TestAutoRemediate, TestUnapplyRemediation, TestUpdateRemediation, TestKubeletConfigRemediation, TestRuntimeSSHConfigWithRemediation) now t.Parallel() and target their acquired lane.
• Distinct MachineConfigPools reboot independently (MCO parallelizes across pools), so lanes proceed at the same time.

Also includes the CMP-4485 SSH remediation fix (label-based remediation discovery) as the first commit.

Safety / scoping

• Lane count defaults to 1 (legacy single e2e pool) so the other e2e suites that share SetUp are unaffected. The e2e-serial Makefile target raises it to 3 via E2E_PARALLEL_POOLS (+ -parallel 3).
• Teardown deletes per-lane ScanSettings but skips the node-label restore / pool deletion (those would reboot every lane node back to rendered-worker, wasted since the CI cluster is torn down right after).

Known risk this experiment is testing

With 3 lanes on a 3-worker cluster, every worker is a reboot target, so the compliance-operator pod has no idle worker to fall back to during simultaneous reboots. It should survive (MCO drives the reboots; tests poll with long timeouts), but if it flakes/wedges, set E2E_PARALLEL_POOLS=2 to keep one baseline worker (2 lanes).

Expected result

Reboot block ~37 min (post-SSH-fix) → ~13–15 min at 3 lanes. CI timing on this PR is the real measurement.

Jira: https://redhat.atlassian.net/browse/CMP-4485

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Vincent056

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Vincent056]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1304-11a24b1001fe871abd668cab82b58df2d1221ecd

[openshift-ci]

@Vincent056: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-serial	11a24b1	link	true	/test e2e-aws-serial
ci/prow/e2e-aws-parallel	11a24b1	link	true	/test e2e-aws-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[abushkin-redhat]

Hi Vincent,

The PR looks good, though during code review with Claude I found two potential bugs:

1. RemoveObsoleteRemediationAndCheck ignores its pool parameter (common.go:3172)

   • The function accepts a pool string parameter and correctly passes it to WaitForMachinePoolUpdate at line 3199, but at line 3172 it fetches poolBeforeRemediation using the hardcoded TestPoolName ("e2e") instead of pool: err = f.Client.Get(context.TODO(), types.NamespacedName{Name: TestPoolName}, poolBeforeRemediation)
   • With E2E_PARALLEL_POOLS=3 (the Makefile default), pools are named e2e-0, e2e-1, e2e-2 — "e2e" doesn't exist. TestUpdateRemediation will fail with NotFound on every make e2e-serial run.
   • Fix: TestPoolName → pool.

2. for err := range metricErrs iterates indices, not errors (utils.go:371)

   • Changed from for _, err := range metricErrs to for err := range metricErrs. In Go 1.22+, for v := range slice yields the index (int), not the value. log.Println(err) prints 0, 1, 2 instead of the actual error messages, making metric assertion failures undiagnosable.
   • Fix: restore for _, err := range metricErrs.

[openshift-ci]

Stale PRs are closed after 21d of inactivity.

If this PR is still relevant, comment to refresh it or remove the stale label.
Mark the PR as fresh by commenting /remove-lifecycle stale.

If this PR is safe to close now please do so with /close.

/lifecycle stale