CMP-4616: Fix flaky TestScheduledSuiteNoStorage e2e test

[yuumasato]

Summary

• TestScheduledSuiteNoStorage validates that no PVCs are created when RawResultStorage is disabled — it tests storage behavior, not compliance outcome
• The test only accepted ResultCompliant, but under cluster pressure from ~70 parallel tests the scan can return NON_COMPLIANT or ERROR, causing an immediate hard failure
• Use WaitForSuiteScansStatusAnyResult accepting both COMPLIANT and NON_COMPLIANT
• Move the PVC assertion after the suite completes, matching the pattern in TestScheduledSuitePlatformNoStorage

Test plan

• e2e-aws-parallel CI job passes without TestScheduledSuiteNoStorage flaking

Jira: CMP-4616

🤖 Generated with Claude Code

[openshift-ci-robot]

@yuumasato: This pull request references CMP-4616 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.1.0" version, but no target version was set.

Details

In response to this:

Summary

• TestScheduledSuiteNoStorage validates that no PVCs are created when RawResultStorage is disabled — it tests storage behavior, not compliance outcome
• The test only accepted ResultCompliant, but under cluster pressure from ~70 parallel tests the scan can return NON_COMPLIANT or ERROR, causing an immediate hard failure
• Use WaitForSuiteScansStatusAnyResult accepting both COMPLIANT and NON_COMPLIANT
• Move the PVC assertion after the suite completes, matching the pattern in TestScheduledSuitePlatformNoStorage

Test plan

• e2e-aws-parallel CI job passes without TestScheduledSuiteNoStorage flaking

Jira: CMP-4616

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1369-6dfabb63106660ea58e4960a55e008541850a8a4

[openshift-ci]

@guzalv: changing LGTM is restricted to collaborators

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[abushkin-redhat]

Hey @yuumasato, everything looks good. Claude did find little nit though:

The commit message says no_netrc_files can return "NON_COMPLIANT or ERROR under cluster pressure," but this only accepts Compliant/NonCompliant. An unlisted result makes waitForSuiteScansStatusMulti return an error, which stops wait.Poll immediately (no retry) → hard t.Fatal — so a ResultError scan still flakes this test. Since it validates storage, not outcome (and no PVC is created on error), adding it fits the intent:

err = f.WaitForSuiteScansStatusAnyResult(f.OperatorNamespace, suiteName, compv1alpha1.PhaseDone,
compv1alpha1.ResultCompliant, compv1alpha1.ResultNonCompliant, compv1alpha1.ResultError)

If tolerating ERROR is intentionally undesirable, it may be worth dropping it from the commit message instead.

[yuumasato]

@abushkin-redhat Nice catch, updated.

[abushkin-redhat]

/lgtm

[github-actions]

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1369-9535c011eea3b1d91d6d173abc2a88ad00a54bef

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abushkin-redhat, guzalv, yuumasato

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [abushkin-redhat,yuumasato]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Vincent056]

LGTM overall — accepting NON_COMPLIANT is right for a storage-behavior test, and moving the PVC assertion after suite completion is strictly stronger than the old early check (it now catches PVCs created at any point of the run).

One follow-up worth considering: accepting ResultError makes the no-PVC assertion vacuous when the scan never actually ran (scanner pods/result server never launched), and it would have kept this test green through the recent every-scan-ERRORs openscap-image outage. The other AnyResult callers in this file accept only Compliant+NonCompliant. Fine to address in a follow-up — either drop ResultError or add an assertion that scanner pods executed.