[EPIC] [INFRA] CI Enhancement

[lanfon72]

What's the test to develop? Please describe

The epic contains multiple items to be done.

Describe the items of the test development (DoD, definition of done) you'd like

High priority (by order)

• Fix environment issues
  • [TEST] test_4_vm_host_powercycle.py::TestHostState::test_poweroff_state due to KeyError: 'GITHUB_USERNAME' #2004
  • [TEST] Convert Scripts related to testing automation to version controlled using Github #1649
  • [TEST] Prevent fixture services out of space #1406
  • [TEST] Filter is not working on Jenkins job on ECM lab #1099
• Update Jenkins
  • [INFRA] Bump Java For Jenkins #1940
  • [TEST] Add 4 dedicated nodes for 2 stage upgrade. #1936
• Reform CI process
  • [TASK] Decouple bundled cluster #2188
  • [TASK] Decouple Ansible dependency #2189
  • [TEST] Add CIs to run 1, 2 & 3 nodes upgrade test and run e2e on them after the upgrade #1935
  • [TASK] Enhance the process to update pipeline jobs on QA main Jenkins environment #2032
  • [TASK] Improve slack notification #2187
• Rancher Integration (probably be implemented in another Jenkins instance)
  • [TEST] Implement Rancher upgrade in Rancher integration test #1542
  • [TEST] Setup Rancher Harvester matrix CI on Jenkins #1525
• [TEST] Add arm machines in Jenkins pipeline #1378

Low priority

• Air-gapped
  • [TEST] Build the fully airgapped on bare metal machines for release upgrade testing #967
• OpenStack and related
  • [DevOps] Update Jenkins OpenStack Groovy Script #1644
  • [EPIC] Implement e2e test for the VM import testing on VMware and OpenStack #1534
  • [BUG][DevOps] Fix Syntax Errors in Jenkins OpenStack Pipeline #1440
  • [DevOps] Rollout OpenStack Jenkins Pipeline to Prod #1372
  • [e2e] Implement Tests For VM OpenStack Import  #1373
• Others
  • [EPIC][DevOps] Automation to provision Harvester integrations in both "airgap" & "non-airgap" ecosystems #1101
  • [TEST] Document CI (Jenkins) Workflow for Harvester QA #1648