e2e: wait for network online, notice failed provisioning

test/e2e/README.md

@@ -220,6 +220,127 @@ provision=all ./run_tests.sh policies.test-suite
 Only `all` changes the Kubernetes release, so only `all` ends up matching what
 the run resolved.
 
+A run also stops when it finds a VM whose provisioning never finished:
+
+```
+error: provisioning the VM in /path/to/n4c16-fedora-43-crio started at 2026-11-01T00:04:11+00:00 and did not finish.
+vagrant flags the machine as provisioned all the same, so the tests would run
+against however much of the installation that run got done. Remove the output
+directory to provision a new VM, or set provision=all to provision this one
+again.
+```
+
+vagrant provisions a machine once whether the playbook succeeded or not, so
+without this check the rest of the tests would run against a VM which is
+missing whatever the playbook had not installed yet, and fail with errors of
+their own: no `kubectl`, no `crictl`, no `kubelet`, no cluster. The one real
+error is the one the failed run printed.
+
+A host of its own needs no such check: nothing is recorded about it until
+provisioning finishes, so a run which stops halfway leaves the next one to
+provision it again.
+
+## Running the tests on a host of its own
+
+The tests can run on a host which already exists, real hardware or a virtual
+machine somebody else made, instead of on a VM the framework creates:
+
+```shell
+test_hw_addr=test@10.0.0.1 test_hw_key=~/.ssh/testbox ./run_tests.sh policies.test-suite
+```
+
+The host must let that user in by key, let it run `sudo` without a password,
+and have SELinux permissive or disabled. The first two are checked before
+anything is installed. A host behind a proxy has to be configured to use one
+already: provisioning installs packages and downloads tarballs as the host,
+and the `proxy` option configures the container runtime, not the host.
+
+The run is named after the host, so its output directory is
+`<hostname>-<runtime>`. Since two hosts can answer to the same short
+hostname, a run pointed at a different host than that directory was recorded
+against is refused: set `vm_name` to keep them apart. What the host was
+provisioned with is recorded on the host, in `/etc/nri-e2e-vm-key`, because
+an output directory is not evidence of what is installed on it.
+
+**Provisioning a host changes it and nothing here undoes that**: Kubernetes,
+the container runtime, the CNI plugin and Helm are installed, `kubeadm init`
+is run, `firewalld` is removed, and `/etc/resolv.conf` is rewritten if
+`dns_nameserver` is set. A host which already runs a cluster is refused;
+`provision=all` says that taking that cluster down is what you meant.
+
+What is done differently from a VM, all of it gated on `is_hardware`:
+
+| on a VM | on a host of its own |
+| --- | --- |
+| the root filesystem is grown to fill the disk | left alone |
+| SELinux is turned off | required to be permissive or disabled already |
+| the box cache can package the VM | refused, there is nothing to package |
+| the topology is created to order | whatever the host has |
+
+Swap is not touched on either: kubelet is configured to tolerate it without
+using it.
+
+A test which needs the node rebooted does not run on a host of its own:
+rebooting goes through vagrant, so the same path would switch the host off
+for good. `vm-reboot`, `vm-set-kernel-cmdline-reboot` and
+`vm-kernel-pkgs-install` refuse before changing anything, which rules out the
+`test22-isolcpus` tests and the custom kernel ones.
+
+### Leaving the runtime alone
+
+Give the runtime's release as `installed` to keep the one the host has:
+
+```shell
+test_hw_addr=test@10.0.0.1 test_hw_key=~/.ssh/testbox containerd_release=installed \
+    ./run_tests.sh policies.test-suite
+```
+
+Nothing is then downloaded, unpacked, reconfigured or restarted, and the
+version is read off the host so the run still records what is under test.
+`installed` needs `test_hw_addr`. In exchange the host's runtime has to
+already have:
+
+- **NRI enabled**, which is the default in containerd 2.x and recent CRI-O.
+- **The systemd cgroup driver**, matching what `kubeadm` gives the kubelet.
+- **A CNI plugin directory matching where the binaries are**, which on Fedora
+  is `/usr/libexec/cni` rather than `/opt/cni/bin`.
+- **Working image pulls**, through any proxy the host needs.
+
+### Trying it on localhost
+
+This path can be exercised without separate hardware, by pointing it at
+`127.0.0.1`. **This provisions the machine you run it on**, so use one you
+are willing to have a Kubernetes cluster installed on.
+
+```shell
+# A throwaway user to log in as, and a key for it.
+sudo useradd -G root,wheel test
+ssh-keygen -t ed25519 -N '' -f ~/.ssh/ansible-test-ssh
+sudo install -d -m 700 -o test -g test /home/test/.ssh
+sudo install -m 600 -o test -g test ~/.ssh/ansible-test-ssh.pub \
+    /home/test/.ssh/authorized_keys
+
+# Passwordless sudo for it.
+sudo /bin/bash -c 'echo "test ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/test'
+
+# Check both before handing them to the framework.
+ssh -i ~/.ssh/ansible-test-ssh test@127.0.0.1 sudo -n true && echo ok
+
+# Run one test against it, from the top of the repository.
+test_hw_addr=test@127.0.0.1 test_hw_key=~/.ssh/ansible-test-ssh \
+    ./test/e2e/run_tests.sh ./test/e2e/policies.test-suite/template/n4c16/test01-dra
+
+# The same, leaving the runtime the host already has alone.
+containerd_release=installed test_hw_addr=test@127.0.0.1 test_hw_key=~/.ssh/ansible-test-ssh \
+    ./test/e2e/run_tests.sh ./test/e2e/policies.test-suite/template/n4c16/test01-dra
+```
+
+A second run reuses the cluster the first one built, as long as the versions
+it resolves still match what the host is recorded as carrying, and
+`provision=all` rebuilds it instead. What is refused is a host carrying a
+cluster this framework has no record of, which is what an interrupted first
+run leaves behind.
+
 ## Collecting coverage data
 
 The tests can collect the same kind of coverage data from the plugins they

test/e2e/files/Vagrantfile.in

@@ -15,38 +15,15 @@ HOSTNAME = "SERVER_NAME"
 # How many nodes to create
 N = 1
 
-KERNEL_GETSOURCE = "#{ENV['kernel_getsource']}"
-KERNEL_CONFIG = "#{ENV['kernel_config']}"
-K8S_RELEASE = "#{ENV['k8s_release']}"
-K8S_VERSION = "#{ENV['k8s_version']}"
-K8S_FEATURE_GATES = "#{ENV['k8s_feature_gates']}"
-HELM_RELEASE = "#{ENV['helm_release']}"
-CRI_RUNTIME = "#{ENV['k8scri']}"
-CRIO_RELEASE = "1.28.1"
-CRIO_SRC = ""
-CONTAINERD_RELEASE = ""
-CONTAINERD_SRC = ""
-CACHE_DIR="#{ENV['CACHE_DIR']}"
 NO_PROVISION = "#{ENV['E2E_NO_PROVISION']}"
 
-# provision=all asks for the cluster to be rebuilt, not just for the
-# playbook to run again.
-KUBEADM_RESET = "#{ENV['provision']}" == "all"
-
-if CRI_RUNTIME == "containerd"
-  CONTAINERD_RELEASE = "#{ENV['containerd_release']}"
-  CONTAINERD_SRC = "#{ENV['containerd_src']}"
-else
-  CRIO_RELEASE = "#{ENV['crio_release']}"
-  CRIO_SRC = "#{ENV['crio_src']}"
-end
-
-CNI_PLUGIN = "#{ENV['cni_plugin']}"
-CNI_RELEASE = "#{ENV['cni_release']}"
-
-NRI_RESOURCE_POLICY_SRC = "#{ENV['nri_resource_policy_src']}"
 OUTPUT_DIR = "#{ENV['OUTPUT_DIR']}"
 
+# The variables the provisioning playbook takes are not assembled here.
+# vm-write-ansible-vars writes them to ansible-vars.json in this directory,
+# and the provisioner below takes that file, so a run which provisions a host
+# over ssh rather than a VM feeds the playbook from the very same file.
+
 # Each VM should have its own ssh port so that we can have multiple
 # VM in the host side that will not have conflicting source port
 # values. Use the value that is set in the caller script and placed in
@@ -118,32 +95,7 @@ Vagrant.configure("2") do |config|
   if NO_PROVISION != "1"
     config.vm.provision :ansible do |ansible|
       ansible.playbook = "#{ENV['nri_resource_policy_src']}/test/e2e/playbook/provision.yaml"
-      ansible.extra_vars = {
-        network: "10.217.0.0/16",
-        hostname: HOSTNAME,
-        dns_nameserver: "#{ENV['DNS_NAMESERVER']}",
-        dns_search_domain: "#{ENV['DNS_SEARCH_DOMAIN']}",
-        https_proxy: "#{ENV['HTTPS_PROXY']}",
-        http_proxy: "#{ENV['HTTP_PROXY']}",
-        no_proxy: "#{ENV['NO_PROXY']}",
-        kernel_getsource: KERNEL_GETSOURCE,
-        kernel_config: KERNEL_CONFIG,
-        k8s_release: K8S_RELEASE,
-        k8s_version: K8S_VERSION,
-        k8s_feature_gates: K8S_FEATURE_GATES,
-        kubeadm_reset: KUBEADM_RESET,
-        helm_release: HELM_RELEASE,
-        cri_runtime: CRI_RUNTIME,
-        containerd_release: CONTAINERD_RELEASE,
-        containerd_src: CONTAINERD_SRC,
-        crio_release: CRIO_RELEASE,
-        crio_src: CRIO_SRC,
-        cache_dir: CACHE_DIR,
-        cni_plugin: CNI_PLUGIN,
-        cni_release: CNI_RELEASE,
-        nri_resource_policy_src: NRI_RESOURCE_POLICY_SRC,
-        outdir: OUTPUT_DIR,
-      }
+      ansible.extra_vars = "@#{OUTPUT_DIR}/ansible-vars.json"
     end
   end
 end

test/e2e/lib/test.bash

@@ -216,7 +216,10 @@ verify-metrics-has-no-line() { # script API
 ###
 
 # The kubectl command which addresses the node resource topology of the node.
-export nrt_kubectl_get="kubectl get noderesourcetopologies.topology.node.k8s.io \$(hostname)"
+#
+# The name the run resolved, which is the short hostname and the name the node
+# is registered under. A host's own hostname can be a fully qualified one.
+export nrt_kubectl_get="kubectl get noderesourcetopologies.topology.node.k8s.io ${VM_HOSTNAME:-\$(hostname)}"
 
 nrt-query() { # script API
     # Usage: nrt-query JQ-EXPRESSION