From dc5a69d6f0ef8d724958d045e608651feb37eb30 Mon Sep 17 00:00:00 2001 From: cmulk <17730096+cmulk@users.noreply.github.com> Date: Mon, 7 Sep 2026 21:06:19 -0500 Subject: [PATCH] docs(networking): explain GRE mesh usage and MTU requirements --- docs/experiments.md | 4 +++ docs/gre-mesh.md | 86 +++++++++++++++++++++++++++++++++++++++++++++ docs/settings.md | 1 + mkdocs.yml | 1 + 4 files changed, 92 insertions(+) create mode 100644 docs/gre-mesh.md diff --git a/docs/experiments.md b/docs/experiments.md index 999ba59..eb064f5 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -1,5 +1,9 @@ # Experiments +For experiments spanning multiple minimega hosts, see [GRE Mesh](gre-mesh.md) +to trunk experiment networks between hosts with `--use-gre-mesh`, including +the physical network MTU requirements. + ## Listing Experiments ### From the Web-UI diff --git a/docs/gre-mesh.md b/docs/gre-mesh.md new file mode 100644 index 0000000..3273ca4 --- /dev/null +++ b/docs/gre-mesh.md @@ -0,0 +1,86 @@ +# GRE Mesh + +Use `--use-gre-mesh` when an experiment spans multiple minimega nodes and its +virtual networks need to connect VMs on different hosts. phēnix uses minimega +to connect the hosts' Open vSwitch (OVS) bridges with Layer 2 GRE tunnels. +These tunnels trunk experiment VLANs across the host IP network, so VMs on +the same experiment network communicate transparently across hosts, including +Ethernet broadcast traffic such as ARP. Guest addresses and network membership +do not need to change when VMs run on different hosts. + +For example, two VMs on the same experiment VLAN can communicate even when +one runs on minimega node A and the other on node B. Their Ethernet frames +travel from A's OVS bridge through GRE to B's OVS bridge. The physical network +carries the encapsulated traffic without needing to trunk each experiment +VLAN itself. See minimega's [namespace bridge documentation](https://www.sandia.gov/minimega/namespaces/). + +## Prerequisites + +The minimega nodes must already belong to a working mesh, with IP connectivity +between tunnel endpoints and GRE traffic (IP protocol 47) permitted along the +path. The flag configures experiment network connectivity; VM placement still +depends on the experiment's schedule and available hosts. A single-host +experiment does not need GRE for communication between its local VMs. + +Prepare the physical network MTU before starting experiments; see +[MTU and fragmentation](#mtu-and-fragmentation) below. + +## Enable GRE + +Enable GRE when creating an experiment (the default is disabled): + +```bash +phenix experiment create my-exp -t my-topology --use-gre-mesh +phenix experiment start my-exp +``` + +For an existing experiment, stop it, run `phenix experiment edit my-exp`, +and set `spec.useGREMesh` to `true` before starting it again. Passing the flag +only to `experiment start` does not change this stored setting. + +To enable GRE for experiments created or updated through the UI/API, start +with `phenix ui --use-gre-mesh`. CLI commands inherit an enabled server setting; +`--use-gre-mesh=false` cannot override it. See [Settings & Configuration](settings.md) +for configuration file and environment variable alternatives. + +## Bridge Scope + +GRE mesh setup applies to the experiment's **default bridge**. Interfaces +assigned to other bridges explicitly in the topology need their own +connectivity arrangements. [Bridge Mode](bridge-mode.md) explains how phēnix +selects the default bridge; `--bridge-mode auto` gives each experiment its own +bridge and can be combined with `--use-gre-mesh` at creation. + +## MTU and Fragmentation + +The physical network needs a larger MTU than the guests to accommodate the +encapsulated Ethernet frame and IP/GRE headers. For 1500-byte guest MTUs, +a host IP MTU of 1600 provides headroom for ordinary GRE over IPv4; additional +encapsulation may require more. Configure sufficient MTU/frame size across +**all host interfaces, routed hops, and physical switch ports** carrying the +tunnels. phēnix does not configure these for you. + +An undersized path can cause fragmentation, reduced performance, or dropped +packets even when small pings succeed. If increasing the physical network MTU +is not possible, reduce guest MTUs to leave room for encapsulation. See the +[OVS tunneling guide](https://docs.openvswitch.org/en/latest/howto/tunneling/) +and [RFC 7588](https://www.rfc-editor.org/rfc/rfc7588.html) for details. + +## Verify Connectivity + +Confirm that the communicating VMs actually run on different minimega hosts +and share an experiment VLAN. On each host, `ovs-vsctl show` should show GRE +interfaces attached to the experiment's default bridge. Check the configured +remote endpoints and verify that the host network can reach them. + +Test both small and full-size packets between guests. For a Linux guest with +a 1500-byte IPv4 MTU, for example: + +```bash +ping -M do -s 1472 +``` + +The 1472-byte payload plus IPv4 and ICMP headers makes a 1500-byte guest IP +packet. This checks guest connectivity at that size, but does not prove the +outer GRE packets avoided fragmentation. Inspect traffic on the physical +interfaces or fragmentation counters on the hosts to verify that separately. diff --git a/docs/settings.md b/docs/settings.md index 5597e98..174907a 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -82,6 +82,7 @@ phēnix resolves configuration settings in the following order (highest to lowes | Setting Key | Environment Variable | Default | Description | | :--- | :--- | :--- | :--- | | `bridge-mode` | `PHENIX_BRIDGE_MODE` | `manual` | Bridge naming mode for experiments. `auto` uses the experiment name as the bridge name; `manual` uses the user-specified bridge name, or `phenix` if not specified. See [Bridge Mode](bridge-mode.md). | +| `use-gre-mesh` | `PHENIX_USE_GRE_MESH` | `false` | Enable Layer 2 GRE tunnels for the default bridge of experiments created or updated with this setting. See [GRE Mesh](gre-mesh.md) for usage, server inheritance, and MTU requirements. Restart the server after changing its startup configuration. | | `log.level` | `PHENIX_LOG_LEVEL` | `info` | Global log verbosity (`debug`, `info`, `warn`, `error`). | | `log.console` | `PHENIX_LOG_CONSOLE` | `stderr` | Destination for console logs (`stderr`, `stdout`, or a file path). Uses **Text/Human-Readable** format. Note: Setting this to a file path will prevent console logs from appearing in `docker logs`. | | `log.system.path` | `PHENIX_LOG_SYSTEM_PATH` | `/var/log/phenix/phenix.log` | Path to the persistent system log file (used by UI). Uses **JSON** format. This is independent of `log.console` and is always active. | diff --git a/mkdocs.yml b/mkdocs.yml index cb93e92..fe7a895 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -89,6 +89,7 @@ nav: - State of Health: state-of-health.md - Networking: - Bridge Mode: bridge-mode.md + - GRE Mesh: gre-mesh.md - Netflow: netflow.md - Tunneler: tunneler.md - Integrations: