From a8a22fe466a03b206e770b74d2efd1b2f1fdece3 Mon Sep 17 00:00:00 2001 From: Phill Kelley <34226495+Paraphraser@users.noreply.github.com> Date: Mon, 3 Mar 2025 12:33:58 +1100 Subject: [PATCH 1/2] 2025-03-03 ZeroTier - health checking - alternative proposal This PR follows on from the extensive discussion associated with #37. Never before have I even *contemplated* submitting a PR covering the same ground as an existing open PR. However, on this occasion I thought it might be useful to have a concrete proposal to compare and contrast with #37. I sincerely hope that laying this on the (virtual) table and then minimising further interaction *might* help us converge on a solution.
Changes: * `docker-compose.yml` and `docker-compose-router.yml`: - replaces deprecated `version` statement with `---`. - adds example environment variables. * `Dockerfile` - corrects case of "as" to "AS" (silences build warning). - adds and configures `healthcheck.sh` (as per #37). - includes `tzdata` package (moved from `Dockerfile.router`) so messages have local timestamps. * `Dockerfile.router` - removes `tzdata` (moved to `Dockerfile`). * `entrypoint-router.sh`: - code for first launch auto join of listed networks expanded to include additional help material. * `entrypoint.sh`: - "first launch" auto join of listed networks (code copied from `entrypoint-router.sh`, as modified per above). - "self repair" of permissions in persistent store (code copied from `entrypoint-router.sh`). - adds launch-time message to make it clear that the client is launching (complements messages in `entrypoint-router.sh`). - abstracts some common strings to environment variables (opportunistic change). * `README.md`: - updates examples. - describes new environment variables (including move of `ZEROTIER_ONE_NETWORK_IDS` from `README-router.md`. - documents health-checking. * `README-router.md` - updates examples. - explains relationship of router and client. Added: * `healthcheck.sh`, based on original proposal in #37 and subsequent suggestions for modification by me. I gave serious consideration to the code for synchronising networks in the entry point scripts. The idea is quite attractive. It is safe to automate joins in a "clean slate" situation. However, a *leave* followed by a *join* is not guaranteed to be idempotent. That's because the *leave* destroys the network-specific configuration options (`allowManaged`, `allowGlobal`, `allowDefault`, `allowDNS`). On balance I think it's better left to users to send explicit *leave* commands via the CLI and take responsibility for restoring lost configuration options on any subsequent *join*. I will post the results of testing this PR separately. Signed-off-by: Phill Kelley <34226495+Paraphraser@users.noreply.github.com> --- Dockerfile | 9 +- Dockerfile.router | 2 +- README-router.md | 79 +++++----------- README.md | 171 +++++++++++++++++++++++++++++++++-- docker-compose-router.yml | 7 +- docker-compose.yml | 11 ++- scripts/entrypoint-router.sh | 24 +++-- scripts/entrypoint.sh | 34 ++++++- scripts/healthcheck.sh | 36 ++++++++ 9 files changed, 293 insertions(+), 80 deletions(-) create mode 100755 scripts/healthcheck.sh diff --git a/Dockerfile b/Dockerfile index 3ceaefe..2bbd601 100644 --- a/Dockerfile +++ b/Dockerfile @@ -3,7 +3,7 @@ ARG ALPINE_VERSION=3.21 ARG ZT_COMMIT=185a3a2c76e6bf1b1c0415871f43076638eb007c ARG ZT_VERSION=1.14.2 -FROM ${ALPINE_IMAGE}:${ALPINE_VERSION} as builder +FROM ${ALPINE_IMAGE}:${ALPINE_VERSION} AS builder ARG ZT_COMMIT @@ -27,9 +27,9 @@ LABEL org.opencontainers.image.title="zerotier" \ org.opencontainers.image.licenses="MIT" \ org.opencontainers.image.source="https://github.com/zyclonite/zerotier-docker" -COPY --from=builder /src/zerotier-one /scripts/entrypoint.sh /usr/sbin/ +COPY --from=builder /src/zerotier-one /scripts/entrypoint.sh /scripts/healthcheck.sh /usr/sbin/ -RUN apk add --no-cache --purge --clean-protected libc6-compat libstdc++ \ +RUN apk add --no-cache --purge --clean-protected libc6-compat libstdc++ tzdata \ && mkdir -p /var/lib/zerotier-one \ && ln -s /usr/sbin/zerotier-one /usr/sbin/zerotier-idtool \ && ln -s /usr/sbin/zerotier-one /usr/sbin/zerotier-cli \ @@ -40,3 +40,6 @@ EXPOSE 9993/udp ENTRYPOINT ["entrypoint.sh"] CMD ["-U"] + +HEALTHCHECK --interval=60s --timeout=8s --retries=2 --start-period=60s --start-interval=3s \ + CMD ["/bin/sh", "/usr/sbin/healthcheck.sh"] diff --git a/Dockerfile.router b/Dockerfile.router index 4700fba..8e5a7e0 100644 --- a/Dockerfile.router +++ b/Dockerfile.router @@ -13,7 +13,7 @@ ENV LOG_PATH=/var/log/supervisor COPY scripts/entrypoint-router.sh /usr/sbin/ -RUN apk add --no-cache --purge --clean-protected iptables iptables-legacy tzdata \ +RUN apk add --no-cache --purge --clean-protected iptables iptables-legacy \ && rm -rf /var/cache/apk/* EXPOSE 9993/udp diff --git a/README-router.md b/README-router.md index 4bedc9f..9d8e481 100644 --- a/README-router.md +++ b/README-router.md @@ -17,6 +17,25 @@ This implementation extends the concept so that you have a choice of: * Permitting clients on your LAN to initiate connections with remote devices reachable across your ZeroTier Cloud network; or * Both of the above (ie a full router). +### router vs client + +Both containers run the **same** [ZeroTier One client](https://github.com/zerotier/ZeroTierOne) software. No special parameters are passed to the ZeroTier One client that somehow cause it to become a router. + +Layer Three routing is simply the process of the *host* (not the *container*) making decisions about forwarding IP datagrams (aka "packets") between network interfaces. Routing is enabled by two things: + +1. Turning on the `net.ipv4.ip_forward` kernel parameter; and +2. A set of *iptables* filters. + +The ZeroTier router container manages the insertion and withdrawal of the necessary *iptables* filters. + +The kernel parameter is usually enabled when Docker is installed. However, if IPv4 forwarding is not enabled when the Zerotier router container launches, it reports this problem in the container's log and falls back to behaving as the client. In that situation, you may need to enable the kernel parameter yourself: + +``` +$ sudo sysctl net.ipv4.ip_forward=1 +``` + +Otherwise, the client and router containers are fully interchangeable and can utilise the same persistent store. + ### Command line example ``` console @@ -64,9 +83,11 @@ services: - PUID=999 - PGID=994 - ZEROTIER_ONE_LOCAL_PHYS=eth0 - - ZEROTIER_ONE_USE_IPTABLES_NFT=true + - ZEROTIER_ONE_USE_IPTABLES_NFT=false - ZEROTIER_ONE_GATEWAY_MODE=inbound # - ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» + # - ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» + # - ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 ``` Note: @@ -80,26 +101,9 @@ Note: ### Environment variables -* `TZ` – timezone support. Example: - - ``` yaml - environment: - - TZ=Australia/Sydney - ``` - - Defaults to `Etc/UTC` if omitted. - -* `PUID` + `PGID` – user and group IDs for ownership of persistent store. Example: - - ``` yaml - environment: - - PUID=1000 - - PGID=1000 - ``` - - If omitted, `PUID` defaults to user ID 999, while `PGID` defaults to group ID 994. +All the variables supported by the client container are also supported by the router container. Please see the main [README](./README.md). - These variables are only used to ensure consistent ownership of persistent storage on each launch. They do not affect how the container *runs.* Absent a `user:` directive, the container runs as root and does not downgrade its privileges. +The variables listed below are unique to the router container: * `ZEROTIER_ONE_LOCAL_PHYS` - a space-separated list of physical interfaces that should be configured to participate in NAT-based routing. Examples: @@ -153,7 +157,7 @@ Note: 2. A line-count of zero. This means the container has not been able to configure net-filter rules on the host. If that happens, try the opposite setting for this environment variable (eg `true` instead of `false`). 3. A non-zero line-count. That means the container has been able to propagate net-filter rules into the host's tables, which is what you want. The actual number is not important, just something other than zero. - The container will always come up. Once you've authorised the client in ZeroTier Central, it will be able to join your ZeroTier Cloud network. Tests like `ping` and `traceroute` that you run on the same host will always work. However, if the container is not able to propagate its net-filter rules into the host's tables, traffic *beyond* the host where the container is running will not work properly. The problem is quite subtle so it's always a good idea to check that the host has the expected net-filters. + The container will always come up. Once you've approved the host in ZeroTier Central, it will be able to join your ZeroTier Cloud network. Tests like `ping` and `traceroute` that you run on the same host will always work. However, if the container is not able to propagate its net-filter rules into the host's tables, traffic *beyond* the host where the container is running will not work properly. The problem is quite subtle so it's always a good idea to check that the host has the expected net-filters. * `ZEROTIER_ONE_GATEWAY_MODE` - controls the traffic direction. Examples: @@ -180,39 +184,6 @@ Note: Defaults to `inbound` if omitted. Note that you will probably need one or more static routes configured in your local LAN router so that traffic originating in a local host which is not running the ZeroTier client can be directed to the gateway host. -* `ZEROTIER_ONE_NETWORK_IDS` – a space-separated list of ZeroTier network IDs. - - This variable is *only* effective on first launch. There is no default if it is omitted. Examples: - - - to join a single network: - - ``` yaml - environment: - - ZEROTIER_ONE_NETWORK_IDS=aaaaaaaaaaaaaaaa - ``` - - Equivalent of running the following command after the container first starts: - - ``` - $ docker exec zerotier zerotier-cli join aaaaaaaaaaaaaaaa - ``` - - - to join a multiple networks: - - ``` yaml - environment: - - ZEROTIER_ONE_NETWORK_IDS=aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbb - ``` - - Equivalent of running the following commands after the container first starts: - - ``` - $ docker exec zerotier zerotier-cli join aaaaaaaaaaaaaaaa - $ docker exec zerotier zerotier-cli join bbbbbbbbbbbbbbbb - ``` - - It does not matter whether you use this environment variable or the `join` command, you still need to use ZeroTier Central to approve the computer for each network it joins. - ### Managed route(s) For each ZeroTier container that is configured as a router, ZeroTier needs at least one *Managed Route*. diff --git a/README.md b/README.md index 06d8641..691d0cd 100644 --- a/README.md +++ b/README.md @@ -12,10 +12,15 @@ This is a container based on a lightweight Alpine Linux image and a copy of Zero To run this container in the correct way requires some special options to give it special permissions and allow it to persist its files. Here's an example (tested on Fedora CoreOS): - docker run --name zerotier-one --device=/dev/net/tun --net=host \ - --cap-add=NET_ADMIN --cap-add=SYS_ADMIN \ - -v /var/lib/zerotier-one:/var/lib/zerotier-one zyclonite/zerotier - +``` console +$ docker run --name zerotier-one --device=/dev/net/tun \ + --network=host -d \ + --cap-add=NET_ADMIN --cap-add=SYS_ADMIN \ + --env TZ=Etc/UTC --env PUID=$(id -u) --env PGID=$(id -g) \ + --env ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» \ + -v /var/lib/zerotier-one:/var/lib/zerotier-one \ + zyclonite/zerotier +``` This runs zyclonite/zerotier in a container with special network admin permissions and with access to the host's network stack (no network isolation) and /dev/net/tun to create tun/tap devices. This will allow it to create zt# interfaces on the host the way a copy of ZeroTier One running on the host would normally be able to. @@ -23,14 +28,164 @@ In other words that basically does the same thing that running zerotier-one dire It also mounts /var/lib/zerotier-one to /var/lib/zerotier-one inside the container, allowing your service container to persist its state across restarts of the container itself. If you don't do this it'll generate a new identity every time. You can put the actual data somewhere other than /var/lib/zerotier-one if you want. -To join a zerotier network you can use +To join a zerotier network you can use any of the following methods, or a combination thereof: + +1. The [`ZEROTIER_ONE_NETWORK_IDS`](#joinVar) environment variable. This, however, only works on first launch when the container's persistent store does not exist. + +2. The command line: + + ``` console + $ docker exec zerotier-one zerotier-cli join «networkID» + ``` + +3. Create an empty file with the network as name: + + ``` + /var/lib/zerotier-one/networks.d/«networkID».conf + ``` + + and then restart the container. + +You can also pass the following [Health Checking environment variables](#healthVars) to the `docker run` command: + +``` + --env ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» \ + --env ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 \ +``` + +#### compose file example + +``` yaml +--- + +services: + zerotier: + image: zyclonite/zerotier + container_name: zerotier-one + devices: + - /dev/net/tun + network_mode: host + volumes: + - '/var/lib/zerotier-one:/var/lib/zerotier-one' + cap_add: + - NET_ADMIN + - SYS_ADMIN + restart: unless-stopped + environment: + - TZ=Etc/UTC + - PUID=999 + - PGID=994 + # - ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» + # - ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» + # - ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 +``` + +#### Environment variables + +* `TZ` – timezone support. Example: + + ``` yaml + environment: + - TZ=Australia/Sydney + ``` + + Defaults to `Etc/UTC` if omitted. + +* `PUID` + `PGID` – user and group IDs for ownership of persistent store. Example: + + ``` yaml + environment: + - PUID=1000 + - PGID=1000 + ``` + + If omitted, `PUID` defaults to user ID 999, while `PGID` defaults to group ID 994. + + These variables are only used to ensure consistent ownership of persistent storage on each launch. They do not affect how the container *runs.* Absent a `user:` directive, the container runs as root and does not downgrade its privileges. + +* +`ZEROTIER_ONE_NETWORK_IDS` – a space-separated list of ZeroTier network IDs. + + This variable is *only* effective on first launch. There is no default if it is omitted. Examples: + + - to join a single network: + + ``` yaml + environment: + - ZEROTIER_ONE_NETWORK_IDS=aaaaaaaaaaaaaaaa + ``` + + This is the equivalent of running the following command after the container first starts: + + ``` + $ docker exec zerotier zerotier-cli join aaaaaaaaaaaaaaaa + ``` + + - to join a multiple networks: + + ``` yaml + environment: + - ZEROTIER_ONE_NETWORK_IDS=aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbb + ``` + + This is the equivalent of running the following commands after the container first starts: + + ``` + $ docker exec zerotier zerotier-cli join aaaaaaaaaaaaaaaa + $ docker exec zerotier zerotier-cli join bbbbbbbbbbbbbbbb + ``` + + It does not matter whether you use this environment variable or the `join` command, you still need to use ZeroTier Central to approve the host for each network it joins. + + +##### Health Checking variables + +The container (both client and router) runs a health-checking service. Two environment variables control its behaviour: + +* `ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS` – a space-separated list of ZeroTier network IDs. + + If this variable is present, the health check returns "healthy" providing that all listed network IDs meet the following criteria: + + - the network ID is known to the ZeroTier One client running in the container **and** has the status of "OK". It is the equivalent of running: + + ``` console + $ docker exec zerotier zerotier-cli get aaaaaaaaaaaaaaaa status + OK + ``` + + - If the network is known and "OK", the host's routing table is also checked for the presence of at least one direct route to the associated network interface. + + This variable takes precedence over checking for a minimum number of routes, which is described next. + +* `ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH` - an unsigned numeric value greater than zero. Defaults to 1. + + This variable is only active if the check for specific networks (described above) is omitted. + + This form of health check returns "healthy" providing that the host's routing table contains at least as many direct routes to ZeroTier-associated interfaces as are specified by this variable. + + If your container joins exactly one ZeroTier Network then the default value of 1 is appropriate. + + If your container joins more than one network then whether you should increase the value of this variable to match depends on considerations such as: + + - whether multiple networks represent alternate paths; or + - whether *you* still regard the service as healthy if only a subset of networks remain viable. + +###### First Launch considerations + +In a first-launch situation where the container's persistent store is initialised from scratch: + +1. The container will report "unhealthy" until at least one ZeroTier network has been joined **and** the host has been approved in ZeroTier Central. +2. If health-checking is configured to check for specific networks then the container will report "unhealthy" until **all** specified networks have been joined **and** the host has been approved to join **each** network in ZeroTier Central. + +###### Why Health Checking checks routes - docker exec zerotier-one zerotier-cli join 8056c2e21c000001 +The `zerotier-cli` command running inside the container may report that a ZeroTier Network is "OK" even if the host is aware that the associated network interface is not actually functioning. +An entry in the host's routing table is a pinnacle artefact. A direct route to an interface will only be added to the host's routing table if the interface exists and is in a functioning state. -or create an empty file with the network as name +Typically, each fully-functioning ZeroTier network that a client joins results in exactly one network interface and, therefore, exactly one direct route to that interface in the host's routing table. - /var/lib/zerotier-one/networks.d/8056c2e21c000001.conf +Route insertion and withdrawal is both sensitive to network conditions and efficient so using routes to confirm `zerotier-cli` status reports leads to fewer Type I (the container reporting "healthy" when it is not) and Type II (the container reporting "unhealthy" when it is not) errors. #### Router mode diff --git a/docker-compose-router.yml b/docker-compose-router.yml index c09319e..6e8263a 100644 --- a/docker-compose-router.yml +++ b/docker-compose-router.yml @@ -1,4 +1,5 @@ -version: '3' +--- + services: zerotier: image: "zyclonite/zerotier:router" @@ -20,4 +21,6 @@ services: - ZEROTIER_ONE_LOCAL_PHYS=eth0 - ZEROTIER_ONE_USE_IPTABLES_NFT=false - ZEROTIER_ONE_GATEWAY_MODE=inbound - # - ZEROTIER_ONE_NETWORK_IDS=yourNetworkID + # - ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» + # - ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» + # - ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 diff --git a/docker-compose.yml b/docker-compose.yml index 87292a7..0c3f5bd 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,4 +1,5 @@ -version: '3' +--- + services: zerotier: image: zyclonite/zerotier @@ -11,3 +12,11 @@ services: cap_add: - NET_ADMIN - SYS_ADMIN + restart: unless-stopped + environment: + - TZ=Etc/UTC + - PUID=999 + - PGID=994 + # - ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» + # - ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» + # - ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 diff --git a/scripts/entrypoint-router.sh b/scripts/entrypoint-router.sh index 825dfec..d2a14e8 100755 --- a/scripts/entrypoint-router.sh +++ b/scripts/entrypoint-router.sh @@ -11,15 +11,21 @@ NETWORKS_DIR="${CONFIG_DIR}/networks.d" # set up network auto-join if (a) the networks directory does not exist # and (b) the ZEROTIER_ONE_NETWORK_IDS environment variable is non-null. -if [ ! -d "${NETWORKS_DIR}" -a -n "${ZEROTIER_ONE_NETWORK_IDS}" ] ; then - echo "Assuming container first run." - mkdir -p "${NETWORKS_DIR}" - for NETWORK_ID in ${ZEROTIER_ONE_NETWORK_IDS} ; do - echo "Configuring auto-join of network ID: ${NETWORK_ID}" - touch "${NETWORKS_DIR}/${NETWORK_ID}.conf" - echo "You will need to authorize this host at:" - echo " https://my.zerotier.com/network/${NETWORK_ID}" - done +if [ ! -d "${NETWORKS_DIR}" ] ; then + echo "$(date) - assuming container first run." + if [ -n "${ZEROTIER_ONE_NETWORK_IDS}" ] ; then + mkdir -p "${NETWORKS_DIR}" + for NETWORK_ID in ${ZEROTIER_ONE_NETWORK_IDS} ; do + echo " Configuring auto-join of network ID: ${NETWORK_ID}" + touch "${NETWORKS_DIR}/${NETWORK_ID}.conf" + echo " You will need to authorize this host at:" + echo " https://my.zerotier.com/network/${NETWORK_ID}" + done + else + echo " ZEROTIER_ONE_NETWORK_IDS not set. You will need to join" + echo " networks using zerotier-cli, and then approve this" + echo " host in ZeroTier Central." + fi fi # make sure permissions are always as expected (self-repair) diff --git a/scripts/entrypoint.sh b/scripts/entrypoint.sh index df5bbf1..8964cb9 100755 --- a/scripts/entrypoint.sh +++ b/scripts/entrypoint.sh @@ -8,6 +8,8 @@ fi DEFAULT_PRIMARY_PORT=9993 DEFAULT_PORT_MAPPING_ENABLED=true DEFAULT_ALLOW_TCP_FALLBACK_RELAY=true +CONFIG_DIR="/var/lib/zerotier-one" +NETWORKS_DIR="${CONFIG_DIR}/networks.d" MANAGEMENT_NETWORKS="" if [ ! -z "$ZT_ALLOW_MANAGEMENT_FROM" ]; then @@ -19,7 +21,7 @@ if [ ! -z "$ZT_ALLOW_MANAGEMENT_FROM" ]; then done fi -if [ "$ZT_OVERRIDE_LOCAL_CONF" = 'true' ] || [ ! -f "/var/lib/zerotier-one/local.conf" ]; then +if [ "$ZT_OVERRIDE_LOCAL_CONF" = 'true' ] || [ ! -f "${CONFIG_DIR}/local.conf" ]; then echo "{ \"settings\": { \"primaryPort\": ${ZT_PRIMARY_PORT:-$DEFAULT_PRIMARY_PORT}, @@ -28,7 +30,35 @@ if [ "$ZT_OVERRIDE_LOCAL_CONF" = 'true' ] || [ ! -f "/var/lib/zerotier-one/local \"allowManagementFrom\": [${MANAGEMENT_NETWORKS}], \"allowTcpFallbackRelay\": ${ZT_ALLOW_TCP_FALLBACK_RELAY:-$DEFAULT_ALLOW_TCP_FALLBACK_RELAY} } - }" > /var/lib/zerotier-one/local.conf + }" > ${CONFIG_DIR}/local.conf fi +# set up network auto-join if (a) the networks directory does not exist +# and (b) the ZEROTIER_ONE_NETWORK_IDS environment variable is non-null. +if [ ! -d "${NETWORKS_DIR}" ] ; then + echo "$(date) - assuming container first run." + if [ -n "${ZEROTIER_ONE_NETWORK_IDS}" ] ; then + mkdir -p "${NETWORKS_DIR}" + for NETWORK_ID in ${ZEROTIER_ONE_NETWORK_IDS} ; do + echo " Configuring auto-join of network ID: ${NETWORK_ID}" + touch "${NETWORKS_DIR}/${NETWORK_ID}.conf" + echo " You will need to authorize this host at:" + echo " https://my.zerotier.com/network/${NETWORK_ID}" + done + else + echo " ZEROTIER_ONE_NETWORK_IDS not set. You will need to join" + echo " networks using zerotier-cli, and then approve this" + echo " host in ZeroTier Central." + fi +fi + +# make sure permissions are always as expected (self-repair) +PUID="${PUID:-"999"}" +PGID="${PGID:-"994"}" +if [ "$(id -u)" = '0' -a -d "${CONFIG_DIR}" ]; then + chown -Rc "${PUID}:${PGID}" "${CONFIG_DIR}" +fi + +echo "$(date) - launching ZeroTier-One in client mode" + exec "$@" diff --git a/scripts/healthcheck.sh b/scripts/healthcheck.sh new file mode 100755 index 0000000..4001c13 --- /dev/null +++ b/scripts/healthcheck.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env sh +#This health-check script is sponsored by PMGA TECH LLP + +#Exit Codes +# 0= Success +# 1= Failure + +#Environment Variables +# ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS= +# ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH= + +# minimum routes for health defaults to 1 route +ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=${ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH:-1} + +# Check if specified Networks are all Connected +if [[ -n "${ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS}" ]] ; then + + for network in $ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS; do + [[ "$(zerotier-cli get ${network} status)" = "OK" ]] || exit 1 + interface=$(zerotier-cli get ${network} portDeviceName) + routes=$(ip r | grep "dev ${interface}" | grep -cv "via") + [[ ${routes} -lt 1 ]] && exit 1 + done + +else # Check for Minimum Networks + + # count zerotier-associated direct routes + routes=$(ip r | grep "dev zt" | grep -cv "via") + + # sense less than minimum + [[ ${routes} -lt ${ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH} ]] && exit 1 + +fi + +exit 0 + From 389c2300eedf006adb62f259860da7032df053da Mon Sep 17 00:00:00 2001 From: Phill Kelley <34226495+Paraphraser@users.noreply.github.com> Date: Sun, 6 Apr 2025 12:01:25 +1000 Subject: [PATCH 2/2] `healthcheck.sh`: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Full rewrite, including copious comments explaining theory of operation. In essence, if a «networkID» is mentioned in (internal path): ``` /var/lib/zerotier-one/networks.d ``` then it should be matched by a route in the host's routing table: * Zero «networkID» = zero routes * One «networkID» = one route * Two «networkID» = two routes * ... Any mismatch causes the container to go unhealthy. From the perspective of the health-checking script, the question of *which* network is immaterial so there is no need to employ techniques such as iterating to discover *which* network is causing a problem. This is because the script has no way of communicating anything other than an exit status. Any `echo` statements it issues will not make it into the container's log. The simple presence/absence of a «networkID» in `networks.d` is taken to indicate which networks the user *intends* the container to join. There is no reliance on environment variables to propagate any health-checking information into the container. No other variables are introduced so the argument about naming conventions goes away. `Dockerfile`: 1. Removes `--start-interval` flag from `HEALTHCHECK` command. The flag was preventing `buildah` builds from succeeding. `README.md`: 1. Removes references to the following environment variables: * `ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS` * `ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH` 2. Rewrites explanation of health-checking. `README-router.md`: 1. Removes references to the following environment variables: * `ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS` * `ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH` Signed-off-by: Phill Kelley <34226495+Paraphraser@users.noreply.github.com> --- Dockerfile | 2 +- README-router.md | 2 - README.md | 66 +++++++++++---------------------- scripts/healthcheck.sh | 84 +++++++++++++++++++++++++++--------------- 4 files changed, 77 insertions(+), 77 deletions(-) diff --git a/Dockerfile b/Dockerfile index 2bbd601..7577bdd 100644 --- a/Dockerfile +++ b/Dockerfile @@ -41,5 +41,5 @@ ENTRYPOINT ["entrypoint.sh"] CMD ["-U"] -HEALTHCHECK --interval=60s --timeout=8s --retries=2 --start-period=60s --start-interval=3s \ +HEALTHCHECK --interval=60s --timeout=8s --retries=2 --start-period=60s \ CMD ["/bin/sh", "/usr/sbin/healthcheck.sh"] diff --git a/README-router.md b/README-router.md index 9d8e481..ddcd82e 100644 --- a/README-router.md +++ b/README-router.md @@ -86,8 +86,6 @@ services: - ZEROTIER_ONE_USE_IPTABLES_NFT=false - ZEROTIER_ONE_GATEWAY_MODE=inbound # - ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» - # - ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» - # - ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 ``` Note: diff --git a/README.md b/README.md index 691d0cd..e354968 100644 --- a/README.md +++ b/README.md @@ -46,13 +46,6 @@ To join a zerotier network you can use any of the following methods, or a combin and then restart the container. -You can also pass the following [Health Checking environment variables](#healthVars) to the `docker run` command: - -``` - --env ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» \ - --env ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 \ -``` - #### compose file example ``` yaml @@ -76,8 +69,6 @@ services: - PUID=999 - PGID=994 # - ZEROTIER_ONE_NETWORK_IDS=«yourDefaultNetworkID(s)» - # - ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS=«yourNetworkID(s)toCheck» - # - ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=1 ``` #### Environment variables @@ -128,7 +119,7 @@ services: - ZEROTIER_ONE_NETWORK_IDS=aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbb ``` - This is the equivalent of running the following commands after the container first starts: + This is the equivalent of running the following commands after the container starts for the first time: ``` $ docker exec zerotier zerotier-cli join aaaaaaaaaaaaaaaa @@ -137,45 +128,30 @@ services: It does not matter whether you use this environment variable or the `join` command, you still need to use ZeroTier Central to approve the host for each network it joins. - -##### Health Checking variables - -The container (both client and router) runs a health-checking service. Two environment variables control its behaviour: - -* `ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS` – a space-separated list of ZeroTier network IDs. - - If this variable is present, the health check returns "healthy" providing that all listed network IDs meet the following criteria: - - - the network ID is known to the ZeroTier One client running in the container **and** has the status of "OK". It is the equivalent of running: - - ``` console - $ docker exec zerotier zerotier-cli get aaaaaaaaaaaaaaaa status - OK - ``` + +##### Health Checking - - If the network is known and "OK", the host's routing table is also checked for the presence of at least one direct route to the associated network interface. +The container (both client and router) runs a health-checking service. It works like this: - This variable takes precedence over checking for a minimum number of routes, which is described next. +1. The (internal) path: -* `ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH` - an unsigned numeric value greater than zero. Defaults to 1. - - This variable is only active if the check for specific networks (described above) is omitted. - - This form of health check returns "healthy" providing that the host's routing table contains at least as many direct routes to ZeroTier-associated interfaces as are specified by this variable. - - If your container joins exactly one ZeroTier Network then the default value of 1 is appropriate. - - If your container joins more than one network then whether you should increase the value of this variable to match depends on considerations such as: - - - whether multiple networks represent alternate paths; or - - whether *you* still regard the service as healthy if only a subset of networks remain viable. + ``` + /var/lib/zerotier-one/networks.d/ + ``` + + contains zero or more files matching the pattern: + + ``` + «networkID».conf + ``` + + Each `.conf` file indicates a ZeroTier network which the container has been configured to join. The *count* of those files (which may be zero or more) is the "expected network count". -###### First Launch considerations +2. If the host's routing table does not contain the same number of direct routes to ZeroTier-associated interfaces as the *expected network count,* the container reports "unhealthy". -In a first-launch situation where the container's persistent store is initialised from scratch: +A container state of "unhealthy" only tells you that *something* is wrong. It does not tell you *what* is wrong. It is simply a hints to dig deeper. -1. The container will report "unhealthy" until at least one ZeroTier network has been joined **and** the host has been approved in ZeroTier Central. -2. If health-checking is configured to check for specific networks then the container will report "unhealthy" until **all** specified networks have been joined **and** the host has been approved to join **each** network in ZeroTier Central. +One of the most common reasons for the container to report "unhealthy" is that you have instructed the ZeroTier client (running inside the container) to join a network for which it is not authorised. ###### Why Health Checking checks routes @@ -183,9 +159,9 @@ The `zerotier-cli` command running inside the container may report that a ZeroTi An entry in the host's routing table is a pinnacle artefact. A direct route to an interface will only be added to the host's routing table if the interface exists and is in a functioning state. -Typically, each fully-functioning ZeroTier network that a client joins results in exactly one network interface and, therefore, exactly one direct route to that interface in the host's routing table. +Each fully-functioning ZeroTier network that a client joins results in exactly one network interface and, therefore, exactly one direct route to that interface in the host's routing table. -Route insertion and withdrawal is both sensitive to network conditions and efficient so using routes to confirm `zerotier-cli` status reports leads to fewer Type I (the container reporting "healthy" when it is not) and Type II (the container reporting "unhealthy" when it is not) errors. +Route insertion and withdrawal is both sensitive to network conditions and efficient so counting routes leads to fewer Type I (the container reporting "healthy" when it is not) and Type II (the container reporting "unhealthy" when it is not) errors. #### Router mode diff --git a/scripts/healthcheck.sh b/scripts/healthcheck.sh index 4001c13..0ce61d9 100755 --- a/scripts/healthcheck.sh +++ b/scripts/healthcheck.sh @@ -1,36 +1,62 @@ #!/usr/bin/env sh -#This health-check script is sponsored by PMGA TECH LLP -#Exit Codes -# 0= Success -# 1= Failure - -#Environment Variables -# ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS= -# ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH= - -# minimum routes for health defaults to 1 route -ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH=${ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH:-1} - -# Check if specified Networks are all Connected -if [[ -n "${ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS}" ]] ; then - - for network in $ZEROTIER_ONE_CHK_SPECIFIC_NETWORKS; do - [[ "$(zerotier-cli get ${network} status)" = "OK" ]] || exit 1 - interface=$(zerotier-cli get ${network} portDeviceName) - routes=$(ip r | grep "dev ${interface}" | grep -cv "via") - [[ ${routes} -lt 1 ]] && exit 1 - done - -else # Check for Minimum Networks - - # count zerotier-associated direct routes - routes=$(ip r | grep "dev zt" | grep -cv "via") - - # sense less than minimum - [[ ${routes} -lt ${ZEROTIER_ONE_CHK_MIN_ROUTES_FOR_HEALTH} ]] && exit 1 +# networks configuration directory is here (internal path) +NETWORKS_DIR="/var/lib/zerotier-one/networks.d" + +# On first run, the networks directory does not exist. It is +# not created until one of the following occurs: +# +# 1. The entrypoint script responds to the presence of: +# +# ZEROTIER_ONE_NETWORK_IDS=«networkID» {«networkID»} +# +# 2. The user mimics the action of the entrypoint script by +# executing: +# +# mkdir -p /var/lib/zerotier-one/networks.d +# touch /var/lib/zerotier-one/networks.d/«networkID».conf +# +# 3. The user issues an explicit join via zerotier-cli: +# +# zerotier-cli join «networkID» + +# Does the networks directory exist? +if [ -d "${NETWORKS_DIR}" ] ; then + + # yes! Each time a network is joined (however that occurs), + # two files are created: + # «networkID».conf and «networkID».local.conf + # Those files are also removed on an explicit leave. + # Accordingly, the presence of a «networkID».conf can be + # taken as expressing the user's intention that the container + # should be joined to that network. + + # Count the number of networks that should be joined + EXPECTED_NETWORKS=$( \ + ls "${NETWORKS_DIR}"/*.conf \ + | grep -v "local.conf" \ + | wc -l \ + ) + + # Each network which is joined, authorized and active results in + # an entry in the host's routing table. A direct route is a + # pinnacle indicator which will only be present if everything + # else is working properly. + + # Count the number of zerotier-associated direct routes + DIRECT_ROUTES=$( ip r | grep -c "dev zt.* scope link") + + # Irrespective of whether NETWORKS_DIR is empty or contains one + # or more «networkID».conf files, the number of expected networks + # should always equal the number of direct routes. Any mismatch + # means the container is in an unexpected state and, accordingly, + # is "unhealthy". + + # Sense any mismatch + [ ${EXPECTED_NETWORKS} -ne ${DIRECT_ROUTES} ] && exit 1 fi +# otherwise, the container is "healthy" exit 0