Fix Nexus Ubuntu apt proxy pinned to bionic, breaking package installs on jammy VMs

CHANGELOG.md

@@ -17,6 +17,7 @@ ENHANCEMENTS:
 * Update the version of `super-linter` used in the `build_validation_develop` workflow to 8.7.0 ([#4957](https://github.com/microsoft/AzureTRE/issues/4957))
 
 BUG FIXES:
+* Fix Guacamole Linux VM bootstrap failures on Ubuntu 22.04 (jammy): serve the `ubuntu`, `ubuntu-security`, `docker`, and `microsoft-apt` Nexus apt feeds through raw proxy repositories that pass any Ubuntu release straight through, so one repo per upstream works for every current and future distribution without per-release configuration (Nexus does not support apt group repositories). Migrate existing repos to the new format automatically on upgrade. Add cloud-init wait to Guacamole Linux VM so deployment reports failure if cloud-init errors. (`sonatype-nexus` 3.11.0, `tre-service-guacamole-linuxvm` 1.4.4) ([#4992](https://github.com/microsoft/AzureTRE/issues/4992), [#4540](https://github.com/microsoft/AzureTRE/issues/4540))
 * Fix workspace deletion when backup is enabled for the base, unrestricted and airlock-import-review workspaces by adding a `delete_backups_on_uninstall` flag and a pre-teardown backup cleanup (`remove_backup.sh`) that stops protection and either deletes or retains the Recovery Services Vault, so deletion works with Azure secure-by-default soft delete ([#4962](https://github.com/microsoft/AzureTRE/issues/4962))
 * Fix Nexus shared service security: fetch admin password from Key Vault at runtime via managed identity (IMDS) instead of embedding it in the VM Run Command script content. Fix `deploy_nexus_container.sh` short-circuit path to fail loudly if the container does not start. (`sonatype-nexus` 3.10.0) ([#4983](https://github.com/microsoft/AzureTRE/pull/4983))
 * Fix UI TypeScript deprecation warning by updating `moduleResolution` to `bundler` in `tsconfig.json`. ([#4968](https://github.com/microsoft/AzureTRE/issues/4968))

docs/tre-templates/shared-services/nexus.md

@@ -86,16 +86,16 @@ Nexus Shared Service requires access to resources outside of the Azure TRE VNET.
 | PyPI | PyPI | [https://pypi.org/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/pypi/` | Allow use of pip commands. |
 | Conda | conda | [https://repo.anaconda.com/pkgs] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/conda-repo/` | Configure conda to have access to default conda packages. |
 | Conda Mirror | conda | [https://conda.anaconda.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/conda-mirror/` | Configure conda to have access to conda mirror packages. |
-| Docker | apt | [https://download.docker.com/linux/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker/` | Install Docker via apt on Linux systems. |
+| Docker | raw | [https://download.docker.com/linux/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker/` | Install Docker via apt on Linux systems. A raw proxy passes through any Ubuntu release, so one repo serves all supported distributions. |
 | Docker GPG | raw | [https://download.docker.com/linux/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker-public-key/` | Provide public key to sign apt source for above Docker apt. |
 | Docker Hub | docker | [https://registry-1.docker.io] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker-hub/` | Provide docker access to public images repo. |
-| Ubuntu Packages | apt | [http://archive.ubuntu.com/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/ubuntu/` | Provide access to Ubuntu apt packages on Ubuntu systems. |
-| Ubuntu Security Packages | apt | [http://security.ubuntu.com/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/ubuntu-security/` | Provide access to Ubuntu Security apt packages on Ubuntu systems. |
+| Ubuntu Packages | raw | [http://archive.ubuntu.com/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/ubuntu/` | Ubuntu apt packages. A raw proxy passes through any Ubuntu release, so one repo serves all supported distributions. |
+| Ubuntu Security | raw | [http://security.ubuntu.com/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/ubuntu-security/` | Ubuntu security apt packages. A raw proxy passes through any Ubuntu release, so one repo serves all supported distributions. |
 | Almalinux | yum | [https://repo.almalinux.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/almalinux` | Install Almalinux packages |
 | R-Proxy | r | [https://cran.r-project.org/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/r-proxy` | Provide access to CRAN packages for R |
 | R-Studio Download | raw | [https://download1.rstudio.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/r-studio-download` | Provide access to download R Studio |
 | Fedora Project | yum | [https://download-ib01.fedoraproject.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/fedoraproject` | Install Fedora Project Linux packages |
-| Microsoft Apt | apt | [https://packages.microsoft.com] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-apt` | Provide access to Microsoft Apt packages |
+| Microsoft Apt | raw | [https://packages.microsoft.com] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-apt` | Microsoft apt packages. The remoteUrl is the host root; per-product paths and the Ubuntu release are supplied by the client sources list. A raw proxy passes these through for any release. |
 | Microsoft Keys | raw | [https://packages.microsoft.com/keys/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-keys` | Provide access to Microsoft keys |
 | Microsoft Yum | yum | [https://packages.microsoft.com/yumrepos] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-yum` | Provide access to Microsoft Yum packages |
 | Microsoft Download | raw | [https://download.microsoft.com/download] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-download` | Provide access to Microsoft Downloads |

templates/shared_services/sonatype-nexus-vm/porter.yaml

@@ -1,7 +1,7 @@
 ---
 schemaVersion: 1.0.0
 name: tre-shared-service-sonatype-nexus
-version: 3.10.0
+version: 3.11.0
 description: "A Sonatype Nexus shared service"
 dockerfile: Dockerfile.tmpl
 registry: azuretre

templates/shared_services/sonatype-nexus-vm/scripts/configure_nexus_repos.sh

@@ -124,62 +124,104 @@ if [ "$anon_status_code" -ne 200 ]; then
 fi
 
 echo "Configuring Nexus repositories..."
-# Create or update a proxy for each .json file so modified configurations are
-# applied to an existing Nexus instance without needing to recreate it.
+# Create or update each repo so modified configurations are applied to an existing
+# Nexus instance without needing to recreate it.
+
+configure_repo() {
+  local file="$1"
+  local create="$2"
+  local update="$3"
+  local pass="$4"
+  local response code body
+  # Try to create the repository first.
+  response=$(curl -u admin:"$pass" -XPOST \
+    "$create" \
+    -H 'accept: application/json' \
+    -H 'Content-Type: application/json' \
+    -d @"$file" \
+    -k -s -w $'\n%{http_code}')
+  code=${response##*$'\n'}
+  body=${response%$'\n'*}
+  echo "Response received from Nexus when creating repository: $code"
+  if [ "$code" -eq 201 ]; then
+    return 0
+  fi
+  # If it already exists, update it so configuration changes are applied.
+  code=$(curl -iu admin:"$pass" -XPUT \
+    "$update" \
+    -H 'accept: application/json' \
+    -H 'Content-Type: application/json' \
+    -d @"$file" \
+    -k -s -w "%{http_code}" -o /dev/null)
+  echo "Response received from Nexus when updating repository: $code"
+  if [ "$code" -eq 200 ] || [ "$code" -eq 202 ] || [ "$code" -eq 204 ]; then
+    return 0
+  fi
+  # A proxy repo whose remoteUrl no longer resolves fails Nexus 3.94+ restore
+  # validation, leaving it in a failed state: the name is reserved (create
+  # returns "Name is already used") but it cannot be updated (404). It can't
+  # be reconciled via the API, so warn and skip rather than blocking the whole
+  # upgrade on an already-broken repository.
+  if [ "$code" -eq 404 ] && printf '%s' "$body" | grep -qi 'already used'; then
+    echo "WARNING - Repository is in a failed state in Nexus and cannot be updated (its proxy remote URL may be unreachable). Skipping."
+    return 0
+  fi
+  return 1
+}
+
+configure_one_repo() {
+  local filename="$1"
+  echo "Found config file: $filename. Sending to Nexus..."
+  local base_type repo_type repo_name create_url update_url
+  base_type=$( jq .baseType "$filename" | sed 's/"//g')
+  repo_type=$( jq .repoType "$filename" | sed 's/"//g')
+  repo_name=$( jq .name "$filename" | sed 's/"//g')
+
+  # A repo's format/type cannot be changed in place, so if one already exists
+  # under this name with a different format or type (e.g. an old apt proxy being
+  # replaced by a raw proxy), delete it first so it can be recreated.
+  if ! retry_with_backoff delete_if_type_mismatch "$repo_name" "$base_type" "$repo_type" "$NEXUS_ADMIN_PASSWORD"; then
+    echo "ERROR - Could not remove conflicting repo '$repo_name'"
+    exit 1
+  fi
+
+  create_url="http://localhost/service/rest/v1/repositories/$base_type/$repo_type"
+  update_url="$create_url/$repo_name"
+
+  if ! retry_with_backoff configure_repo "$filename" "$create_url" "$update_url" "$NEXUS_ADMIN_PASSWORD"; then
+    echo "ERROR - Timeout while trying to configure $repo_name"
+    exit 1
+  fi
+}
+
+# Delete an existing repository if its format or type differs from the desired
+# configuration. Nexus won't let a repo change format/type via update, so an
+# in-place migration (for example an apt proxy replaced by a raw proxy under the
+# same name) requires removing the old repo before recreating it.
+delete_if_type_mismatch() {
+  local repo_name="$1"
+  local want_format="$2"
+  local want_type="$3"
+  local pass="$4"
+  local existing
+  existing=$(curl -s -u admin:"$pass" \
+    'http://localhost/service/rest/v1/repositories' \
+    -H 'accept: application/json' \
+    -k | jq -r --arg name "$repo_name" '.[] | select(.name == $name) | "\(.format) \(.type)"')
+  if [ -z "$existing" ] || [ "$existing" = "$want_format $want_type" ]; then
+    return 0
+  fi
+  echo "Repo '$repo_name' exists as '$existing' but config wants '$want_format $want_type' — deleting it so it can be recreated..."
+  local code
+  code=$(curl -s -u admin:"$pass" -XDELETE \
+    "http://localhost/service/rest/v1/repositories/$repo_name" \
+    -k -w "%{http_code}" -o /dev/null)
+  echo "Delete response for '$repo_name': $code"
+  [ "$code" -eq 204 ] || [ "$code" -eq 200 ]
+}
+
 for filename in "$(dirname "${BASH_SOURCE[0]}")"/nexus_repos_config/*.json; do
-    echo "Found config file: $filename. Sending to Nexus..."
-    base_type=$( jq .baseType "$filename" | sed 's/"//g')
-    repo_type=$( jq .repoType "$filename" | sed 's/"//g')
-    repo_name=$( jq .name "$filename" | sed 's/"//g')
-    create_url="http://localhost/service/rest/v1/repositories/$base_type/$repo_type"
-    update_url="$create_url/$repo_name"
-
-    configure_repo() {
-      local file="$1"
-      local create="$2"
-      local update="$3"
-      local pass="$4"
-      local response code body
-      # Try to create the repository first.
-      response=$(curl -u admin:"$pass" -XPOST \
-        "$create" \
-        -H 'accept: application/json' \
-        -H 'Content-Type: application/json' \
-        -d @"$file" \
-        -k -s -w $'\n%{http_code}')
-      code=${response##*$'\n'}
-      body=${response%$'\n'*}
-      echo "Response received from Nexus when creating repository: $code"
-      if [ "$code" -eq 201 ]; then
-        return 0
-      fi
-      # If it already exists, update it so configuration changes are applied.
-      code=$(curl -iu admin:"$pass" -XPUT \
-        "$update" \
-        -H 'accept: application/json' \
-        -H 'Content-Type: application/json' \
-        -d @"$file" \
-        -k -s -w "%{http_code}" -o /dev/null)
-      echo "Response received from Nexus when updating repository: $code"
-      if [ "$code" -eq 200 ] || [ "$code" -eq 202 ] || [ "$code" -eq 204 ]; then
-        return 0
-      fi
-      # A proxy repo whose remoteUrl no longer resolves fails Nexus 3.94+ restore
-      # validation, leaving it in a failed state: the name is reserved (create
-      # returns "Name is already used") but it cannot be updated (404). It can't
-      # be reconciled via the API, so warn and skip rather than blocking the whole
-      # upgrade on an already-broken repository.
-      if [ "$code" -eq 404 ] && printf '%s' "$body" | grep -qi 'already used'; then
-        echo "WARNING - Repository $repo_name is in a failed state in Nexus and cannot be updated (its proxy remote URL may be unreachable). Skipping."
-        return 0
-      fi
-      return 1
-    }
-
-    if ! retry_with_backoff configure_repo "$filename" "$create_url" "$update_url" "$NEXUS_ADMIN_PASSWORD"; then
-      echo "ERROR - Timeout while trying to configure $repo_name"
-      exit 1
-    fi
+    configure_one_repo "$filename"
 done
 
 echo 'Configuring realms...'

templates/shared_services/sonatype-nexus-vm/scripts/nexus_repos_config/docker_proxy_conf.json

@@ -1,36 +1,34 @@
 {
-  "name": "docker",
-  "online": true,
-  "storage": {
-      "blobStoreName": "default",
-      "strictContentTypeValidation": true,
-      "write_policy": "ALLOW"
-  },
-  "proxy": {
-      "remoteUrl": "https://download.docker.com/linux/ubuntu/",
-      "contentMaxAge": 1440,
-      "metadataMaxAge": 1440
-  },
-  "negativeCache": {
-      "enabled": true,
-      "timeToLive": 1440
-  },
-  "httpClient": {
-      "blocked": false,
-      "autoBlock": false,
-      "connection": {
-          "retries": 0,
-          "userAgentSuffix": "string",
-          "timeout": 60,
-          "enableCircularRedirects": false,
-          "enableCookies": false,
-          "useTrustStore": false
-      }
-  },
-  "apt": {
-      "distribution": "bionic",
-      "flat": false
-  },
-  "baseType": "apt",
-  "repoType": "proxy"
-}
+    "name": "docker",
+    "online": true,
+    "storage": {
+        "blobStoreName": "default",
+        "strictContentTypeValidation": false
+    },
+    "proxy": {
+        "remoteUrl": "https://download.docker.com/linux/ubuntu/",
+        "contentMaxAge": 1440,
+        "metadataMaxAge": 1440
+    },
+    "negativeCache": {
+        "enabled": true,
+        "timeToLive": 1440
+    },
+    "httpClient": {
+        "blocked": false,
+        "autoBlock": false,
+        "connection": {
+            "retries": 0,
+            "userAgentSuffix": "string",
+            "timeout": 60,
+            "enableCircularRedirects": false,
+            "enableCookies": false,
+            "useTrustStore": false
+        }
+    },
+    "raw": {
+        "contentDisposition": "ATTACHMENT"
+    },
+    "baseType": "raw",
+    "repoType": "proxy"
+}

templates/shared_services/sonatype-nexus-vm/scripts/nexus_repos_config/microsoft_apt_proxy_conf.json

@@ -3,8 +3,7 @@
     "online": true,
     "storage": {
         "blobStoreName": "default",
-        "strictContentTypeValidation": true,
-        "write_policy": "ALLOW"
+        "strictContentTypeValidation": false
     },
     "proxy": {
         "remoteUrl": "https://packages.microsoft.com",
@@ -27,10 +26,9 @@
             "useTrustStore": false
         }
     },
-    "apt": {
-      "distribution": "bionic",
-      "flat": false
+    "raw": {
+        "contentDisposition": "ATTACHMENT"
     },
-    "baseType": "apt",
+    "baseType": "raw",
     "repoType": "proxy"
-}
+}

templates/shared_services/sonatype-nexus-vm/scripts/nexus_repos_config/ubuntu_proxy_conf.json

@@ -1,36 +1,34 @@
 {
-  "name": "ubuntu",
-  "online": true,
-  "storage": {
-      "blobStoreName": "default",
-      "strictContentTypeValidation": true,
-      "write_policy": "ALLOW"
-  },
-  "proxy": {
-      "remoteUrl": "http://archive.ubuntu.com/ubuntu/",
-      "contentMaxAge": 1440,
-      "metadataMaxAge": 1440
-  },
-  "negativeCache": {
-      "enabled": true,
-      "timeToLive": 1440
-  },
-  "httpClient": {
-      "blocked": false,
-      "autoBlock": false,
-      "connection": {
-          "retries": 0,
-          "userAgentSuffix": "string",
-          "timeout": 60,
-          "enableCircularRedirects": false,
-          "enableCookies": false,
-          "useTrustStore": false
-      }
-  },
-  "apt": {
-      "distribution": "bionic",
-      "flat": false
-  },
-  "baseType": "apt",
-  "repoType": "proxy"
-}
+    "name": "ubuntu",
+    "online": true,
+    "storage": {
+        "blobStoreName": "default",
+        "strictContentTypeValidation": false
+    },
+    "proxy": {
+        "remoteUrl": "http://archive.ubuntu.com/ubuntu/",
+        "contentMaxAge": 1440,
+        "metadataMaxAge": 1440
+    },
+    "negativeCache": {
+        "enabled": true,
+        "timeToLive": 1440
+    },
+    "httpClient": {
+        "blocked": false,
+        "autoBlock": false,
+        "connection": {
+            "retries": 0,
+            "userAgentSuffix": "string",
+            "timeout": 60,
+            "enableCircularRedirects": false,
+            "enableCookies": false,
+            "useTrustStore": false
+        }
+    },
+    "raw": {
+        "contentDisposition": "ATTACHMENT"
+    },
+    "baseType": "raw",
+    "repoType": "proxy"
+}