[Feature Request]: Docker options to install marketplace plugins

Feature Request: Add Marketplace Plugin Auto-Installation to `apache/hop` Docker Container

1. Problem Statement & Motivation

When deploying Apache Hop in containerized environments (Kubernetes, Docker Compose, Airflow, CI/CD runners), pipelines frequently require plugins hosted in corporate artifactories (JFrog Artifactory, Sonatype Nexus, AWS CodeArtifact) or community repositories (such as Project Data Hopper’s Nexus at https://repository.data-hopper.com/repository/hop-community-plugins/).

Currently, installing a custom plugin in the standard apache/hop container requires one of the following:

Building a custom Docker image inheriting from apache/hop, defeating the convenience of using standard official images with mounted projects.
Writing a custom entrypoint script (HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH).
Using indirect repository specs: e.g., running hop marketplace apply -f <spec-url> or hop marketplace repo import <yaml-url>.

As seen with specifications like hop-marketplace-repo.yaml, these files contain extensive catalog and GUI metadata (kind, browse: true, categories, description, homepage, minHopVersion, installPath). In the context of a headless runtime container, all that extra metadata is unnecessary. The container runtime only needs to know:

Which plugin(s) to install (coordinates / artifact IDs).
Where to fetch them (the Maven/Artifactory repository base URL).
Credentials (if the artifactory requires authentication).

Apache Hop already has an established precedent for JDBC drivers: HOP_DRIVERS_DOWNLOAD and HOP_DRIVERS_MAVEN_REPO. We should provide an equivalent mechanism for marketplace plugins.

2. Proposed Container Environment Variables

Following the convention established by HOP_DRIVERS_*:

Environment Variable	Description	Example
`HOP_PLUGINS_DOWNLOAD`	Comma-separated list of plugin coordinates to install before Hop starts. Supports short names, `artifactId`, `artifactId:version`, or `groupId:artifactId:version`.	`hopper-edw:0.10.0` `org.hopper:hopper-edw:0.10.0,hop-tech-parquet`
`HOP_PLUGINS_MAVEN_REPO`	Optional corporate Artifactory / Nexus base Maven repository URL. If omitted, uses the default Hop marketplace repositories (ASF & Central).	`https://repository.data-hopper.com/repository/hop-community-plugins/` `https://artifactory.corp/artifactory/hop-plugins/`
`HOP_PLUGINS_REPO_ID`	Optional identifier for the repository in Hop configuration (default: `corporate-repo`).	`data-hopper-community`
`HOP_PLUGINS_REPO_AUTH_TYPE`	Optional authentication type: `auto` (default), `basic`, or `token`.	`basic`, `token`
`HOP_PLUGINS_REPO_USERNAME`	Optional username for authenticated repositories (can also fall back to `HOP_MARKETPLACE_USERNAME`).	`svc-hop-runner`
`HOP_PLUGINS_REPO_PASSWORD`	Optional password or bearer token (can also fall back to `HOP_MARKETPLACE_PASSWORD`).	`${ARTIFACTORY_TOKEN}`

(Optional declarative fallback):

HOP_PLUGINS_ENV_FILE: If a team does already have a hop-env.yaml or hop-marketplace-repo.yaml available via HTTP(S) or local volume, setting this variable could trigger hop marketplace apply -f "${HOP_PLUGINS_ENV_FILE}".

3. Implementation in Container Scripts

Both load-and-execute.sh and run-web.sh would add a helper function install_marketplace_plugins that executes before Hop starts (alongside install_jdbc_drivers).

Proposed Function: `install_marketplace_plugins()`

# Download Marketplace plugins on container start, before Hop is launched.
# Driven by environment variables:
#   HOP_PLUGINS_DOWNLOAD          comma-separated plugin coordinates:
#                                 e.g. "hopper-edw:0.10.0,org.apache.hop:hop-tech-parquet:2.19.0"
#   HOP_PLUGINS_MAVEN_REPO        optional Maven/Artifactory repository base URL
#   HOP_PLUGINS_REPO_ID           optional repository ID (default: "corporate-repo")
#   HOP_PLUGINS_REPO_USERNAME     optional username for the repository
#   HOP_PLUGINS_REPO_PASSWORD     optional password / token for the repository
#   HOP_PLUGINS_REPO_AUTH_TYPE    optional auth type: auto (default), basic, token
#   HOP_PLUGINS_ENV_FILE          optional URL or path to a hop-env.yaml / hop-marketplace-repo.yaml
install_marketplace_plugins() {
  local plugins="${HOP_PLUGINS_DOWNLOAD:-}"
  local env_file="${HOP_PLUGINS_ENV_FILE:-}"

  # 1. Alternative: Apply full install spec if specified
  if [ -n "${env_file}" ]; then
    log "Applying Hop marketplace environment file: ${env_file}"
    if ! "${DEPLOYMENT_PATH}"/hop marketplace apply -f "${env_file}"; then
      log "Error: failed to apply marketplace environment file '${env_file}'"
      exitWithCode 8
    fi
  fi

  if [ -z "${plugins}" ]; then
    return 0
  fi

  local repo_url="${HOP_PLUGINS_MAVEN_REPO:-}"
  local repo_id="${HOP_PLUGINS_REPO_ID:-corporate-repo}"

  # 2. Register repository if a custom URL is provided
  if [ -n "${repo_url}" ]; then
    log "Configuring marketplace repository '${repo_id}' (${repo_url})"
    local repo_args=("--id=${repo_id}" "--url=${repo_url}" "--primary")
    [ -n "${HOP_PLUGINS_REPO_USERNAME:-}" ]  && repo_args+=("--username=${HOP_PLUGINS_REPO_USERNAME}")
    [ -n "${HOP_PLUGINS_REPO_PASSWORD:-}" ]  && repo_args+=("--password=${HOP_PLUGINS_REPO_PASSWORD}")
    [ -n "${HOP_PLUGINS_REPO_AUTH_TYPE:-}" ] && repo_args+=("--auth-type=${HOP_PLUGINS_REPO_AUTH_TYPE}")

    # Register in hop-config.json (or update if already exists)
    if ! "${DEPLOYMENT_PATH}"/hop marketplace repo add "${repo_args[@]}"; then
      log "Warning: failed to add repository '${repo_id}' (may already exist, proceeding to install)"
    fi
  fi

  # 3. Install each requested plugin
  log "Installing marketplace plugins: ${plugins}"
  local spec
  for spec in ${plugins//,/ }; do
    spec="$(echo "${spec}" | tr -d '[:space:]')"
    [ -z "${spec}" ] && continue

    log "Installing marketplace plugin '${spec}'"
    local install_args=("${spec}")
    if [ -n "${repo_url}" ]; then
      install_args+=("--repo=${repo_id}")
    fi

    # shellcheck disable=SC2086
    if ! "${DEPLOYMENT_PATH}"/hop marketplace install "${install_args[@]}"; then
      log "Error: failed to install marketplace plugin '${spec}'"
      exitWithCode 8
    fi
  done
}

Execution Location

In load-and-execute.sh:

# Download requested JDBC drivers (HOP_DRIVERS_DOWNLOAD) before Hop starts.
install_jdbc_drivers

# Download requested Marketplace plugins (HOP_PLUGINS_DOWNLOAD) before Hop starts.
install_marketplace_plugins

4. Complementary CLI Improvement: Add `--repo-url` to `hop marketplace install`

In MarketplaceCommand.java:

Currently, hop marketplace install only accepts --repo <repo-id>, which requires the repository to be registered beforehand in hop-config.json.
In contrast, DriverInstallCommand.java accepts --repo <url> (a direct repository URL).

Suggestion for Marketplace CLI:
Add an option --repo-url <url> to hop marketplace install (or permit passing a full URL to --repo).
When --repo-url is passed:

hop marketplace install org.hopper:hopper-edw:0.10.0 --repo-url https://repository.data-hopper.com/repository/hop-community-plugins/

The installer uses an ad-hoc, in-memory MarketplaceRepository directly without needing a separate repo add step in hop-config.json. This makes marketplace plugin installation completely stateless.

5. Usage Examples

Example 1: Installing Data Hopper EDW from community repository

docker run -it --rm \
  --env HOP_PLUGINS_DOWNLOAD="org.hopper:hopper-edw:0.10.0" \
  --env HOP_PLUGINS_MAVEN_REPO="https://repository.data-hopper.com/repository/hop-community-plugins/" \
  --env HOP_PROJECT_FOLDER=/project \
  --env HOP_PROJECT_NAME=edw-project \
  --env HOP_RUN_CONFIG=local \
  --env HOP_FILE_PATH='${PROJECT_HOME}/load_data_vault.hpl' \
  --mount type=bind,source=/path/to/project,target=/project \
  apache/hop:latest

Example 2: Installing multiple plugins from an authenticated corporate Artifactory

docker run -it --rm \
  --env HOP_PLUGINS_DOWNLOAD="com.mycorp:custom-pipeline-transform:1.4.0,org.hopper:hopper-edw:0.10.0" \
  --env HOP_PLUGINS_MAVEN_REPO="https://artifactory.mycompany.corp/artifactory/hop-plugins-release/" \
  --env HOP_PLUGINS_REPO_USERNAME="ci-runner" \
  --env HOP_PLUGINS_REPO_PASSWORD="${ARTIFACTORY_TOKEN}" \
  --env HOP_PLUGINS_REPO_AUTH_TYPE="token" \
  --env HOP_PROJECT_FOLDER=/project \
  --env HOP_PROJECT_NAME=etl \
  --env HOP_FILE_PATH='${PROJECT_HOME}/main.hwf' \
  --env HOP_RUN_CONFIG=local \
  --mount type=bind,source=/opt/etl,target=/project \
  apache/hop:latest

6. Documentation Section

Add a new section Downloading Marketplace Plugins in docker-container.adoc directly following the Downloading JDBC drivers section, documenting:

The HOP_PLUGINS_DOWNLOAD and HOP_PLUGINS_MAVEN_REPO variables.
Coordinate formats (groupId:artifactId:version, artifactId:version, and short names).
Authentication parameters for private repositories.

Issue Priority

Priority: 2

Issue Component

Component: Containers, Component: Marketplace

Issue apache#8722 : Skip repeat marketplace installs and stop putting…

Issue apache#8722 : Install marketplace plugins from the web image at…

Issue apache#8722 : Document the stock apache/hop container

Issue #8722 : Docker options to install marketplace plugins (#8723)

Navigation Menu

Repository navigation

[Feature Request]: Docker options to install marketplace plugins #8722

Feature Request: Add Marketplace Plugin Auto-Installation to `apache/hop` Docker Container

1. Problem Statement & Motivation

2. Proposed Container Environment Variables

3. Implementation in Container Scripts

Proposed Function: `install_marketplace_plugins()`

Execution Location

4. Complementary CLI Improvement: Add `--repo-url` to `hop marketplace install`

5. Usage Examples

Example 1: Installing Data Hopper EDW from community repository

Example 2: Installing multiple plugins from an authenticated corporate Artifactory

6. Documentation Section

Issue Priority

Issue Component

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature Request]: Docker options to install marketplace plugins #8722

Description

Feature Request: Add Marketplace Plugin Auto-Installation to apache/hop Docker Container

1. Problem Statement & Motivation

2. Proposed Container Environment Variables

3. Implementation in Container Scripts

Proposed Function: install_marketplace_plugins()

Execution Location

4. Complementary CLI Improvement: Add --repo-url to hop marketplace install

5. Usage Examples

Example 1: Installing Data Hopper EDW from community repository

Example 2: Installing multiple plugins from an authenticated corporate Artifactory

6. Documentation Section

Issue Priority

Issue Component

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Feature Request: Add Marketplace Plugin Auto-Installation to `apache/hop` Docker Container

Proposed Function: `install_marketplace_plugins()`

4. Complementary CLI Improvement: Add `--repo-url` to `hop marketplace install`