Missing essential files (pom.xml and App.java) required for Hop SDK configuration and successful pipeline execution

[shivanandullagaddi]

Hi Team,

While setting up Apache Hop SDK integration to execute a Hop pipeline programmatically, we are missing the key files required for successful configuration and execution.

Specifically, we need the following files:

pom.xml — containing the correct Maven dependencies for the Hop SDK libraries

App.java (or RunPipeline.java) — the main Java class responsible for initializing the Hop environment, loading the pipeline metadata, and executing the pipeline

Without these files, the project cannot resolve Hop SDK classes such as:

org.apache.hop.core.HopEnvironment
org.apache.hop.core.exception.HopException
org.apache.hop.pipeline.engine.IPipelineEngine
org.apache.hop.pipeline.engine.PipelineEngineFactory

This causes build and runtime errors (e.g., “The type org.apache.hop.core.exception.HopException cannot be resolved”).

Versions:

Apache Hop SDK version: 2.15.0

Java version: 17 (recommended)

Maven version: 3.9.x or later

Expected Outcome:

Once pom.xml and App.java are added and properly configured:

The project should compile successfully

Hop environment should initialize correctly

Pipeline execution through the Hop SDK should complete without errors

[bamaer]

https://hop.apache.org/dev-manual/latest/sdk/index.html

also: converting this to a discussion, because this is obviously not a bug.

[shivanandullagaddi]

Thank you for sharing the documentation. I’ve reviewed it, and while it explains the concepts and usage of the Hop SDK (such as pipeline initialization and execution), it does not include details about the Maven configuration (pom.xml) or the required dependencies for integrating the Hop SDK into a standalone Java application.

To proceed further with configuring and running a pipeline programmatically, we specifically need:

A working pom.xml file that includes the correct Hop SDK dependencies and repository references (e.g., for hop-core, hop-engine, hop-metadata-api, etc.) for version 2.15.0.

A reference Java example (App.java) showing how to initialize the Hop environment and execute a pipeline using these dependencies.

Without the Maven configuration details, the SDK-related classes such as HopEnvironment, HopException, and PipelineEngineFactory are unresolved during build.

Could you please share or point us to the official or sample pom.xml for Hop SDK integration (version 2.15.0)?

[bamaer]

The samples repository that is referred from the docs is here.
That should give you everything you need to get started. If that doesn't work, reach out and we can help you or even do this for you ( but that's commercial and not part of the community support we can provide).

[shivanandullagaddi]

Hi @bamaer

Thanks for sharing the Hop external plugins build. by using this i have modified my pom.xml and App.java files under the maven project to get connect to Hop, But while executing the App.java file that should call pipeline instead i am facing the errors can you help to resolve it and move on for the successful pipeline callout from the Hop Sdk.

Errors:

Initializing Apache Hop Environment...
Hop configuration file not found, not serializing: C:\Users\shiva\OneDrive\Desktop\demo\config\hop-config.json
SLF4J: Class path contains multiple SLF4J providers.
SLF4J: Found provider [org.slf4j.nop.NOPServiceProvider@10e8a8f5]
SLF4J: Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@4310f342]
SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual provider is of type [org.slf4j.nop.NOPServiceProvider@10e8a8f5]
? Loading pipeline: C:/Users/shiva/Downloads/apache-hop-client-2.15.0/hop/config/projects/default/pipelines/pipeline_example_1.hpl
? Hop Exception:
API error: metadata provider can't be null. When loading a pipeline Hop needs to be able to reference external metadata objects

org.apache.hop.core.exception.HopXmlException:
API error: metadata provider can't be null. When loading a pipeline Hop needs to be able to reference external metadata objects

This is the pom.xml file

<modelVersion>4.0.0</modelVersion>
<groupId>com.shiva.hopdemo</groupId>
<artifactId>demo</artifactId>
<version>1.0-SNAPSHOT</version>
<name>HopSDKDemo</name>

<properties>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<repositories>
    <!-- Apache releases repository for Hop artifacts -->
    <repository>
        <id>apache-releases</id>
        <url>https://repository.apache.org/content/repositories/releases/</url>
    </repository>
    <!-- Snapshots if needed -->
    <repository>
        <id>apache-snapshots</id>
        <url>https://repository.apache.org/content/repositories/snapshots/</url>
    </repository>
</repositories>

<dependencies>
    <!-- Hop Core -->
    <dependency>
        <groupId>org.apache.hop</groupId>
        <artifactId>hop-core</artifactId>
        <version>2.15.0</version>
    </dependency>

    <!-- Hop Engine -->
    <dependency>
        <groupId>org.apache.hop</groupId>
        <artifactId>hop-engine</artifactId>
        <version>2.15.0</version>
    </dependency>

    <!-- Hop Client (optional, contains useful runtime classes) -->
    <dependency>
        <groupId>org.apache.hop</groupId>
        <artifactId>hop-client</artifactId>
        <version>2.15.0</version>
    </dependency>

    <!-- Logging -->
    <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>1.4.12</version>
    </dependency>

    <!-- JUnit for testing -->
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-api</artifactId>
        <version>5.10.0</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <!-- Compiler plugin -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.13.0</version>
            <configuration>
                <release>17</release>
            </configuration>
        </plugin>

        <!-- Exec plugin to run App.java -->
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>3.1.0</version>
            <configuration>
                <mainClass>com.shiva.hopdemo.App</mainClass>
            </configuration>
        </plugin>
    </plugins>
</build>

Here is the App.java file

package com.shiva.hopdemo;

import org.apache.hop.core.HopEnvironment;
import org.apache.hop.core.Result;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.logging.LogLevel;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.engine.IPipelineEngine;
import org.apache.hop.pipeline.engine.PipelineEngineFactory;

import java.io.File;

public class App {

public static void main(String[] args) {
    try {
        System.out.println("Initializing Apache Hop Environment...");

        // Initialize Hop Environment
        HopEnvironment.init();

        // Create variables
        Variables variables = new Variables();

        // Specify pipeline file path
        String pipelineFilePath = "pipelines/pipeline_example_1.hpl";
       
        // Check if file exists
        File pipelineFile = new File(pipelineFilePath);
        if (!pipelineFile.exists()) {
            // Try absolute path as fallback
            pipelineFilePath = "C:/Users/shiva/Downloads/apache-hop-client-2.15.0/hop/config/projects/default/pipelines/pipeline_example_1.hpl";
            pipelineFile = new File(pipelineFilePath);
            if (!pipelineFile.exists()) {
                System.err.println(" Pipeline file not found: " + pipelineFilePath);
                System.err.println(" Please create a sample pipeline first using Hop GUI");
                return;
            }
        }

        System.out.println(" Loading pipeline: " + pipelineFilePath);

        // Load pipeline metadata
        PipelineMeta pipelineMeta = new PipelineMeta(pipelineFilePath, null, variables);

        // Create pipeline engine
        IPipelineEngine<PipelineMeta> pipelineEngine = PipelineEngineFactory.createPipelineEngine(
            variables,
            "local",
            null,
            pipelineMeta
        );

        // Set logging level
        pipelineEngine.setLogLevel(LogLevel.BASIC);

        System.out.println(" Starting pipeline execution...");

        // Execute pipeline
        pipelineEngine.execute();

        // Wait for completion
        pipelineEngine.waitUntilFinished();

        // Get results
        Result result = pipelineEngine.getResult();

        // Display results
        System.out.println("========================================");
        System.out.println(" PIPELINE EXECUTION RESULTS");
        System.out.println("========================================");
        System.out.println(" Errors: " + result.getNrErrors());
        System.out.println(" Lines Read: " + result.getNrLinesRead());
        System.out.println(" Lines Written: " + result.getNrLinesWritten());
        //System.out.println(" Execution Time: " + result.getExecutionDuration() + "ms");
       
        if (result.getNrErrors() == 0) {
            System.out.println(" Result: SUCCESS");
        } else {
            System.out.println(" Result: FAILED ");
        }
        System.out.println("========================================");

    } catch (HopException e) {
        System.err.println("Hop Exception: " + e.getMessage());
        e.printStackTrace();
    } catch (Exception e) {
        System.err.println("Unexpected error: " + e.getMessage());
        e.printStackTrace();
    }
}

}

[hansva]

You did not add the error message.
You will also need to include each and every transform that your pipeline executes and its dependencies.

[shivanandullagaddi]

Hi @hansva this was the problem when i run the App.java

mvn clean compile exec:java "-Dexec.mainClass=com.shiva.hopdemo.App"
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------------< com.shiva.hopdemo:demo >-----------------------
[INFO] Building HopSDKDemo 1.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- clean:3.2.0:clean (default-clean) @ demo ---
[INFO] Deleting C:\Users\shiva\OneDrive\Desktop\demo\target
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ demo ---
[INFO] skip non existing resourceDirectory C:\Users\shiva\OneDrive\Desktop\demo\src\main\resources
[INFO]
[INFO] --- compiler:3.13.0:compile (default-compile) @ demo ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 1 source file with javac [debug target 17] to target\classes
[INFO]
[INFO] --- exec:3.1.0:java (default-cli) @ demo ---
Initializing Apache Hop Environment...
Hop configuration file not found, not serializing: C:\Users\shiva\OneDrive\Desktop\demo\config\hop-config.json
SLF4J: Class path contains multiple SLF4J providers.
SLF4J: Found provider [org.slf4j.nop.NOPServiceProvider@10e8a8f5]
SLF4J: Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@4310f342]
SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual provider is of type [org.slf4j.nop.NOPServiceProvider@10e8a8f5]
? Loading pipeline: C:/Users/shiva/Downloads/apache-hop-client-2.15.0/hop/config/projects/default/pipelines/pipeline_example_1.hpl
? Hop Exception:
API error: metadata provider can't be null. When loading a pipeline Hop needs to be able to reference external metadata objects

org.apache.hop.core.exception.HopXmlException:
API error: metadata provider can't be null. When loading a pipeline Hop needs to be able to reference external metadata objects

    at org.apache.hop.pipeline.PipelineMeta.<init>(PipelineMeta.java:1658)
    at com.shiva.hopdemo.App.main(App.java:45)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:279)
    at java.base/java.lang.Thread.run(Thread.java:840)  

[WARNING] thread Thread[Timer-0,5,com.shiva.hopdemo.App] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[Timer-0,5,com.shiva.hopdemo.App] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=com.shiva.hopdemo.App,maxpri=10]
java.lang.IllegalThreadStateException
at java.lang.ThreadGroup.destroy (ThreadGroup.java:803)
at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:319)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:569)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.340 s
[INFO] Finished at: 2025-11-06T15:58:51+05:30
[INFO] ------------------------------------------------------------------------

And coming to this point: You will also need to include each and every transform that your pipeline executes and its dependencies.

where we can have it? any sample guide