How to Programmatically Create or Update Hop Pipelines (.hpl) via API or SDK?

[shivanandullagaddi]

Hi Team,

I want to discuss a use case involving dynamic pipeline creation in Apache Hop, and I am trying to understand whether Hop currently supports it, or if there is an intended best-practice approach.

Create or update Hop pipeline definitions (.hpl) programmatically, using either:

Hop Server REST API
Java Hop SDK
Any officially supported API
Or recommended architectural approach

Background
I reviewed the available Hop Server REST API documentation:

[https://hop.apache.org/manual/latest/hop-server/rest-api.html]
The current REST APIs appear to focus on:
Registering a pipeline
Running a pipeline
Monitoring pipeline execution
Removing pipelines
Getting pipeline images/status

But I could not find any endpoint or SDK function that can:

Create a new pipeline definition programmatically
Modify an existing .hpl file
Add transforms/hops dynamically
Save pipeline definitions back into a project or file system

Use Case

I am integrating Apache Hop with Salesforce Apex, where pipelines must be:

Constructed dynamically (structure changes per user)
Modified at runtime
Saved into the Hop project
Executed via Hop Server

Right now, the only option appears to be:
Manually building a pipeline in Hop GUI
Exporting it
Sending the XML through /addPipeline with execution configs

But this does not let me programmatically create or modify the pipeline itself.

Thanks

[bamaer]

This is from one of the first lines on the Apache Hop home page: Visual development enables developers to be more productive than they can be through code.
API-based development may come to Apache Hop at some point, but that definitely won't happen any time soon. Until that changes, you'll have to design your pipelines in Hop Gui/Web. Once that's done, you have all the flexibility to execute if/where/when/how you like...

[shivanandullagaddi]

Hi @bamaer

Thanks for the clarification
can't we create pipeline with Hop sdk configurations?

[bamaer]

There is a Java SDK and the source code is available, so in theory, everything is possible. However, please realize that writing a user interface, even a simple one, easily takes a couple of weeks (if not months) of work.

[dmainou]

Hey are I reckon you are barking at the wrong tree.
HOP is a graphical interphase on purpose.
In saying that, you can build templated that can run based on config files that enable you to acomplaish the same thing for an endless number of scenarios.
e.g.
figure out the shape of a file . Load said file to a data store.

[campen]

@shivanandullagaddi - you can use a Salesforce api endpoint to get objects, then use a different Salesforce endpoint to get the metadata. From there you would need to build a translation table to convert the Salesforce data types to Hop data types. But then you can build an api call with all of the elements in the call for an object to export Salesforce data via the Salesforce api to another source. This all using ETL Metadata injection.

You could even programmatically build tables based on the current metadata (or add new fields). As mentioned above you would need to build the framework.

[sramazzina]

Hi @shivanandullagaddi. In the last months I built a POC were a set of workflows and pipelines that together was used to build a complete end to end process. The definitions of these workflows and pipelines was given by using a set of high level metadata descriptors given through excel files (but they maybe whatever you want).

That was totally possible and the results was really good because we generated a fully functional Hop process. For sure, you can do what you are trying to do using Hop objects, the same as I do, but you must know some things related to the internals' of Hop.