Gentle transition to the marketplace

[hansva]

Hi All,

Starting from 2.19, Apache Hop will have a marketplace.

This will be a great way to reduce our overall base client size, and external parties will be able to provide their own plugins via the marketplace system. I think this will further drive Apache Hop to become a great place for ETL development.

I would like to propose easing our users into this system; this will also give us the time to receive feedback and fix potential bugs.
My proposal for the 2.19 release would be to only exclude some of the more "niche" plugins that take up a lot of space.

• Beam
• Spark
• Drools
• Tika
• Cassandra
• Detectlanguage
• Stanford
• Databricks

This brings the client down to around 400MB, while keeping most of our functionality.
I think removing all features such as AWS, Google, and Azure support in this initial version will confuse our users.

Let me know what you think.
Hans

[sramazzina]

Wonderful idea. These are very specific plugins that most of the time aren't used in normal use cases

[nadment]

My opinion would be to make all the 'tech' plugins and the 'beam' and 'spark' engines optional via the marketplace to start with.
Possibly some plugins that take up a lot of space, on a case-by-case basis.
I would add back 'Mail' and 'Rest', which are currently optional.

[hansva]

My main concern is getting all this tested before the release, we have no tests where only one of the tech plugins is being installed. We could be missing libraries that are just "there" at this point in time

[bamaer]

I agree with @nadment as a longer term goal, but I share @hansva's concern right now. I think we should minimize the blast radius for this initial release.

[nadment]

I also agree that it's best to proceed slowly with plugins that aren't used very often; there may be some side effects.

[sramazzina]

I agree with @nadment that Mail plugin must be part of the distro and not available by default through the marketplace. I think almost anyone uses it regularly. I was trying some of our flows with latest 2.19 and the majority of them can't run because of the missing of that plugin. Apart from that, it's a wonderful experience now. Thanks anyone for the hard work made to reach this point

[zoomingrocket]

What about hop-run? How will that function or will this be just for development GUI version? Currently, to optimize hop-run, we manually take out the libs not required, so a similar mechanism will help overall job execution performance.

[hansva]

hop-run uses the same folder as the UI, so it's the same set of plugins.
If you are talking about the Docker image, then it currently contains everything. Other image flavours will be added later but there is no time to get this done and tested before we should be releasing.

[zoomingrocket]

Got it. No, I am just considering the standalone hop-run (non docker), which we use to fork and execute multiple jobs. So the plugin required will reside in a one-time setup like hop-config.json for server based executions?

[hansva]

You can add an environment file which will specify which plugins need to be installed: https://hop.apache.org/manual/next/hop-tools/hop-marketplace.html

[steewel]

...the Docker image, then it currently contains everything.

Was just looking for that piece of information, tyvm

[sramazzina]

One minor point. We must choose a different toolbar icon for the marketplace because is the same as plugins' explorer and this makes a lot of confusion IMHO

[nadment]

Icon proposal

Artifact

Repository

Marketplace

[hansva]

Looks good @nadment

[liosc]

Very nice. Just a consideration: the cart icon gives me the impression of focusing on the economic transaction, a place where at the end you will spend money. An icon like this

, for example, gives a more "broad" and "welcoming" approach.

[zoomingrocket]

Agree, more than the marketplace, it's a plugin store. Unless the intention in the future is to have commercial offerings via this framework

[nadment]

Better like this

[dave-csc]

Hi all,

I still have to try the new marketplace (will hear from me soon, I think 😅). But I already have an idea to make the smoothest transition possible.

If I got correctly, most plugins have been detached from the main installation to the new marketplace. So, when upgrading to Hop 2.19, this can lead to "broken" stuff here and there when using transforms / actions / other features related to those plugins.

To mitigate this, a warning like "You must install the plugin [plugin name] to use [feature name]" could be shown, along with the procedure to do so if desired:

• in Hop GUI with a redirection to the marketplace (maybe the Eclipse IDE marketplace can be inspirational)
• in Hop Run with the command to invoke to install such plugin (similar to what happens with uninstalled commands in Ubuntu)

At this stage, this is useful for making the migration to Hop 2.19 (and future versions) easier.

In the future, this feedback can be useful to align Hop distributions when passing workflows / pipelines / configurations to someone who hasn't the required plugin installed.

[bamaer]

Most plugins are still in the main installation, and a warning is already shown in hop gui and (optionally iirc) in hop run.