"Execute a process" transform limits

[bellumic]

Hi All,

I use "Execute a process" to execute a curl command in my process and I'm encountering some issues.
In specific the component hangs and therefore HOP does it as well. I know this is an open issue and somebody proposed a fix for it in 'Ideas' category.
What I am wondering is the reason why the component hangs. Is there any limitation I should be aware of? For example I get a Json as response, is there any length limit for it? It doesn't seems a long one to me (5.38kb) and 180 lines (beautified in Postman)
I tried to run the curl in other platforms (Postman and Git bash) and it works correctly.
The "curl" command in Git bash is actually the same I use in HOP:

Also the same process works for other URLs (the URL base is the same, what changes is the ID at the end of the URL)
So I'm wondering what is actually blocking it.

Thanks in advance,
Michael

[bamaer]

so this same command works in the transform in some cases, but hangs with one or more specific ids? If that's the case, are there any special characters in the id? Do your failing commands work if you write them to the logs and copy/paste them in a terminal?

[bamaer]

Don't know how many rows you're processing, but if it's a significant amount, Hop could be running out of available memory. if it's still on the default of 2048m, one thing you could try is increasing the -Xmx parameter in hop-gui.bat and hop-run.bat (or set it in HOP_OPTIONS)

[bellumic]

I've done it, but no changes. Also normally when HOP runs out of memory it completely goes in crash.
In this case it's just the component that keeps running forever:

I also tried the shell component in workflow, dumping the response to a file. That one works :

[bellumic]

This is the length of the API response which fail in "Execute a process" transform:

1. 4747
2. 4836
3. 6438
4. 8638
5. 4411
6. 5772
7. 4895
8. 4466
9. 4495
10. 4543

The max length of correct responses out of 142 calls is 3758.

I'm wondering if in the code of the transform there is any parameter/variable with max length 4000 or so that is limiting all the response length. Would that be possible?
And in that case would that be possible to fix, even on my own (I don't know Java well enough)?

[hansva]

It could be that there is a limit.
If you can create a simple reproduction path using a file, public api or a script generating a string of a specific length then feel free to create a ticket so someone can take a look at it.
Workaround could be to have your command write to a file and then pick up that file in a pipeline.

[bellumic]

@hansva Thanks for your reply. Unluckly I cannot provide a reproduction of it at the moment. I will switch back to the "files" solution in with the shell transform in the Workoflow.