Catch the full (or partial) log with "Workflow log" mechanism

[bellumic]

Hi all!

I followed the guidelines in the following page ( https://hop.apache.org/manual/latest/how-to-guides/logging-workflow-log.html ) about logging and I successfully logged the workflow in the DB.

Although, it seems mostly logging the "metrics" part. My question now is: is it possible to somehow catch the log as well? I need to save it.

Thanks in advance for any help.
Michael

[dave-csc]

Hi @bellumic,

check discussion #4896 to get an idea of how to get the log text (with limitations regarding when your workflow becomes too complex).

The pipeline I uploaded saves data in a file and optionally sends it via mail, you should adapt it to save the log in a database.

Hope this helps :)

[bellumic]

Hi Dave,

The process was already set and logs correctly logging into the database, but I couldn't find the log string, which is actually saved in "workflowLogging" field. I just missed it, my bad!

Thanks for your reply.
Michael

[bellumic]

@dave-csc Since I see you already went through pretty much what I am now, I would like to ask you a question regarding logging and deployment.

Have you tried to deploy your project to any platform by any chance?

I'm now running it in a docker server, but I have an issue with the triggering of the logging pipeline:
It seems that the pipeline responsible to catch the workflow log and insert into the database, doesn't get triggered when deployed and ran into the server. About setup it should be all correct.

it seems there is something internal to HOP that triggers the logging pipeline, but I don't know what and therefore I can't replicate it.

Have you ever faced this kind of issue?

Thanks in advance,
Michael

[hansva]

that's a know issue #3368

[bellumic]

Ah ok I see, so for the moment it's not possible to work with it.

Do you know if there is any known workaround for it?

Thanks in advance,
Michael

[dave-csc]

Hi @bellumic,

I currently don't use Docker for my deployments, it would add a layer of complexity without adding much value (since all the remaining resources - files, database, REST API, etc. - must be "ported" into the container somehow). My deployment server is actually a Git-based replica of the development client.

You probably want to check if you have configured Hop correctly to get the logs saved. You need to configure a Workflow log metadata (in the Metadata perspective), mine is configured as follows:

• the flag Enabled is on
• Pipeline executed to capture logging: use the full path of your logging pipeline. You'll probably need to enter them from the Docker perspective (i.e. the right part of the -v flag), rather than the local perspective (the left part)
• check the flag Execute at the end of the workflow, uncheck the other two (start and during): this is because you will get only the complete log at the end, rather than a starter or incremental one
• Capture output of the following workflows: I set this to ${PROJECT_HOME}/main.hwf. This way you just need to name the main workflow main.hwf in any of your projects (even new ones), and you'll get the logging platform "for free"

Hope this helps :)