How could I change a variable defined in JVM scope in JavaScript Action?

[LittleSaya]

I have a set of workflow and pipeline which looks like this:

The problem is the Repeat Action can not be halted, because the condition can not be met.

I need to initialize the variable before the Repeat Action, and change that variable in a deeply nested workflow, but it seems that parent_workflow.setVariable() can not change the variable defined in JVM scope.

What is the proper manner to change that variable? Or is there a workaround to this?

more background:

I'm transferring data between an FTP tunnel, there are about 30 tables, and they are transferred one-by-one. I can't wait until all tables are transferred, so I use a loop to detect and process tables as soon as they arrive, and I use a variable processed_tables to record how many tables have been processed, when processed_tables hits the total number of tables, end the Repeat action.

[hansva]

I would have to create a test for this.
But I don't think you need a JVM variable, I think a set variable with scope Valid in the root workflow should do the trick. Have you tried this approach? I don't really see why the logic here should be different from our repeat action sample.

[LittleSaya]

I tried as you said to set the variable in root workflow, but it did not work, I made a small project for this:
default.zip

[bamaer]

if I understand this correctly, you set the variable value in the JavaScript action. The earlier actions in the workflow (like your repeat) can't pick that value up. I would try to move the entire processing into the loop in a child workflow that is called from the repeat action and set the variable in that child workflow.

[LittleSaya]

You mean calling the parent_workflow.setVariable() in upload_session.hwf? That did work, but in each repeat I also need to iterate through a list of table names to "detect" tables, I used pipeline to do this, so I have to call parent_workflow.setVariable() inside a workflow which is nested inside that pipeline.

[ossDataEngineer]

@bamaer - I have a similar requirement, I want to repeat a workflow execution until END_LOOP variable is set to value = fileDownloadComplete. I additionally want to exit out of this repeat if var_retry_attempt>3 (that is retry the file downloading 3 times and abort)

I set var_retry_attempt=0 in parent-workflow with scope (Valid in the root workflow). Try to set the variable to value 1 in child workflow (i.e repeat workflow) but the value is not reflected and the repeat loop continues to execute infinitely

[ossDataEngineer]

One solution:

Parent Workflow - Set Variable var_retry_attempt=0 (Valid in current workflow)
Child Workflow (where the file download happens) on repeat and END_LOOP is set after successful attempt and retry count<3 is checked
Child pipeline - responsible to increment var_retry_attempt using Set variables transform (Valid in root workflow)

@bamaer and @LittleSaya - This worked, which indicates that Variable changing is only possible via pipelines and not via Workflow.

[ossDataEngineer]

Second solution:

Parent Workflow - Set Variable var_retry_attempt=0 (Valid in current workflow)
Child Workflow (where the file download happens) on repeat and END_LOOP is set after successful attempt and retry count<3 is checked, Javascript action with following snippet:

//'Keep variables after substitutions' must be checked on Repeat Workflow shape in parent workflow, for this to work
curr_retry_count = parseInt(parent_workflow.getVariable("var_retry_attempt"))
new_retry_count = curr_retry_count+1
parent_workflow.setVariable("var_retry_attempt", new_retry_count.toString())

true

[LittleSaya]

Ok, found a solution to my specific requirement which may not fit in other circumstances:

Create a file for each processed table, and count the number of files in that folder, for example:

If I have 3 tables: table_aaa, table_bbb, table_ccc, I create one file for each of them:

${PROJECT_HOME}/count/table_aaa
${PROJECT_HOME}/count/table_bbb
${PROJECT_HOME}/count/table_ccc

then I count the number of file in ${PROJECT_HOME}/count.

Basically this moves the variable out of Hop. A more general approach may be invoking a shell script to calculate/write the variable then using Set Variable to read the variable.