[mattcasters]

Text File Output kept each write in a 5000-byte buffer, so slow input did not show up in the file until that buffer filled or the transform closed the file. HOP_FILE_OUTPUT_MAX_STREAM_LIFE was already documented as the flush interval, but it defaulted to 0 and the elapsed-time check subtracted the clock in the wrong order, so a configured interval never flushed.

The comparison is now the time since the last flush. The variable defaults to 5000 milliseconds, and a value of 0 (what existing Hop configurations already store) uses that same default. A positive value still sets the interval in milliseconds. Rows written after a flush mark the file dirty again, so the next interval is not skipped, and closing a file flushes the buffer even when that flag is clear. Compression output streams pass flush() through to the underlying stream.

TextFileOutputFlushTest and TextFileOutputTest passed in hop-transform-textfile. CompressionOutputStreamTest passed in hop-engine. A full reactor build and the Hop GUI were not run.

fixes #2406

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
• Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

• I hereby declare this contribution to be licensed under the Apache License Version 2.0, January 2004
• In any other case, please file an Apache Individual Contributor License Agreement.

[bamaer]

Blockers

• Files can be left unclosed at the end of the run. Both flushOpenFiles implementations close a stream only if it's dirty (TextFileOutputData.java:214-216 and :406-408). Before this PR the interval flush never ran, so open files always stayed dirty. Now FileStream.flush() clears the flag, and dispose() → close() → flushOpenFiles(true) skips every stream that hasn't been written to since the last interval flush.

  • Single file: this happens whenever the interval flush runs after the last row, which with slow input (the case [Task]: Flush Text File Output #2406 is about) is almost every run.
  • Filename taken from a field: every file not written to since the last interval flush stays open.
  • Result:
    • gzip without a trailer, zip without a central directory
    • S3, Azure and HDFS objects never committed
    • open handles on local files

  TextFileOutputFlushTest never ends the stream, so it doesn't catch this.

  Suggested fix: in both implementations, flush when dirty, but close whenever closeAfterFlush && isOpen(), whatever the dirty flag says. Add a test where the interval flush runs after the last row, the stream then ends, and the file is closed and readable (for example a gzip file that decompresses).

Should fix

• The new flush can't be switched off. getFlushInterval() (TextFileOutput.java:298) maps 0, negative values and non-numbers to 5000. The interval flush never ran on main, so every existing Text File Output, including high-volume and remote targets, now flushes every 5 seconds with no way to opt out. Existing hop-config.json files store 0 for this variable.

  Suggested fix: let a negative value (for example -1) disable the interval flush, and document it in Const, variables.adoc and textfileoutput.adoc.

• Every row can scan all open files when the filename comes from a field. markCurrentFileDirty() (TextFileOutput.java:831) is called for each row. When rows alternate between files, getLastStream() isn't the current file, so it walks every open stream (up to HOP_FILE_OUTPUT_MAX_STREAM_COUNT, 1024 by default). initFileStreamWriter already marks the stream dirty on that path.

  Suggested fix: keep a reference to the current FileStream when data.writer is assigned, and mark that one dirty directly.