Repository navigation
Issue #8946 : Write local Spark files on Windows without winutils - #8951
Open
mattcasters wants to merge 1 commit into
Open
mattcasters wants to merge 1 commit into
mattcasters wants to merge 1 commit into
Conversation
mattcasters
marked this pull request as ready for review
October 15, 2026 21:03
Contributor
Author
|
The original issue seems solved but I can't run the Spark pipeline on my corporate laptop to verify. |
|
Hi Matt, I tested the PR on Windows with Spark 
The Help buttons work correctly, though. I may have missed something in the setup, but I wanted to share my test results. Reading the CSV works, and the pipeline runs successfully when I disable the hop to Spark file output; the error occurs only when writing the output. 
|
Contributor
Author
|
Thanks for testing! Your log is missing the line "HADOOP_HOME and hadoop.home.dir are unset. Local Spark paths will keep the permissions of this process instead of calling winutils." The fix prints that line when it kicks in, and your stack trace goes through Hadoop's own RawLocalFileSystem.setPermission rather than the new HopWindowsLocalFileSystem. So I think the Spark engine plugin in your install is still the old build. Could you check that the jar under plugins/engines/spark was built from this branch (commit 34363e6), restart Hop, and try again? |
Hadoop's local file system shells out to winutils.exe when Spark file output creates the output directory. On Windows that fails with "HADOOP_HOME and hadoop.home.dir are unset" when neither is set. When Hop creates the Spark session and winutils is missing, skip those permission changes. An explicit spark.hadoop.fs.file.impl in the run configuration still wins.
cursor
Bot
force-pushed
the
issue-8946
branch
from
October 21, 2026 10:44
34363e6 to
2dce275
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Spark file output on Windows
local[*]fails while Hadoop creates the output directory.RawLocalFileSystem.setPermissionrunswinutils.exe, and that lookup throwsHADOOP_HOME and hadoop.home.dir are unsetwhen neither is set. The same path is still in main.When Hop creates the Spark session on Windows and winutils is not installed, the session uses a local file system that skips those permission changes. Files keep the permissions of the Hop process.
hdfs://ands3a://are unchanged. IfHADOOP_HOMEalready points atbin/winutils.exe, Hadoop's own local file system stays in place. Aspark.hadoop.fs.file.implline in the run configuration is applied after this default and wins.The help-button 404 does not apply to main. Spark file input and Spark file output were undocumented in 2.19, so
/manual/latest/404s. The pages are in main and published at/manual/next/. A 2.20 snapshot build opens that version.Fixes #8946
Test plan
HopWindowsLocalFileSystemTest(2) on Linux: decision table, and mkdir/create/setPermission/read through the wrapperhop-engines-sparklocal[*]Spark file output to afile:///directory withHADOOP_HOMEunset