Import-DbaParquet - Release the parquet file when the import is done - #10543
Merged
Conversation
Get-ParquetReader hands a file stream to ParquetReader.CreateAsync without passing leaveStreamOpen, and that parameter defaults to true. Disposing the reader therefore left the stream open and the imported file locked until the garbage collector got around to finalizing it, which is an arbitrary amount of time after the import finished. The visible symptom was a test that could not clean up after itself: on PowerShell 7 the Pester run of Import-DbaParquet failed roughly every other time in Remove-TestDrive with "The process cannot access the file staging.ecdc_parquet_test.parquet because it is being used by another process", while all 17 tests passed. For users the effect is worse than a failing cleanup, because a file cannot be deleted, moved or overwritten after importing it until a collection happens to run. leaveStreamOpen is now passed as false, so disposing the reader closes the file. (do Import-DbaParquet) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10542.
Import-DbaParquetkept the imported file locked after the import had finished, until the garbage collectorgot around to finalizing the stream. How long that took varied from run to run, so a file could not reliably
be deleted, moved or overwritten right after importing it.
Cause
Get-ParquetReaderopens a file stream and passes it to Parquet.NET without saying what should happen to it:The overload in play is
leaveStreamOpendefaults to true, so disposing the reader deliberately leaves the stream open. Thecommand does dispose the reader, but nothing disposes the stream, so the handle lived until finalization.
leaveStreamOpenis now passed asfalse. All four arguments are passed explicitly because the streamoverload can only be selected by its full signature - with fewer arguments the
string filePathoverload is acandidate too.
Testing
The symptom that made this visible was a test that could not clean up after itself: on PowerShell 7 the Pester
run failed in
Remove-TestDrivewithThe process cannot access the file 'staging.ecdc_parquet_test.parquet' because it is being used by another process, while all 17 tests passed. It failed about every other run,which is why it read as test flakiness rather than a defect.
Import-DbaParquet.Tests.ps1on PowerShell 7.4, five consecutive runsImport-DbaParquet.Tests.ps1on Windows PowerShell 5.1No test change was needed - the test was right and the command was wrong.
Note
This touches the same file as #10538 and #10541 but a different part of it. All three merge cleanly, verified
by merging them locally and running the tests above against the combination.
馃 Generated with Claude Code