Spark: Fix DeleteOrphanFilesSparkAction sibling path matching bug - #16999
Spark: Fix DeleteOrphanFilesSparkAction sibling path matching bug#16999CoderNasim2023 wants to merge 1 commit into
Conversation
This commit fixes issue apache#16493 where file_list_view was scoped using raw string prefix matching, which allowed sibling paths to fall inside orphan cleanup (e.g. s3://bucket/table-backup matched s3://bucket/table). Added a trailing slash to the location before prefix matching in filteredCompareToFileList to ensure precise directory scoping.
| Dataset<Row> files = compareToFileList; | ||
| if (location != null) { | ||
| files = files.filter(files.col(FILE_PATH).startsWith(location)); | ||
| String strippedLocation = LocationUtil.stripTrailingSlash(location); |
There was a problem hiding this comment.
Why don't we just always add the separator here? What does the check below get us?
There was a problem hiding this comment.
LocationUtil.stripTrailingSlash(location) + LocationUtil.PATH_SEPARATOR;
|
[#16498][(https://github.com//pull/16498) seems to be solving the same issue. @CoderNasim2023 you should coordinate with @wombatu-kun to see how to get this through in a singular effort rather than duplicating the work. |
|
Welcome, and thanks for tackling this, @CoderNasim2023! This bug is already being fixed in #16498 (open since May 21), with the same approach - normalizing To avoid splitting the effort, would you be up for reviewing or testing #16498 instead? That would help it land. I'll leave it to the maintainers to decide whether to close this one as a duplicate. |
|
Okk Mr @wombatu-kun please do response on this That would help . I'll leave it to the maintainers to decide whether to close this one and if its solve dthe issue then please ro merge it @RussellSpitzer |
4 participants
**This commit fixes issue #16493 where file_list_view was scoped using raw string prefix matching, which allowed sibling paths to fall inside orphan cleanup (e.g. s3://bucket/table-backup matched s3://bucket/table).
Added a trailing slash to the location before prefix matching in filteredCompareToFileList to ensure precise directory scoping.**
AI Disclosure