gh-150880: Normalize Windows scandir wildcard paths

[zainnadeem786]

Summary

On Windows, os.listdir() and os.scandir() append a wildcard before calling FindFirstFileW().

For paths with a trailing space, this changes the path from a final-component trailing-space case into an internal path component, so Windows no longer trims the space. This makes listdir() and scandir() behave differently from APIs such as open(), os.stat(), and os.path.exists().

This PR normalizes non-extended Windows base paths with GetFullPathNameW() before appending the wildcard used for enumeration.

Extended paths beginning with \\?\ continue to bypass normalization.

Changes

• Normalize only the wildcard path used by Windows listdir() and scandir().
• Preserve existing DirEntry.path construction behavior.
• Preserve extended-path semantics.
• Add regression coverage for trailing-space directory paths.

Tests

PCbuild\build.bat -p x64 -d
PCbuild\amd64\python_d.exe -m unittest -v test.test_os.test_os.TestScandir.test_windows_trailing_space_path
PCbuild\amd64\python_d.exe -m test test_os
PCbuild\amd64\python_d.exe -m test test_pathlib test_ntpath
git diff --check

Close #150880

[miss-islington-app]

Thanks @zainnadeem786 for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

[bedevere-app]

GH-152963 is a backport of this pull request to the 3.15 branch.

[bedevere-app]

GH-152964 is a backport of this pull request to the 3.14 branch.

[bedevere-app]

GH-152965 is a backport of this pull request to the 3.13 branch.

[vstinner]

os.scandir() doesn't normalize the entries path. Is it a deliberate choice? The space in the last 'C:\\victor\\python\\main \\.azure-pipelines' result is surprising.

>>> p=os.getcwd()
>>> list(os.scandir(p))[0].path
'C:\\victor\\python\\main\\.azure-pipelines'
>>> list(os.scandir(p+' '))[0].path
'C:\\victor\\python\\main \\.azure-pipelines'

[zooba]

Unintentional. If that's done in native code, it probably needs to be updated to use the same join function

[zainnadeem786]

Thanks for pointing this out, @vstinner.

I'll reproduce this on my side and investigate where DirEntry.path is being constructed. If it turns out to be a separate issue from the wildcard normalization addressed in this PR, I'll prepare a follow-up fix and submit a separate PR.

Thanks for taking another look!

[vstinner]

os_scandir_impl() should normalize the path first.

[zooba]

os_scandir_impl() should normalize the path first.

This is probably true, actually, as it'll make sure that a relative path doesn't get calculated to the wrong value if chdir is called during the scan.

[zainnadeem786]

Thanks, @vstinner and @zooba.

I investigated this further on Windows. The root cause appears to be that GH-152906 normalizes only the temporary wildcard path passed to FindFirstFileW(), while the original path remains stored in the ScandirIterator and is later used to build DirEntry.path.

I also confirmed the chdir() concern: os.scandir("dir") opens the directory immediately, but the later DirEntry.path can still be relative to whatever the current working directory is when the entry is consumed.

Based on your comments, the right follow-up seems to be normalizing the scandir base path once inside os_scandir_impl() before storing it in the iterator, instead of only adjusting DirEntry.path.

One compatibility point I noticed: this would make relative Windows DirEntry.path values absolute, which differs from the current documentation saying the path is only absolute if the scandir() argument was absolute.

If that behavior change is acceptable for Windows in order to match the actual directory opened and avoid the chdir() issue, I can prepare a follow-up PR with tests for both the trailing-space case and the chdir() case.

[zooba]

the current documentation saying the path is only absolute if the scandir() argument was absolute.

Oh, it says that? Yeah, we can't do that change then.

I don't want to encode platform-specific normalisation rules in Python (we'll never get them 100% right). If we can't use an OS API to make the path suitable for the OS, then it's on the user. It probably just has to be a failure in this case, though we can update the docs to say that other than the filename, the path is preserved from the provided argument, and the end result may not be valid (e.g. the CWD changes or the result of joining the path isn't valid).

[zainnadeem786]

Thanks for the clarification, @zooba.

That makes sense. I agree that preserving the documented API semantics is more important than trying to mirror Windows normalization rules here.

I'll take a look at preparing a documentation update to clarify that, apart from the filename, DirEntry.path preserves the original argument and may not remain valid if the working directory changes or if the supplied path itself isn't valid for later path construction.

Thanks again for the guidance!

[zainnadeem786]

Thanks for the clarification, @zooba.

I've prepared the documentation-only follow-up as discussed.

PR: #153218

This follows the implementation direction discussed here and only clarifies the documented DirEntry.path semantics without changing runtime behavior.