Optimize pathlib.PurePath.__fspath__()

[barneygale]

Feature or enhancement

Return an unnormalized path from pathlib.PurePath.__fspath__()

Pitch

Code like open(Path('./README.txt')) or Path('/home//barney').iterdir() shouldn't require us to normalize the path in pathlib (e.g. remove . segments, doubled slashes, etc), as OS APIs are perfectly happy with unnormalized paths.

We can improve the performance of most Path methods, and the effective performance of passing a Path object to any API that accepts os.PathLike, by skipping normalization in __fspath__().

Prerequisites

Pathlib must not normalize paths on construction:

• Optimize pathlib path construction #101362

Pathlib's normalization must not change the meaning of paths:

• pathlib strips trailing slash #65238
• [pathlib] issues with Windows device paths #78079

Previous discussion

• GH-101362: Optimise pathlib by deferring path normalisation #101560

Linked PRs

• [WIP] GH-102783: Speed up pathlib.PurePath.__fspath__() by returning raw path #112397

[eryksun]

On Windows, an exception should be made to normalize extended paths in __fspath__(). If an extended path isn't normalized, the open will fail. One needs to replace slashes with backslashes; collapse repeated slashes; and resolve "." and ".." components.

Scripts have to be prepared to handle extended paths coming from os.path.realpath(), __file__, and attributes of the sys module such as executable, prefix, argv, and path. Extended paths can also come from extension modules that use API functions such as GetModuleFileNameW() (if an EXE/DLL was loaded using an extended path), GetFinalPathNameByHandleW(), and any of the PathCch*() functions that implements the flag PATHCCH_ALLOW_LONG_PATHS.

For path construction, there's an open issue that proposes to modify ntpath.join() to fully normalize any drive path or UNC path via normpath(). For paths without a drive, it's proposed to replace slashes with backslashes and collapse repeated slashes. For example, this change would support getting a useful result from something like ntpath.join(filename, "../spam//eggs.py") given filename is an extended path such as "\\?\C:\foo\bar.py".

[barneygale]

On Windows, an exception should be made to normalize extended paths in __fspath__(). If an extended path isn't normalized, the open will fail. One needs to replace slashes with backslashes; collapse repeated slashes; and resolve "." and ".." components.

I don't think pathlib's normalization should change the meaning of paths. It sounds like we should adjust the normalization routine to preserve forward slashes, repeated slashes, and "." components, for these kinds of path. ".." components are already be preserved I think.

[eryksun]

I don't think pathlib's normalization should change the meaning of paths. It sounds like we should adjust the normalization routine to preserve forward slashes, repeated slashes, and "." components, for these kinds of path. ".." components are already be preserved I think.

In almost all cases, extended paths are used solely to access a long path. The need to get a literal path is rare, especially in regard to slashes and "." and ".." component names. Microsoft's filesystems will fail an open that has forward slashes or repeated backslashes. NTFS will fail an open that has "." or ".." component names. FAT filesystems allow creating "." and ".." component names as a file's long name, with some other random short name, but it's very dysfunctional.

People assume that open(filename / "../spam//eggs.txt") will just work. But if filename is an extended path -- one that's injected completely outside of the control of the script -- then this open will fail if the path isn't normalized. IMO, no purity in regards to literal paths is worth this headache.

[pitrou]

OS APIs are perfectly happy with unnormalized paths

What happens if a path is mounted on e.g. a FUSE filesystem or a Samba share? Does the OS unnormalize before handing the path to the filesystem?

[barneygale]

Resolving as "can't do", because #65238 is resolved as "won't fix"