Add os.path.isrelative() and improve ntpath.isabs()

[jongfoster]

BPO	1669539
Nosy	@mhammond, @terryjreedy, @tiran, @tjguk, @ezio-melotti, @bitdancer, @zware, @serhiy-storchaka
Dependencies	bpo-15414: os.path.join behavior on Windows (ntpath.join) is not well documented
Files	win32_absolute_paths.patch: Patch v1 win32_absolute_paths_2.patch: Patch v2 - with docs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2007-02-26.23:07:51.000>
labels = ['type-feature', 'library']
title = 'Add os.path.isrelative() and improve ntpath.isabs()'
updated_at = <Date 2019-04-26.17:48:31.669>
user = 'https://bugs.python.org/jongfoster'

bugs.python.org fields:

activity = <Date 2019-04-26.17:48:31.669>
actor = 'BreamoreBoy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2007-02-26.23:07:51.000>
creator = 'jongfoster'
dependencies = ['15414']
files = ['7798', '7799']
hgrepos = []
issue_num = 1669539
keywords = ['patch']
message_count = 15.0
messages = ['51978', '51979', '51980', '51981', '51982', '51983', '51984', '58743', '110297', '117582', '138215', '166066', '218560', '222855', '229053']
nosy_count = 11.0
nosy_names = ['mhammond', 'terry.reedy', 'jorend', 'christian.heimes', 'tim.golden', 'ezio.melotti', 'eckhardt', 'r.david.murray', 'zach.ware', 'serhiy.storchaka', 'mdengler']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1669539'
versions = ['Python 3.5']

Linked PRs

• GH-44626, GH-105476: Fix ntpath.isabs() handling of part-absolute paths #113829

[jongfoster]

Hi,

I consider this to be a bug in os.path.isabs():

PythonWin 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32.
>>> import os.path
>>> s = '\\Python25'
>>> os.path.isabs(s)
True
>>> os.path.abspath(s)
'C:\\Python25'
>>> os.chdir('d:')
>>> os.path.abspath(s)
'D:\\Python25'
>>> 

If s is really an absolute path as isabs() claims, then why does abspath() return a different path (i.e. not s)? And worse, note that a call to os.chdir() changes the meaning of s! So s is clearly not an absolute path, and isabs() is wrong.

It turns out that on Windows there are really 4 different kinds of paths:

1. Completely relative, e.g. foo\bar
2. Completely absolute, e.g. c:\foo\bar or \\server\share
3. Halfbreeds with no drive, e.g. \foo\bar
4. Halfbreeds relative to the current working directory on a specific drive, e.g. c:foo\bar

Python 2.5's os.path.isabs() method considers both (2) and (3) to be absolute; I agree with the classification of (2) but strongly disagree about case (3).

Oh, and os.path.join is also broken, e.g. os.path.join('foo', 'a:bar') gives 'foo\\a:bar', which is an invalid path.

Another consequence of this is that os.path.isabs() is not enough to classify paths. Sometimes you really need a relative path, so we really need (at least) a new os.path.isrelative(), which can return "not isabs(s)" on POSIX platforms, and do the right thing on Win32.

The attached patch:

• Changes the behaviour of os.path.isabs() and os.path.join() on Win32, to classify pathnames as described above (including adding UNC support)
• Adds os.path.isrelative() on all platforms
• Changes a couple of Win32 os.path tests where I have deliberately broken backward compatibility
• Adds lots of new tests for these 3 functions on Win32

This does change the behaviour of isabs(), and it is possible that existing applications might be depending on the current behaviour. Silently changing the behaviour might break those applications. I'm not sure what the best course of action is - applying this for 2.6, putting a new API in (os.path.isreallyabs()?), or waiting for Python 3000...?

Kind regards,

Jon Foster

[jongfoster]

P.S. The patch is against Python SVN trunk

[jorend]

Thanks for submitting this. Nice summary of the different cases. I think very few people have given this that much thought. :)

You need a patch for the docs if you want this to even get considered. Then the main obstacle is convincing the team that the nice new logical behavior is worth the breakage. I would be surprised if it got through.

And Python 3000 is not going to be the "subtle breakage" release. I would give you better odds if your patch created new APIs instead, and *documented* exactly what's useful about the model you're suggesting.

[jorend]

There is an interesting thread on python-dev right now about possibly tweaking the behavior of splitext() in 2.6. So maybe there is hope after all.

[jongfoster]

Thanks for the comments. Updated patch attached, which includes the doc changes.
File Added: win32_absolute_paths_2.patch

[mhammond]

I agree the current behaviour is not ideal - but the right way forward isn't clear. Note the existing (strange) behaviour of os.path.join:

>>> os.path.join("c:\\", "\\a")
'c:\\a'
>>> os.path.join("c:\\foo", "\\a")
'\\a'

if isabs('\\a') returns False, there is almost an expectation that the last example should return 'c:\\foo\\a' - but that would almost never give what was expected. 'c:\\a' seems the correct result, but my quick scan of the test code in the patch shows this isn't exercised.

FWIW, about the only common use of isabs() in my code is something like:

log_directory = get_option_from_config_file(...)
if not os.path.isabs(log_directory):
  # make log_directory absolute based on app directory.
  log_directory = os.path.join(app_directory, log_directory)

Best I can tell, the above would not break if isabs('/foo') started returning False, so long as os.path.join('c:\\foo', '\\bar') resulted in 'c:\\bar'.

Off the top-of-my-head, another strategy may be to introduce isrelative() in 2.6, and in 2.7 introduce a warning when isabs() is passed one of the halfbreeds, warning that 2.8 will return a different value, and pointing to the new isrelative(). 2.8 then gets the full, 'correct' implementation as described here.

[jongfoster]

Hi Mark,

Thanks for reviewing this.

You wrote:

os.path.join("c:\\foo", "\\a")
...
'c:\\a' seems the correct result

Yes indeed, and this patch makes it do that. Hiding in the patch is this changed test case:

-tester("ntpath.join('c:/a', '/b')", '/b')
+tester("ntpath.join('c:/a', '/b')", 'c:/b') # CHANGED: Was '/b')

and these new ones:

+tester('ntpath.join("c:\\a", "\\b")', "c:\\b")
+tester('ntpath.join("c:/a", "/b")', "c:/b")

(Hmm looks like one of those tests is a duplicate... not a big deal I guess).

I guess another way to handle these changes is to introduce new isabsolute(), isrelative(), and joinpath() functions, and deprecate isabs() and join(). (By "deprecate" I mean at least put a warning in the docs that it shouldn't be used; whether or not to make the code raise a DeprecationWarning is a separate decision). But that's kinda ugly from an API design point of view. I'm happy to roll a patch that does that, if we decide that fixing isabs() isn't possible.

It's a pity there isn't a way to do something like "from __future__ import fixed_win_paths". But given how dynamic Python is, I guess that would be hard.

Kind regards,

Jon

[tiran]

The problem should be addressed and fixed before the next alpha release.

[BreamoreBoy]

Nosy list changed as both have an interest in Windows issues.

[eckhardt]

I just stumbled across the issue with isabs(). I'd also say that Mark Hammond already provided the typical use case for this, i.e. that you allow relative paths for convenience when storing them or putting them on the commandline, but for actual use you first convert them so that their meaning doesn't change under your feet, e.g. because they depend on volatile things like the current working directory or drive.

If you assume that is the intention or typical use case, then calling isabs() to determine if the path is stable doesn't work. Of course, the wording of the documentation then needs to change, too, as it explicitly says "after chopping off a potential drive letter".

Concerning the behaviour of path.join() and support for "\\server\share" paths, I'm not sure, but I think that these are issues that can be discussed/changed separately.

[terryjreedy]

A bug for tracker purposes is a discrepancy between doc and code. That can be fixed in current versions. A design change is a feature request and can only go in future versions. A deprecation warning for one cycle is desirable when appropriate.

"os.path.isabs(path)
Return True if path is an absolute pathname. On Unix, that means it begins with a slash, on Windows that it begins with a (back)slash after chopping off a potential drive letter."

From what you report, the code does just that, so there is no bug.
Breaking code just because you do not like the current definition is very unlikely to be accepted.

Adding a .isrelative(path) that seems to be equivalent to 'not .ispath' is unlikely to accepted. People who care about the presence of a drive letter can write "path[1]==':'"

"os.path.join(path1[, path2[, ...]])
Join one or more path components intelligently. If any component is an absolute path, all previous components (on Windows, including the previous drive letter, if there was one) are thrown away, and joining continues."

>>> import os
>>> os.path.join('foo', 'a:bar')
'foo\\a:bar'

Producing a non-path does not seem intelligent, so I agree that this a bug. I am not sure what this should produce. 'foo/bar', 'a:foo/bar', and raising an exception would be candidates. All would be better than the current useless return. In any case, a bug fix patch can only fix bugs and should not be intermixed with feature patches.

As to the second sentence: your proposal to change documented and inplemented semantics is, again, a feature request, not a bug fix.

If you want to continue with your feature requests, you can if you wish reclassify this issue, retitle again, and open a new behavior issue for the path bug and upload a bugfix-only patch. Before or along with doing so, I would suggest discussion on the python-ideas list.

[bitdancer]

I've opened a bpo-15414 for fixing the documentation for os.path.join on Windows, and possibly existing bugs. So I'm reclassifying this as a feature request, since that is most of what the discussion is about.

[serhiy-storchaka]

ntpath.join() was fixed in bpo-19456.

[BreamoreBoy]

I've asked for a commit review on bpo-15414 so can we close this?

[zware]

To kick this along a bit, do the following testcases seem like the right behavior to others, based on Mark Hammond's roadmap in msg51983? If there's some agreement, I'll work on getting a modernized patch put together.

# currently (3.4)
assertTrue(ntpath.isabs(r"\driveless\halfbreed"))
assertFalse(ntpath.isabs(r"D:rived\halfbreed"))
assertTrue(ntpath.isabs(r"\\any\UNC\path"))
assertTrue(ntpath.isabs(r"O:\bviously\absolute"))
assertFalse(ntpath.isabs(r"obviously\relative"))


# 3.5
# Halfbreeds are not relative, keep same isabs behavior but warn
with assertWarnsRegex(FutureWarning, "will return False in 3.6"):
    assertTrue(ntpath.isabs(r"\driveless\halfbreed"))
# same behavior
assertFalse(ntpath.isabs(r"D:rived\halfbreed"))
assertTrue(ntpath.isabs(r"\\any\UNC\path"))
assertTrue(ntpath.isabs(r"O:\bviously\absolute"))
assertFalse(ntpath.isabs(r"obviously\relative"))

# new functions
assertFalse(ntpath.isrelative(r"\driveless\halfbreed"))
assertFalse(ntpath.isrelative(r"D:rived\halfbreed"))
assertFalse(ntpath.isrelative(r"\\any\UNC\path"))
assertFalse(ntpath.isrelative(r"O:\bviously\absolute"))
assertTrue(ntpath.isrelative(r"obviously\relative"))

assertTrue(posixpath.isrelative("foo/bar"))

assertTrue(macpath.isrelative(":foo:bar"))
assertTrue(macpath.isrelative("foobar"))


# 3.6
assertFalse(ntpath.isabs(r"\driveless\halfbreed"))
# all else the same

I'll note that this is a bit extra complicated by the fact that MS calls r"\driveless\halfbreed" an "absolute path", see http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.100%29.aspx#fully_qualified_vs.\_relative_paths
That gives me the impression that Windows has notions of "fully qualified" paths (r"C:\foo", "C:\\", r"\\any\UNC\path"), absolute paths (r"\foo", etc.), and relative paths (r"foo\bar", "C:foo", and annoyingly, "C:\foo\..\bar"). My opinion is that we should just declare "we don't quite agree with MS on this one" and go with the semantics outlined above, though we're currently mostly in agreement with them.

[barneygale]

(cross-post from #66498)

Would a new strict argument for isabs() help? It wouldn't affect how UNC paths (two leading slashes) are handled, as they're always absolute, but otherwise:

1. isabs('blah', strict=None): true if the path has a root (default)
   • Backwards-compatible but rarely useful
2. isabs('blah', strict=False): true if the path has either drive or a root
   • Useful for checking if a path is entirely relative (negate its result)
3. isabs('blah', strict=True): true if the path has both a drive and a root
   • Useful for checking if a path is entirely absolute

[barneygale]

Cross-posting @zooba's comment from #66498 (now closed):

Would a new strict argument for isabs() help?

No, we should just strive for the most consistency. Testing for starting with two sep/altseps or {A-Z}:{sep} should be enough. And it should be consistent with how join() handles each segment.

If users really want to know whether a path is relative but has a drive segment, they can splitdrive.

Steve, are you saying we should consider this a bugfix and change the behaviour of isabs() without a deprecation period? That would run counter to Terry's earlier comment. And if not that, then what's the way forward here do you think? Grateful for your advice!

[zooba]

I think 10-years-ago-@zware has the right position:

That gives me the impression that Windows has notions of "fully qualified" paths (r"C:\foo", "C:\", r"\any\UNC\path"), absolute paths (r"\foo", etc.), and relative paths (r"foo\bar", "C:foo", and annoyingly, "C:\foo..\bar"). My opinion is that we should just declare "we don't quite agree with MS on this one"

But I'm choosing to assume that the bit we disagree with is the definition of "absolute path". We think an absolute path is fully qualified, and so abspath and isabs should always return a fully qualified path, and the fact that Microsoft uses a different name for this is their problem and not ours.

I'm also encouraged to this view because, as you noted, the hypothetical isabs(path, strict=None) (current behaviour) isn't particularly useful. A path that returns True for the current check may or may not require additional context in order to be resolved, and so it's not "absolute" in our sense.

To be "absolute" in our sense, we need to be able to identify the device the path belongs to, and allow the device to resolve the rest of the path (and we pretend that devices are always going to use POSIX-ish path semantics, which is usually close enough). So an absolute path must identify a device (i.e. C:, \\.\C:\ or \\machine\share\), and must start from the root of that device (i.e. C:\path - note that formats starting with \\.\ or \\?\ can only start from the root).

[zooba]

Steve, are you saying we should consider this a bugfix and change the behaviour of isabs() without a deprecation period?

So yes, isabs() is incorrect and doesn't match the docs.

I think "on Windows ... begins with a (back)slash after chopping off a potential drive letter" sufficiently suggests the intended behaviour (assuming "potential drive letter" means "a letter for a drive that has the potential to exist", as opposed to meaning "an optional drive letter").

I wouldn't be opposed to clarifying the docs while we're here, though. They don't have to be massively detailed, but can probably be a little more robust. And we should definitely have NEWS and What's New mentions.

[barneygale]

17 years later... it's fixed! Thanks for the patch @jonfoster, I adapted your test cases :)

Fixed in 3.13+ / e4ff131 / #113829

[jaraco]

In pypa/setuptools#4669, we learned that this change has broken some assumptions (as intended, I guess), but it leaves the user without a useful "is absolute even if not fully qualified" check.

I'm also encouraged to this view because, as you noted, the hypothetical isabs(path, strict=None) (current behaviour) isn't particularly useful.

That function would be useful in the distutils case, where the caller is interested in making sure the path is relative (i.e. no leading slash). By changing the meaning of isabs, it's now the responsibility of downstream to implement is_drive_absolute where needed and to adapt across Python versions.

Personally, I felt the old isabs behavior was preferable, because it gave a more consistent behavior across platforms. I prefer to find interfaces that bridge the gaps between Windows and non-Windows systems rather than those that reflect the idiosyncrasies. I like to treat my Windows systems much like Unix systems, having only one drive and mounting new disks to other paths on that C drive, such that \foo really can be treated as absolute in most cases.

Shouldn't a backward-incompatible change like this have gone through a deprecation period, warning when a drive-relative absolute path was passed to isabs on Windows (e.g. isabs('/'))? In the current state, developers now have to contend with variance across the matrix of platform and Python version.

Perhaps this change should be reverted and instead a replacement created, something like isreallyabs (or ispureabs), which could be used by the caller when isabs is insufficient?

Probably distutils will switch to use pathlib for this "make relative" function to get compatibility across Python versions. I'm hoping it's no more than the two lines of code that already exist.

[barneygale]

but it leaves the user without a useful "is absolute even if not fully qualified" check.

That use case is rare enough that I don't think it deserves a spot in os.path personally. I try to steer people away from part-absolute paths (e.g. \foo or C:foo) because they're too easy to misunderstand.

That function would be useful in the distutils case, where the caller is interested in making sure the path is relative (i.e. no leading slash).

Isn't distutils also interested in whether the path has a drive letter? os.path.splitroot() would work here, but it's 3.12+ only:

def is_relative(path):
    drive, root, _tail = os.path.splitroot(path)
    return not root and not drive

Personally, I felt the old isabs behavior was preferable, because it gave a more consistent behavior across platforms. I prefer to find interfaces that bridge the gaps between Windows and non-Windows systems rather than those that reflect the idiosyncrasies. I like to treat my Windows systems much like Unix systems, having only one drive and mounting new disks to other paths on that C drive, such that \foo really can be treated as absolute in most cases.

I agree with most of this, but I reached a different conclusion. For me, the cross-platform definition of "absolute path" is roughly "unaffected by context", i.e. independent of the current drive or directory. For cross-platform code I try to leave the details to os.path rather than using POSIX-style paths throughout and never changing to another Windows drive.

FWIW, the new behaviour also agrees with pathlib.PurePath.is_absolute().

Shouldn't a backward-incompatible change like this have gone through a deprecation period, warning when a drive-relative absolute path was passed to isabs on Windows (e.g. isabs('/'))? In the current state, developers now have to contend with variance across the matrix of platform and Python version.

Maybe it should have. We skipped backporting to reduce disruption, but perhaps that's not enough.

[barneygale]

Should we revert this change, and instead emit a (deprecation?) warning when a drive-less rooted path is given to ntpath.isabs()? Penny for any opinions further to Jason's and my own.

[jaraco]

In distutils, I ended up re-implementing the make_relative logic with pathlib.Path.relative_to(pathlib.Path.anchor), so the compatibility concerns are now moot for that use-case. I'm unsure how many other cases are affected by this change. Perhaps it's best to leave it as implemented, since it is more internally consistent.

[zooba]

Penny for any opinions further to Jason's and my own.

Here late, but I agree that "unaffected by local¹ context" is the best definition for isabs. No doubt it was discussions between Barney and myself that landed on that definition though, so maybe that opinion isn't worth a penny.

"Unaffected by abspath" is another good definition, though I believe we have that backwards because abspath uses isabs to check whether it needs to do anything.

My tendency on the kinds of situations distutils was in is to make paths absolute as early as possible, and keep track of what they should be relative to for cases where I need to output that. Since abspath is safe/consistent on all (path) inputs, and calculating relative paths from absolute paths should be consistent across versions and platforms, this seems an approach that will work everywhere.

Footnotes

1. A relative path is unaffected by moving it to another machine, while an absolute path is theoretically no longer valid, but I don't have a good name for this kind of context. ↩

[jaraco]

Thanks for the greater context. My recommendation, unless there are other reports of use-cases adversely affected by this change, let's accept the new implementation and charge ahead.

[the-can-of-soup]

Just wanted to point out that it might be a security risk that paths like / and \ return False on Windows as of Python 3.13, even though in Windows (at least Windows 10) both paths refer to the root of the drive. I often use os.path.isabs (along with a few other checks) to make sure user input doesn't refer to files outside the current working directory, but a bad actor could easily use this behavior to get around the checks.

[zooba]

I often use os.path.isabs (along with a few other checks) to make sure user input doesn't refer to files outside the current working directory, but a bad actor could easily use this behavior to get around the checks.

You need a better check anyway, to detect malicious paths like a\b\..\..\..\..\..\..\Windows\System32\.... I personally resolve the path and then try pathlib.Path.relative_to and handle the exception, but I'm sure there's an ntpath equivalent.