Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-117587: Add C implementation of os.path.abspath #117855

Closed
wants to merge 87 commits into from

Conversation

nineteendo
Copy link
Contributor

@nineteendo nineteendo commented Apr 13, 2024

Benchmark

posixpath.py by @eryksun:

absolute, with "/a" repeated
len   speedup
100   1.521x
500   1.157x
1000  1.128x
2000  1.068x
4000  1.019x

relative 'a', with cwd length
len     speedup
93+2    1.861x
548+2   1.729x
1003+2  1.881x
2043+2  1.556x
4058+2  1.548x

ntpath.py

script
::speedup-posixpath.abspath.bat
@echo off
echo 10 chars && call main\python -m timeit -s "import os" "os.path.abspath('a/' * 5)" && call speedup-posixpath.abspath\python -m timeit -s "import os" "os.path.abspath('a/' * 5)"
echo 100 chars && call main\python -m timeit -s "import os" "os.path.abspath('a/' * 50)" && call speedup-posixpath.abspath\python -m timeit -s "import os" "os.path.abspath('a/' * 50)"
echo 1000 chars && call main\python -m timeit -s "import os" "os.path.abspath('a/' * 500)" && call speedup-posixpath.abspath\python -m timeit -s "import os" "os.path.abspath('a/' * 500)"
10 chars
500000 loops, best of 5: 635 nsec per loop # before
500000 loops, best of 5: 517 nsec per loop # after
# -> 1.23x faster
100 chars
200000 loops, best of 5: 1.53 usec per loop # before
200000 loops, best of 5: 1.23 usec per loop # after
# -> 1.24x faster
1000 chars
50000 loops, best of 5: 9.87 usec per loop # before
50000 loops, best of 5: 7.73 usec per loop # after
# -> 1.28x faster

@nineteendo nineteendo marked this pull request as ready for review April 13, 2024 20:03
@erlend-aasland erlend-aasland changed the title gh-117587: Speedup posixpath.abspath gh-117587: Add C implementation of posixpath.abspath Apr 13, 2024
Modules/posixmodule.c Outdated Show resolved Hide resolved
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Modules/posixmodule.c Outdated Show resolved Hide resolved
Python/fileutils.c Outdated Show resolved Hide resolved
nineteendo and others added 2 commits April 13, 2024 23:43
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Lib/posixpath.py Outdated Show resolved Hide resolved
@nineteendo
Copy link
Contributor Author

The same approach doesn't work on Window, because ntpath.join() is much slower than nt._getfullpathname():

PS C:\Users\wanne\cpython> python -m timeit -s "import nt" "nt._getfullpathname('.')"; python -m timeit -s "import os" "os.path.join(r'C:\Users\wanne\cpython', '')"
1000000 loops, best of 5: 286 nsec per loop # _getfullpathname
200000 loops, best of 5: 1.41 usec per loop # join

@eryksun

This comment was marked as resolved.

@nineteendo
Copy link
Contributor Author

Preferably, abspath() should not be naively implemented by simply joining a relative path with the working directory. It doesn't work for drive-relative paths. That's a long-standing bug in the generic implementation ntpath._abspath_fallback().

I didn't even get that far in the testing process. I already knew it wasn't going to work.
I did notice this while trying to eliminate normpath() from the current implementation.

@bedevere-app
Copy link

bedevere-app bot commented May 29, 2024

Thanks for making the requested changes!

@erlend-aasland: please review the changes made to this pull request.

Lib/test/test_ntpath.py Outdated Show resolved Hide resolved
Modules/posixmodule.c Outdated Show resolved Hide resolved
@bedevere-app
Copy link

bedevere-app bot commented May 30, 2024

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@nineteendo
Copy link
Contributor Author

erlend-aasland, I have made the requested changes; please review again.

@bedevere-app
Copy link

bedevere-app bot commented May 30, 2024

Thanks for making the requested changes!

@erlend-aasland: please review the changes made to this pull request.

Python/fileutils.c Outdated Show resolved Hide resolved
Copy link
Contributor

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nitpicks

Python/fileutils.c Outdated Show resolved Hide resolved
Python/fileutils.c Outdated Show resolved Hide resolved
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants