Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-103977: compile re expressions in platform.py only if required #103981

Merged
merged 12 commits into from
Apr 30, 2023

Conversation

eendebakpt
Copy link
Contributor

@eendebakpt eendebakpt commented Apr 28, 2023

In platform.py there are several regular expressions compiled during import that might not be used at all. The regular expressions are only called once at most, since they are called from within cached methods. We can eliminate the cost of compilation by moving the regular expressions into the methods.

The import time of platform.py (measured with https://github.com/sympy/sympy/blob/master/bin/test_import, adapted for platform.py):

Main: The speed of "import platform" is: 7.567131 +- 0.208541 [ms]
PR: The speed of "import platform" is: 6.513338 +- 0.404813 [ms]

Notes:

  • All regular expressions modified are within cached methods. The only exceptions is are _libc_search and component_re which are used in the method libc_ver (not cached). The libc_ver is called from platform which is cached.
    If we expect libc_ver to be called many times, it should be cached as well, or the _libc_search regular expression moved back as a module variable.
  • A couple of ms of the import time is spend in importing other modules (such as collections)

@hauntsaninja This is a followup of #102492

@JelleZijlstra
Copy link
Member

Can you run make patchcheck to fix the whitespace issue?

@TeamSpen210
Copy link

Since they’re now all local variables, it might be nice to remove the underscore prefixes.

…sF70A.rst

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x has failed when building commit f186557.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/345/builds/4626) and take a look at the build logs.
  4. Check if the failure is related to this commit (f186557) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/345/builds/4626

Failed tests:

  • test_threading

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  14% (1/7)        
remote: Compressing objects:  28% (2/7)        
remote: Compressing objects:  42% (3/7)        
remote: Compressing objects:  57% (4/7)        
remote: Compressing objects:  71% (5/7)        
remote: Compressing objects:  85% (6/7)        
remote: Compressing objects: 100% (7/7)        
remote: Compressing objects: 100% (7/7), done.        
remote: Total 9 (delta 1), reused 2 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'f186557dc3e77495ba65b9b7e492ab6ddb7cecc3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at f186557dc3 gh-103977: compile re expressions in platform.py only if required (#103981)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: Segmentation fault

Thread 0x00007fb2ea5bb280 (most recent call first):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 74 in _write_status
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 95 in addSuccess
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/testresult.py", line 122 in addSuccess
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 648 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1106 in _run_suite
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1232 in run_unittest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module>
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi (total: 1)
Segmentation fault
make: *** [Makefile:1954: buildbottest] Error 139

Cannot open file '/root/buildarea/3.x.angelico-debian-amd64/build/test-results.xml' for upload

@JelleZijlstra
Copy link
Member

Seems unlikely that this would cause a segfault in test_threading.

@hugovk
Copy link
Member

hugovk commented Apr 30, 2023

And backport to 3.11?

@JelleZijlstra
Copy link
Member

I don't think so; we don't backport perf improvements.

carljm added a commit to carljm/cpython that referenced this pull request May 1, 2023
* main: (26 commits)
  pythongh-104028: Reduce object creation while calling callback function from gc (pythongh-104030)
  pythongh-104036: Fix direct invocation of test_typing (python#104037)
  pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-103761)
  pythongh-103895: Improve how invalid `Exception.__notes__` are displayed (python#103897)
  Adjust expression from `==` to `!=` in alignment with the meaning of the paragraph. (pythonGH-104021)
  pythongh-88496: Fix IDLE test hang on macOS (python#104025)
  Improve int test coverage (python#104024)
  pythongh-88773: Added teleport method to Turtle library (python#103974)
  pythongh-104015: Fix direct invocation of `test_dataclasses` (python#104017)
  pythongh-104012: Ensure test_calendar.CalendarTestCase.test_deprecation_warning consistently passes (python#104014)
  pythongh-103977: compile re expressions in platform.py only if required (python#103981)
  pythongh-98003: Inline call frames for CALL_FUNCTION_EX (pythonGH-98004)
  Replace Netlify with Read the Docs build previews (python#103843)
  Update name in acknowledgements and add mailmap (python#103696)
  pythongh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (python#103927)
  Remove non-existing tools from Sundry skiplist (python#103991)
  pythongh-103793: Defer formatting task name (python#103767)
  pythongh-87092: change assembler to use instruction sequence instead of CFG (python#103933)
  pythongh-103636: issue warning for deprecated calendar constants (python#103833)
  Various small fixes to dis docs (python#103923)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants