Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Update pylatest-pip-openblas-pandas build to Python 3.11 #29444

Merged
merged 8 commits into from
Jul 10, 2024

Conversation

lesteve
Copy link
Member

@lesteve lesteve commented Jul 10, 2024

Follow up on #29436 (comment).

There seems to be some consistent reports that coverage is slow on Python 3.12 nedbat/coveragepy#1665. Setting COVERAGE_CORE=sysmon may be a way to speed that up on Python 3.12 which seems to do the trick: nedbat/coveragepy#1665 (comment)

@lesteve lesteve changed the title CI Update pylatest-pip-openblas-pandas to Python 3.12 CI Update pylatest-pip-openblas-pandas build to Python 3.12 Jul 10, 2024
Copy link

github-actions bot commented Jul 10, 2024

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: a08e849. Link to the linter CI: here

@lesteve
Copy link
Member Author

lesteve commented Jul 10, 2024

Summary for pylatest_pip_openblas_pandas_linux-64 CI timings:

  • coverage does make running the tests 2 times slower on Python 3.12
  • COVERAGE_CORE=sysmon makes the problem even worse ...

Here are the details:

  • update to Python 3.12 build log test only ~32 minutes (total ~40 minutes)
    image

  • coverage turned off build log test only ~16 minutes (total ~26 minutes)
    image

  • COVERAGE_CORE=sysmon build log test only ~ 62 minutes 😱 (total ~73 minutes). Second run to double-check build log test only ~57 minutes (total ~68 minutes)
    image

Other builds seem to run a lot faster without coverage e.g. the OSX ones. Right now here are the lock-files using Python 3.12:

❯ rg -l python-3.12 build_tools/**/*
build_tools/azure/pylatest_conda_forge_mkl_osx-64_conda.lock
build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
build_tools/azure/pylatest_conda_mkl_no_openmp_osx-64_conda.lock
build_tools/azure/pylatest_pip_openblas_pandas_linux-64_conda.lock
build_tools/azure/pylatest_pip_scipy_dev_linux-64_conda.lock
build_tools/github/pylatest_conda_forge_cuda_array-api_linux-64_conda.lock
build_tools/azure/pylatest_pip_openblas_pandas_linux-64_conda.lock
build_tools/azure/pylatest_conda_mkl_no_openmp_osx-64_conda.lock
build_tools/azure/pylatest_conda_forge_mkl_osx-64_conda.lock
build_tools/azure/pylatest_pip_scipy_dev_linux-64_conda.lock

@ogrisel
Copy link
Member

ogrisel commented Jul 10, 2024

Thanks for the analysis. Maybe we can update the CPython version pin comment to reference the upstream issue at python/cpython#107674 and use CPython 3.11 (instead of CPython 3.9) in the mean time.

@lesteve
Copy link
Member Author

lesteve commented Jul 10, 2024

I have opened nedbat/coveragepy#1812 to see what they say.

I'll update the lock-files to use Python 3.11.

@lesteve
Copy link
Member Author

lesteve commented Jul 10, 2024

No idea why codecov status is pending, but that should not prevent this from being merged 😉

Edit: I triggered a build to have a fully green CI.

@lesteve lesteve changed the title CI Update pylatest-pip-openblas-pandas build to Python 3.12 CI Update pylatest-pip-openblas-pandas build to Python 3.11 Jul 10, 2024
@ogrisel ogrisel merged commit ab9f748 into scikit-learn:main Jul 10, 2024
34 checks passed
@lesteve lesteve deleted the pylatest-3.12 branch July 11, 2024 06:54
glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants