-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Packaging] Bump embedded Python version to 3.12.7 #29887
Conversation
️✔️AzureCLI-FullTest
|
️✔️AzureCLI-BreakingChangeTest
|
Packaging |
|
||
echo "== Testing pip install on Python 3.11 ==" | ||
docker run \ | ||
--rm -v $PYPI_FILES:/mnt/pypi mcr.microsoft.com/mirror/docker/library/python:3.11-slim \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MAR does not provide newer Python docker image anymore, use local Python instead.
Ref: https://mcr.microsoft.com/en-us/artifact/mar/mirror/docker/library/python/tags
@@ -606,7 +597,7 @@ jobs: | |||
-e CLI_VERSION=$CLI_VERSION \ | |||
-e HOMEBREW_UPSTREAM_URL=$HOMEBREW_UPSTREAM_URL \ | |||
--name azurecli \ | |||
mcr.microsoft.com/mirror/docker/library/python:3.11-slim \ | |||
mcr.microsoft.com/azurelinux/base/python:3 \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Formula generation is independent of the Python version.
Use Azure Linux's 3.12 to avoid updates for the next four years.
docker source: https://mcr.microsoft.com/en-us/artifact/mar/azurelinux/base/python/tags
shutil.rmtree(self._cache_dir, | ||
ignore_errors=True, | ||
onerror=lambda _, p, tr: self._logger.error('Fail to remove file %s', p)) | ||
shutil.rmtree(self._cache_dir, ignore_errors=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
onerror
is only deprecated in 3.12, and is replaced by onexc
. But there is no equivalent of onexc
in earlier Python versions: python/cpython#112645.
onerror
is meaningless when ignore_errors=True
, so I remove it.
Ref: https://docs.python.org/3/library/shutil.html#shutil.rmtree
source env/bin/activate | ||
cd azure-cli-extensions | ||
python ../scripts/ci/automation_full_test.py "12" "$(Instance_idx)" "latest" "" "True" "extension" | ||
displayName: 'Rerun tests' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
azdev
can't find extension list when running in AzureCLI@1
, move it to new step.
Traceback (most recent call last):
File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 664, in <module>
extension_main()
File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 654, in extension_main
autoscheduling.get_extension_modules()
File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 546, in get_extension_modules
raise RuntimeError("No extension detected")
RuntimeError: No extension detected
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the error and do we know why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's better not to change it until we figure out why.
afd296a
to
f350858
Compare
667728c
to
d53b7dd
Compare
@@ -24,7 +24,11 @@ exit_code=0 | |||
# Disable k8s-extension temporarily: https://github.com/Azure/azure-cli-extensions/pull/6702 | |||
# Disable alias temporarily: https://github.com/Azure/azure-cli/pull/27717 | |||
# hybridaks is going to be deprecated: https://github.com/Azure/azure-cli/pull/29838 | |||
ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks' | |||
# db-up is going to be deprecated: https://github.com/Azure/azure-cli/pull/29887 | |||
ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks db-up' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
db-up
's pymssql==2.2.7
does not provide wheel for 3.12. This extension is deprecated: Azure/azure-cli-extensions#8127
Our self-hosted agent provided by 1ES fails to build Windows package when using 3.12.7, but everything works fine in 3.12.6 embedded Python. When calling The microsoft-hosted agent works normally. The only difference I found is that it has Python 3.12.7. I suspect that calling commands without This issue exists for a long time, but only triggered in this PR. Here are some debug info:
|
3995731
to
ea25cd5
Compare
REM Install wheel to force pip install azure-cli in legacy mode | ||
REM see https://github.com/Azure/azure-cli/pull/29887 | ||
echo Installing setuptools wheel | ||
%PYTHON_DIR%\python.exe -Im pip install setuptools wheel |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For setuptools
:
Although setuptools
is added to azure-cli
dependency list in #27196, azure-cli
is installed with --no-deps
. Need to install it manually.
For wheel
:
In 3.11, wheel
is also installed when running get-pip.py
. In 3.12, it's not installed. wheel
is not included in final package in Linux and macOS. Actually, pip can still install wheel package without wheel
. setuptools
also removed wheel
dependency in pypa/setuptools#1386.
However, wheel
is still a factor for pip
to use pep517 or not: https://github.com/pypa/pip/blob/ec5faeac4ef6bae97df0e779566ceb2b0de89d3f/src/pip/_internal/pyproject.py#L107 (ref: pypa/packaging.python.org#1517). If wheel
is not installed, pip
will use pep517 and fails, see #29887 (comment). The reason is that the -I
param is missing in pip's build env. (Ref: pypa/pip#9081). See discussion in: pypa/pip#13023
So I also install wheel
. (I think it's okay to remove wheel
in the future)
# 3.11's get-pip.py
Successfully installed pip-24.2 setuptools-75.1.0 wheel-0.44.0
Pip set up successful
# see https://github.com/Azure/azure-cli/pull/29887 | ||
extra_dependencies = ['pip', 'setuptools'] | ||
for dependency in extra_dependencies: | ||
nodes[dependency] = research_package(dependency) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make_graph
explicitly exclude pip
and setuptools
: https://github.com/tdsmith/homebrew-pypi-poet/blob/fdafc615bcd28f29bcbe90789f07cc26f97c3bbc/poet/poet.py#L131
Adding pip
and setuptools
in CLI's dependency does not work as brew's pip
uses --no-deps
option. Homebrew/homebrew-core#194884 (comment)
# Does not exit if az extension add fails until all extensions have been tested | ||
set +e |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will set +e
make the script no longer fail in case of extension installation failure?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is similar to #14295 (comment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test_extensions.sh
calls scripts/ci/artifacts.sh
which calls scripts/ci/build.sh
.
build.sh
has set -e
.
Once -e
is set, this script fails immediately if one extension fails. set +e
ensures all extension have been tested. Finally, line 59 returns exit_code
.
@@ -2,6 +2,8 @@ | |||
|
|||
root=$(cd $(dirname $0); pwd) | |||
|
|||
tdnf install -y ca-certificates |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We switched the docker image from mcr.microsoft.com/mirror/docker/library/python:3.11-slim
to mcr.microsoft.com/azurelinux/base/python:3
which doesn't intall ca-certificates
by default.
Bump bundled Python to 3.12 and upgrade Python version in CI
The package size has also decreased, as the pyc file size is optimized in 3.12.
The windows zip package drops from 314MB to 299MB.
Related issue: #27957
Main changes:
-I
inbuild.cmd
pip
andsetuptools
in homebrewHistory Notes
[Core] Resolve CVE-2024-6119
[Core] Resolve CVE-2024-5535
[Core] Resolve CVE-2024-4741
[Core] Resolve CVE-2024-4603
[Core] Resolve CVE-2024-2511