Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Packaging] Bump embedded Python version to 3.12.7 #29887

Merged
merged 5 commits into from
Oct 24, 2024
Merged

Conversation

bebound
Copy link
Contributor

@bebound bebound commented Sep 11, 2024

Bump bundled Python to 3.12 and upgrade Python version in CI

The package size has also decreased, as the pyc file size is optimized in 3.12.
The windows zip package drops from 314MB to 299MB.
Related issue: #27957

Main changes:

  • Use Python 3.12 in CI
  • Bump bundled Python to 3.12
  • Add -I in build.cmd
  • Add pip and setuptools in homebrew

History Notes

[Core] Resolve CVE-2024-6119
[Core] Resolve CVE-2024-5535
[Core] Resolve CVE-2024-4741
[Core] Resolve CVE-2024-4603
[Core] Resolve CVE-2024-2511

Copy link

azure-client-tools-bot-prd bot commented Sep 11, 2024

️✔️AzureCLI-FullTest
️✔️acr
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️acs
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️advisor
️✔️latest
️✔️3.12
️✔️3.9
️✔️ams
️✔️latest
️✔️3.12
️✔️3.9
️✔️apim
️✔️latest
️✔️3.12
️✔️3.9
️✔️appconfig
️✔️latest
️✔️3.12
️✔️3.9
️✔️appservice
️✔️latest
️✔️3.12
️✔️3.9
️✔️aro
️✔️latest
️✔️3.12
️✔️3.9
️✔️backup
️✔️latest
️✔️3.12
️✔️3.9
️✔️batch
️✔️latest
️✔️3.12
️✔️3.9
️✔️batchai
️✔️latest
️✔️3.12
️✔️3.9
️✔️billing
️✔️latest
️✔️3.12
️✔️3.9
️✔️botservice
️✔️latest
️✔️3.12
️✔️3.9
️✔️cdn
️✔️latest
️✔️3.12
️✔️3.9
️✔️cloud
️✔️latest
️✔️3.12
️✔️3.9
️✔️cognitiveservices
️✔️latest
️✔️3.12
️✔️3.9
️✔️compute_recommender
️✔️latest
️✔️3.12
️✔️3.9
️✔️config
️✔️latest
️✔️3.12
️✔️3.9
️✔️configure
️✔️latest
️✔️3.12
️✔️3.9
️✔️consumption
️✔️latest
️✔️3.12
️✔️3.9
️✔️container
️✔️latest
️✔️3.12
️✔️3.9
️✔️containerapp
️✔️latest
️✔️3.12
️✔️3.9
️✔️core
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️cosmosdb
️✔️latest
️✔️3.12
️✔️3.9
️✔️databoxedge
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️dla
️✔️latest
️✔️3.12
️✔️3.9
️✔️dls
️✔️latest
️✔️3.12
️✔️3.9
️✔️dms
️✔️latest
️✔️3.12
️✔️3.9
️✔️eventgrid
️✔️latest
️✔️3.12
️✔️3.9
️✔️eventhubs
️✔️latest
️✔️3.12
️✔️3.9
️✔️feedback
️✔️latest
️✔️3.12
️✔️3.9
️✔️find
️✔️latest
️✔️3.12
️✔️3.9
️✔️hdinsight
️✔️latest
️✔️3.12
️✔️3.9
️✔️identity
️✔️latest
️✔️3.12
️✔️3.9
️✔️iot
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️keyvault
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️kusto
️✔️latest
️✔️3.12
️✔️3.9
️✔️lab
️✔️latest
️✔️3.12
️✔️3.9
️✔️managedservices
️✔️latest
️✔️3.12
️✔️3.9
️✔️maps
️✔️latest
️✔️3.12
️✔️3.9
️✔️marketplaceordering
️✔️latest
️✔️3.12
️✔️3.9
️✔️monitor
️✔️latest
️✔️3.12
️✔️3.9
️✔️mysql
️✔️latest
️✔️3.12
️✔️3.9
️✔️netappfiles
️✔️latest
️✔️3.12
️✔️3.9
️✔️network
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️policyinsights
️✔️latest
️✔️3.12
️✔️3.9
️✔️privatedns
️✔️latest
️✔️3.12
️✔️3.9
️✔️profile
️✔️latest
️✔️3.12
️✔️3.9
️✔️rdbms
️✔️latest
️✔️3.12
️✔️3.9
️✔️redis
️✔️latest
️✔️3.12
️✔️3.9
️✔️relay
️✔️latest
️✔️3.12
️✔️3.9
️✔️resource
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️role
️✔️latest
️✔️3.12
️✔️3.9
️✔️search
️✔️latest
️✔️3.12
️✔️3.9
️✔️security
️✔️latest
️✔️3.12
️✔️3.9
️✔️servicebus
️✔️latest
️✔️3.12
️✔️3.9
️✔️serviceconnector
️✔️latest
️✔️3.12
️✔️3.9
️✔️servicefabric
️✔️latest
️✔️3.12
️✔️3.9
️✔️signalr
️✔️latest
️✔️3.12
️✔️3.9
️✔️sql
️✔️latest
️✔️3.12
️✔️3.9
️✔️sqlvm
️✔️latest
️✔️3.12
️✔️3.9
️✔️storage
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️synapse
️✔️latest
️✔️3.12
️✔️3.9
️✔️telemetry
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9
️✔️util
️✔️latest
️✔️3.12
️✔️3.9
️✔️vm
️✔️2018-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.12
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.12
️✔️3.9
️✔️latest
️✔️3.12
️✔️3.9

Copy link

azure-client-tools-bot-prd bot commented Sep 11, 2024

️✔️AzureCLI-BreakingChangeTest
️✔️Non Breaking Changes

@yonzhan
Copy link
Collaborator

yonzhan commented Sep 11, 2024

Packaging

@bebound bebound changed the title [Packaging] Bump bundled python to 3.12.6 [Packaging] Bump bundled Python to 3.12.6 Sep 11, 2024
@microsoft-github-policy-service microsoft-github-policy-service bot added the Auto-Assign Auto assign by bot label Sep 11, 2024
@bebound bebound mentioned this pull request Sep 11, 2024
4 tasks
@bebound bebound changed the title [Packaging] Bump bundled Python to 3.12.6 [Packaging] Bump embedded Python version to 3.12.6 Sep 11, 2024

echo "== Testing pip install on Python 3.11 =="
docker run \
--rm -v $PYPI_FILES:/mnt/pypi mcr.microsoft.com/mirror/docker/library/python:3.11-slim \
Copy link
Contributor Author

@bebound bebound Sep 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAR does not provide newer Python docker image anymore, use local Python instead.

Ref: https://mcr.microsoft.com/en-us/artifact/mar/mirror/docker/library/python/tags

@@ -606,7 +597,7 @@ jobs:
-e CLI_VERSION=$CLI_VERSION \
-e HOMEBREW_UPSTREAM_URL=$HOMEBREW_UPSTREAM_URL \
--name azurecli \
mcr.microsoft.com/mirror/docker/library/python:3.11-slim \
mcr.microsoft.com/azurelinux/base/python:3 \
Copy link
Contributor Author

@bebound bebound Sep 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formula generation is independent of the Python version.
Use Azure Linux's 3.12 to avoid updates for the next four years.

docker source: https://mcr.microsoft.com/en-us/artifact/mar/azurelinux/base/python/tags

shutil.rmtree(self._cache_dir,
ignore_errors=True,
onerror=lambda _, p, tr: self._logger.error('Fail to remove file %s', p))
shutil.rmtree(self._cache_dir, ignore_errors=True)
Copy link
Contributor Author

@bebound bebound Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onerror is only deprecated in 3.12, and is replaced by onexc. But there is no equivalent of onexc in earlier Python versions: python/cpython#112645.

onerror is meaningless when ignore_errors=True, so I remove it.

Ref: https://docs.python.org/3/library/shutil.html#shutil.rmtree

source env/bin/activate
cd azure-cli-extensions
python ../scripts/ci/automation_full_test.py "12" "$(Instance_idx)" "latest" "" "True" "extension"
displayName: 'Rerun tests'
Copy link
Contributor Author

@bebound bebound Sep 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdev can't find extension list when running in AzureCLI@1, move it to new step.

Traceback (most recent call last):
  File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 664, in <module>
    extension_main()
  File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 654, in extension_main
    autoscheduling.get_extension_modules()
  File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 546, in get_extension_modules
    raise RuntimeError("No extension detected")
RuntimeError: No extension detected

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the error and do we know why?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better not to change it until we figure out why.

@bebound bebound marked this pull request as ready for review October 10, 2024 03:54
@bebound bebound changed the title [Packaging] Bump embedded Python version to 3.12.6 [Packaging] Bump embedded Python version to 3.12.7 Oct 10, 2024
@@ -24,7 +24,11 @@ exit_code=0
# Disable k8s-extension temporarily: https://github.com/Azure/azure-cli-extensions/pull/6702
# Disable alias temporarily: https://github.com/Azure/azure-cli/pull/27717
# hybridaks is going to be deprecated: https://github.com/Azure/azure-cli/pull/29838
ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks'
# db-up is going to be deprecated: https://github.com/Azure/azure-cli/pull/29887
ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks db-up'
Copy link
Contributor Author

@bebound bebound Oct 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bebound
Copy link
Contributor Author

bebound commented Oct 15, 2024

Our self-hosted agent provided by 1ES fails to build Windows package when using 3.12.7, but everything works fine in 3.12.6 embedded Python.

When calling ./python.exe -m pip xxxx, its return code is 57005 and does not have any output. If I use ./python.exe -Im pip xxxx, pip can run, but still fails to install azure-cli from src. The scripts/compact_aaz.py also fails with error code 1, without any error message.

The microsoft-hosted agent works normally. The only difference I found is that it has Python 3.12.7.

I suspect that calling commands without -I uses 3.12.6's module, which is incompatible with 3.12.7, causing failures. Therefore, I added -I to all Python commands in build.cmd.

This issue exists for a long time, but only triggered in this PR.

Here are some debug info:

# In self-hosted agent, it uses 3.12.6\x64\Lib
python.exe -m site 
sys.path = [
    'D:\\a\\_work\\1\\s\\artifacts\\Python',
    'D:\\a\\_work\\1\\s\\artifacts\\Python\\python312.zip',
    'C:\\hostedtoolcache\\windows\\Python\\3.12.6\\x64\\Lib',
    'C:\\hostedtoolcache\\windows\\Python\\3.12.6\\x64\\DLLs',
    'D:\\a\\_work\\1\\s\\artifacts\\Python\\Lib',
    'D:\\a\\_work\\1\\s\\artifacts\\Python\\Lib\\site-packages',
]
USER_BASE: 'C:\\Users\\cloudtest\\AppData\\Roaming\\Python' (doesn't exist)
USER_SITE: 'C:\\Users\\cloudtest\\AppData\\Roaming\\Python\\Python312\\site-packages' (doesn't exist)
ENABLE_USER_SITE: True

python.exe -Im site
sys.path = [
    'D:\\a\\_work\\1\\s\\artifacts\\Python\\python312.zip',
    'D:\\a\\_work\\1\\s\\artifacts\\Python',
    'D:\\a\\_work\\1\\s\\artifacts\\Python\\Lib',
    'D:\\a\\_work\\1\\s\\artifacts\\Python\\Lib\\site-packages',
]
USER_BASE: 'C:\\Users\\cloudtest\\AppData\\Roaming\\Python' (doesn't exist)
USER_SITE: 'C:\\Users\\cloudtest\\AppData\\Roaming\\Python\\Python312\\site-packages' (doesn't exist)
ENABLE_USER_SITE: False

python.exe -Im pip debug
pip version: pip 24.2 from D:\a\_work\1\s\artifacts\Python\Lib\site-packages\pip (python 3.12)
sys.version: 3.12.7 (tags/v3.12.7:0b05ead, Oct  1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)]
sys.executable: D:\a\_work\1\s\artifacts\Python\python.exe
sys.getdefaultencoding: utf-8
xxxxxx

python.exe -m pip debug
no output, return code 57005

%PYTHON_DIR%\python.exe -Ic "import pip;print(pip.__path__)"
['D:\\a\\_work\\1\\s\\artifacts\\Python\\Lib\\site-packages\\pip']
return code 0

%PYTHON_DIR%\python.exe -c "import pip;print(pip.__path__)"
['D:\\a\\_work\\1\\s\\artifacts\\Python\\Lib\\site-packages\\pip']
return code 0

install azure-cli from src:
install src
Using pip 24.2 from D:\a\_work\1\s\artifacts\Python\Lib\site-packages\pip (python 3.12)
Non-user install because user site-packages disabled
Created temporary directory: C:\Users\cloudtest\AppData\Local\Temp\pip-build-tracker-9hjc5ufp
Initialized build tracking at C:\Users\cloudtest\AppData\Local\Temp\pip-build-tracker-9hjc5ufp
Created build tracker: C:\Users\cloudtest\AppData\Local\Temp\pip-build-tracker-9hjc5ufp
Entered build tracker: C:\Users\cloudtest\AppData\Local\Temp\pip-build-tracker-9hjc5ufp
Created temporary directory: C:\Users\cloudtest\AppData\Local\Temp\pip-install-8u39htfy
Created temporary directory: C:\Users\cloudtest\AppData\Local\Temp\pip-ephem-wheel-cache-0fijydv4
Processing d:\a\_work\1\s\src\azure-cli
  Added file:///D:/a/_work/1/s/src/azure-cli to build tracker 'C:\\Users\\cloudtest\\AppData\\Local\\Temp\\pip-build-tracker-9hjc5ufp'
  Created temporary directory: C:\Users\cloudtest\AppData\Local\Temp\pip-build-env-trza6ryi
  Installing build dependencies: started
  Running command pip subprocess to install build dependencies
  error: subprocess-exited-with-error
  
  pip subprocess to install build dependencies did not run successfully.
  exit code: 57005
  
  See above for output.
# In microsoft-hosted agent, it uses 3.12.7\x64\Lib

python.exe -m site 
sys.path = [
    'D:\\a\\1\\s\\artifacts\\Python',
    'D:\\a\\1\\s\\artifacts\\Python\\python312.zip',
    'C:\\hostedtoolcache\\windows\\Python\\3.12.7\\x64\\Lib',
    'C:\\hostedtoolcache\\windows\\Python\\3.12.7\\x64\\DLLs',
    'D:\\a\\1\\s\\artifacts\\Python\\Lib',
    'D:\\a\\1\\s\\artifacts\\Python\\Lib\\site-packages',
]
USER_BASE: 'C:\\Users\\runneradmin\\AppData\\Roaming\\Python' (doesn't exist)
USER_SITE: 'C:\\Users\\runneradmin\\AppData\\Roaming\\Python\\Python312\\site-packages' (doesn't exist)
ENABLE_USER_SITE: True

REM Install wheel to force pip install azure-cli in legacy mode
REM see https://github.com/Azure/azure-cli/pull/29887
echo Installing setuptools wheel
%PYTHON_DIR%\python.exe -Im pip install setuptools wheel
Copy link
Contributor Author

@bebound bebound Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For setuptools:
Although setuptools is added to azure-cli dependency list in #27196, azure-cli is installed with --no-deps. Need to install it manually.

For wheel:
In 3.11, wheel is also installed when running get-pip.py. In 3.12, it's not installed. wheel is not included in final package in Linux and macOS. Actually, pip can still install wheel package without wheel. setuptools also removed wheel dependency in pypa/setuptools#1386.

However, wheel is still a factor for pip to use pep517 or not: https://github.com/pypa/pip/blob/ec5faeac4ef6bae97df0e779566ceb2b0de89d3f/src/pip/_internal/pyproject.py#L107 (ref: pypa/packaging.python.org#1517). If wheel is not installed, pip will use pep517 and fails, see #29887 (comment). The reason is that the -I param is missing in pip's build env. (Ref: pypa/pip#9081). See discussion in: pypa/pip#13023

So I also install wheel. (I think it's okay to remove wheel in the future)

# 3.11's get-pip.py

Successfully installed pip-24.2 setuptools-75.1.0 wheel-0.44.0
Pip set up successful

# see https://github.com/Azure/azure-cli/pull/29887
extra_dependencies = ['pip', 'setuptools']
for dependency in extra_dependencies:
nodes[dependency] = research_package(dependency)
Copy link
Contributor Author

@bebound bebound Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make_graph explicitly exclude pip and setuptools: https://github.com/tdsmith/homebrew-pypi-poet/blob/fdafc615bcd28f29bcbe90789f07cc26f97c3bbc/poet/poet.py#L131

Adding pip and setuptools in CLI's dependency does not work as brew's pip uses --no-deps option. Homebrew/homebrew-core#194884 (comment)

Comment on lines +30 to +31
# Does not exit if az extension add fails until all extensions have been tested
set +e
Copy link
Member

@jiasli jiasli Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will set +e make the script no longer fail in case of extension installation failure?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is similar to #14295 (comment).

Copy link
Contributor Author

@bebound bebound Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_extensions.sh calls scripts/ci/artifacts.sh which calls scripts/ci/build.sh.

build.sh has set -e.

Once -e is set, this script fails immediately if one extension fails. set +e ensures all extension have been tested. Finally, line 59 returns exit_code.

@@ -2,6 +2,8 @@

root=$(cd $(dirname $0); pwd)

tdnf install -y ca-certificates
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We switched the docker image from mcr.microsoft.com/mirror/docker/library/python:3.11-slim to mcr.microsoft.com/azurelinux/base/python:3 which doesn't intall ca-certificates by default.

@bebound bebound merged commit b7d9b30 into Azure:dev Oct 24, 2024
52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot Packaging
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants