Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Packaging] Support x86 and x64 MSI builds #26640

Merged
merged 8 commits into from
Jul 20, 2023
Merged

[Packaging] Support x86 and x64 MSI builds #26640

merged 8 commits into from
Jul 20, 2023

Conversation

heaths
Copy link
Member

@heaths heaths commented Jun 9, 2023

Close #18766

@azure-client-tools-bot-prd
Copy link

azure-client-tools-bot-prd bot commented Jun 9, 2023

️✔️AzureCLI-FullTest
️✔️acr
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️acs
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️advisor
️✔️latest
️✔️3.10
️✔️3.9
️✔️ams
️✔️latest
️✔️3.10
️✔️3.9
️✔️apim
️✔️latest
️✔️3.10
️✔️3.9
️✔️appconfig
️✔️latest
️✔️3.10
️✔️3.9
️✔️appservice
️✔️latest
️✔️3.10
️✔️3.9
️✔️aro
️✔️latest
️✔️3.10
️✔️3.9
️✔️backup
️✔️latest
️✔️3.10
️✔️3.9
️✔️batch
️✔️latest
️✔️3.10
️✔️3.9
️✔️batchai
️✔️latest
️✔️3.10
️✔️3.9
️✔️billing
️✔️latest
️✔️3.10
️✔️3.9
️✔️botservice
️✔️latest
️✔️3.10
️✔️3.9
️✔️cdn
️✔️latest
️✔️3.10
️✔️3.9
️✔️cloud
️✔️latest
️✔️3.10
️✔️3.9
️✔️cognitiveservices
️✔️latest
️✔️3.10
️✔️3.9
️✔️config
️✔️latest
️✔️3.10
️✔️3.9
️✔️configure
️✔️latest
️✔️3.10
️✔️3.9
️✔️consumption
️✔️latest
️✔️3.10
️✔️3.9
️✔️container
️✔️latest
️✔️3.10
️✔️3.9
️✔️core
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️cosmosdb
️✔️latest
️✔️3.10
️✔️3.9
️✔️databoxedge
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️dla
️✔️latest
️✔️3.10
️✔️3.9
️✔️dls
️✔️latest
️✔️3.10
️✔️3.9
️✔️dms
️✔️latest
️✔️3.10
️✔️3.9
️✔️eventgrid
️✔️latest
️✔️3.10
️✔️3.9
️✔️eventhubs
️✔️latest
️✔️3.10
️✔️3.9
️✔️feedback
️✔️latest
️✔️3.10
️✔️3.9
️✔️find
️✔️latest
️✔️3.10
️✔️3.9
️✔️hdinsight
️✔️latest
️✔️3.10
️✔️3.9
️✔️identity
️✔️latest
️✔️3.10
️✔️3.9
️✔️iot
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️keyvault
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️kusto
️✔️latest
️✔️3.10
️✔️3.9
️✔️lab
️✔️latest
️✔️3.10
️✔️3.9
️✔️managedservices
️✔️latest
️✔️3.10
️✔️3.9
️✔️maps
️✔️latest
️✔️3.10
️✔️3.9
️✔️marketplaceordering
️✔️latest
️✔️3.10
️✔️3.9
️✔️monitor
️✔️latest
️✔️3.10
️✔️3.9
️✔️mysql
️✔️latest
️✔️3.10
️✔️3.9
️✔️netappfiles
️✔️latest
️✔️3.10
️✔️3.9
️✔️network
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️policyinsights
️✔️latest
️✔️3.10
️✔️3.9
️✔️privatedns
️✔️latest
️✔️3.10
️✔️3.9
️✔️profile
️✔️latest
️✔️3.10
️✔️3.9
️✔️rdbms
️✔️latest
️✔️3.10
️✔️3.9
️✔️redis
️✔️latest
️✔️3.10
️✔️3.9
️✔️relay
️✔️latest
️✔️3.10
️✔️3.9
️✔️resource
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️role
️✔️latest
️✔️3.10
️✔️3.9
️✔️search
️✔️latest
️✔️3.10
️✔️3.9
️✔️security
️✔️latest
️✔️3.10
️✔️3.9
️✔️servicebus
️✔️latest
️✔️3.10
️✔️3.9
️✔️serviceconnector
️✔️latest
️✔️3.10
️✔️3.9
️✔️servicefabric
️✔️latest
️✔️3.10
️✔️3.9
️✔️signalr
️✔️latest
️✔️3.10
️✔️3.9
️✔️sql
️✔️latest
️✔️3.10
️✔️3.9
️✔️sqlvm
️✔️latest
️✔️3.10
️✔️3.9
️✔️storage
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️synapse
️✔️latest
️✔️3.10
️✔️3.9
️✔️telemetry
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️util
️✔️latest
️✔️3.10
️✔️3.9
️✔️vm
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9

@azure-client-tools-bot-prd
Copy link

azure-client-tools-bot-prd bot commented Jun 9, 2023

️✔️AzureCLI-BreakingChangeTest
️✔️Non Breaking Changes

@yonzhan
Copy link
Collaborator

yonzhan commented Jun 9, 2023

Thank you for your contribution! We will review the pull request and get back to you soon.

@heaths heaths marked this pull request as ready for review June 9, 2023 03:26
@yonzhan yonzhan requested a review from bebound June 9, 2023 03:31
azure-pipelines.yml Outdated Show resolved Hide resolved
@jiasli jiasli changed the title Support x86 and x64 MSI builds [Packaging] Support x86 and x64 MSI builds Jun 9, 2023
azure-pipelines.yml Outdated Show resolved Hide resolved
azure-pipelines.yml Outdated Show resolved Hide resolved
Separate agents will build separate packages and publish them as needed to avoid regressions to existing pipeline.
azure-pipelines.yml Outdated Show resolved Hide resolved
@jiasli
Copy link
Member

jiasli commented Jul 21, 2023

The upgrade behavior is as below:

  • If the architecture doesn't change (installing x86 MSI on top of x86 MSI, or installing x64 MSI on top of x64 MSI), only version upgrade is allowed; installing the same version or downgrade is not allowed.
  • Installing 64-bit MSI on top of 32-bit MSI will uninstall the 32-bit MSI automatically. Upgrade or installing the same version is allowed; downgrade is not allowed.
  • Installing 32-bit MSI on top of 64-bit MSI will keep both MSI's side-by-side. TODO: We will forbid this behavior in the future.

@jiasli
Copy link
Member

jiasli commented Aug 1, 2023

We have finalized related names to be

Architecture x86 x64
Product Name Microsoft Azure CLI (32-bit) Microsoft Azure CLI (64-bit)
MSI file name azure-cli-2.x.x.msi azure-cli-2.x.x-x64.msi
MSI short link https://aka.ms/installazurecliwindows https://aka.ms/installazurecliwindowsX64

The product name has been changed from Microsoft Azure CLI to Microsoft Azure CLI (32-bit) in order to better reflect the installed MSI's architecture and encourage users to migrate to the 64-bit MSI.

The product name change should not be considered a breaking change, because the official document https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows doesn't declare the product name, so the product name is not guaranteed to remain the same across versions.

If you have script relying on the product name Microsoft Azure CLI, please consider changing it accordingly.

@aluty
Copy link

aluty commented Aug 2, 2023

Documentation not yet updated with 32-bit and 64-bit information?
https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows?tabs=azure-cli#install-or-update

@bebound
Copy link
Contributor

bebound commented Aug 2, 2023

@aluty The PR is not merged yet, we are working on it. MicrosoftDocs/azure-docs-cli#3759

@jiasli
Copy link
Member

jiasli commented Aug 2, 2023

This new azure-cli-2.51.0.msi triggers #26464.

image

The weird thing is Python version is not upgraded between Azure CLI 2.50.0 and 2.51.0. I checked C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe from azure-cli-2.50.0.msi and azure-cli-2.51.0.msi.

Both are created on Tuesday, ‎February ‎7, ‎2023, ‏‎5:30:06 PM:

image

They have the same SHA256 cc78554da7fdb92f460e3f63d852fa0147485d112c8f55a69b3216e9bfe01242, so they are identical.

I am not sure about the exact root cause. Possible root causes are

  1. Upgrade method change from MajorUpgrade to Upgrade and InstallExecuteSequence
  2. Product name change from Microsoft Azure CLI to Microsoft Azure CLI (32-bit)

@heaths
Copy link
Member Author

heaths commented Aug 2, 2023

  1. The file in use is Resource Manager, which is working as designed. It's either shut it down, or the machine will have to be restarted later to clean it up. As long as it isn't running, it won't be a problem.
  2. The 64-bit MSI will install python into C:\Program Files\ not C:\Program Files (x86) where only 32-bit apps are supposed to go. To figure out why it wasn't upgraded (and are you sure you updated Python from 2.50 to 2.51?), you'll need to capture logs:
msiexec /i <path to MSI> /l*vx %TEMP%\install.log

Attach or send to me and I can take a look. I did test the various upgrade paths (new install, old -> new for x86, old x86 -> new x64, and new x86 -> new x64) and they all worked for me.

@jiasli
Copy link
Member

jiasli commented Aug 3, 2023

Thanks @heaths for the comment.

The file in use is Resource Manager

May I know which "Resource Manager" you are referring to? Is it a file?

How az upgrade (a python.exe process) works is that it subprocesses msiexec.exe and waits for the msiexec.exe process to exit:

exit_code = subprocess.call(['msiexec.exe', '/i', msi_path])

It currently only supports 32-bit Azure CLI and 32-bit MSI.

This issue doesn't happen if you directly install MSI and not through az upgrade, as there is no python.exe running.

Previously before this PR, we have observed:

  • if Python version is not upgraded during the installation, this Files in Use window will not be shown
  • if Python version is upgraded, e.g. 3.10.9 -> 3.10.11, this Files in Use window will be shown

are you sure you updated Python from 2.50 to 2.51?

We did not upgrade Python from Azure CLI 2.50 to 2.51 and the python.exe are exactly the same 3.10.10 in those Azure CLI versions:

set PYTHON_VERSION=3.10.10

The weird thing I mentioned in #26640 (comment) contradicts the previous behavior and we are not sure why.

@heaths
Copy link
Member Author

heaths commented Aug 3, 2023

Restart Manager is a Windows feature that's been in since Vista, IIRC. You can search for it. Because python.exe is waiting for msiexec.exe to finish, this will happen. It isn't new based on this PR. You can pass REBOOT=ReallySuppress to the command line to ignore it, but msiexec.exe may return a 3010 error code saying a reboot is required. Better, just kick off msiexec.exe and exit. Why wait for it? Let it run in basic UI mode and any errors will be surfaced to the user.

@jiasli
Copy link
Member

jiasli commented Aug 4, 2023

Because python.exe is waiting for msiexec.exe to finish, this will happen.

Our previously observation is that if Python version is not upgraded during the installation, this Files in Use window will not be shown. That's why we are curious about it.

Why wait for it? Let it run in basic UI mode and any errors will be surfaced to the user.

This is exactly what #26464 is doing.

#14803 added the az upgrade command. It launches msiexec.exe to install the MSI from URL https://aka.ms/installazurecliwindows, and then waits for msiexec.exe to complete.

We refined it in #19192 to download the MSI to a temporary folder under TEMP (env var) first, and then install that local MSI. For example:

> az upgrade
...
Updating Azure CLI with MSI from https://aka.ms/installazurecliwindows
Downloading MSI to C:\Users\xxx\AppData\Local\Temp\tmpr1onzz23\azure-cli-2.51.0.msi

Once az upgrade knows msiexec.exe succeeds, it'll

  1. upgrade extensions:

    subprocess.call(['az', 'extension', 'update', '-n', ext_name],
    shell=platform.system() == 'Windows')

  2. delete the local MSI:

If az upgrade kicks off msiexec.exe and exits,

  1. it will not have the chance to upgrade extensions. Users will have to run az upgrade again to trigger the upgrading extension logic. We can't install extensions first as some extensions may depend on newer versions of Azure CLI core.

  2. it will not have the chance to delete the local MSI. Since the MSI is downloaded to a temporary folder under TEMP, a new az upgrade process won't know where the previous az upgrade's MSI is saved (in the above example, tmpr1onzz23), so the MSI will be left there until the user manually runs Disk Cleanup.

@heaths
Copy link
Member Author

heaths commented Aug 7, 2023

Our previously observation is that if Python version is not upgraded during the installation, this Files in Use window will not be shown. That's why we are curious about it.

And that confirms what I said in my comment above. If the file (component) doesn't need to be upgraded, Restart Manager isn't used for that file because it won't be touched. Once it's upgraded, though, this functionality is working exactly as designed. So you can either make sure (as much as you can) python.exe isn't running (don't wait), or try passing REBOOT=ReallySuppress. The latter may or may not work. Alternatively, find a way to create the process for python.exe that opens it with FILE_SHARE_DELETE or, if possible, create a temp copy of it e.g., _python.exe and exec that.

@jiasli
Copy link
Member

jiasli commented Aug 8, 2023

make sure (as much as you can) python.exe isn't running (don't wait)

We will do this in #26464.

find a way to create the process for python.exe that opens it with FILE_SHARE_DELETE

There is no fancy logic currently to create the python.exe process - python.exe is launched by the az.cmd entry script:

"%~dp0\..\python.exe" -IBm azure.cli %*

create a temp copy of it e.g., _python.exe and exec that

In order to know whether _python.exe succeeds, the original python.exe needs to be alive, making it unable to be updated. I provided more detail in #26464 (comment).

@heaths
Copy link
Member Author

heaths commented Aug 8, 2023

In order to know whether _python.exe succeeds, the original python.exe needs to be alive, making it unable to be updated. I provided more detail in #26464 (comment).

Nothing is stopping you from copying the python.exe process to another file and executing that in the same directory, so that all library and package references are resolved correctly. python.exe would be replaced while _python.exe (e.g.) continues to run and you can still wait for it. This is a common workaround for cases like this.

@jiasli
Copy link
Member

jiasli commented Aug 9, 2023

python.exe would be replaced while _python.exe (e.g.) continues to run and you can still wait for it.

As far as I know, this is not possible on Windows, because Windows doesn't support replacing a process like exec* on Linux. os.exec* on Windows internally calls CreateProcess to create a new process and exits immediately. The new process will be created as a background process and the control will be given back immediately to the terminal. There's no way of knowing if the new process succeeds or not.

This is of course not what we want - we want the control to be given back to the terminal after _python.exe finishes, and we also need to know if _python.exe succeeds.

Python exposes exec* via os.exec*, but these functions behave differently on Windows and Linux. This issue has reported multiple times to the Python repo more than 10 years ago, such as python/cpython#63323.


Another workaround is to modify the az.cmd entry script to install the MSI, but the Return on investment (ROI) seems to be very low (#26464 (comment)).

@heaths
Copy link
Member Author

heaths commented Aug 9, 2023

Apps using the console subsystem do not return control immediately - only windows subsystem apps. python.exe is compiled to use the console subsystem. Even if it used the windows subsystem, you can use start /wait python.exe ... to wait for it and get its %ERRORLEVEL% after running. You could always have your batch script copy it and wait on _python.exe as well.

The main point is that: if you're going to hold python.exe in use, unless you pass REBOOT=ReallySuppress to the command line when calling msiexec.exe (and I'm not 100% sure that will work; should, but it's been a long time since I've tested that) and python.exe is getting upgrade, this is exactly the behavior you should expect. This has been in place since Windows Vista or so. Maybe XP. Restart Manager and its use in MSI is not new.

@jiasli
Copy link
Member

jiasli commented Aug 10, 2023

Apps using the console subsystem do not return control immediately - only windows subsystem apps. python.exe is compiled to use the console subsystem.

What I was thinking is the pure Python solution. Using exec* to invoke _python.exe does return control immediately, as demonstrated by python/cpython#63323.

It doesn't return immediately if the caller python.exe invokes the callee _python.exe using subprocess and wait for the callee to complete. In that case, the caller will still be "file in use".

You could always have your batch script copy it and wait on _python.exe as well.

This is doable with a pure cmd script solution, but I think that'll overly complicate the az.cmd entry script and make it too heavy.

@jiasli
Copy link
Member

jiasli commented Aug 17, 2023

Azure DevOps runner image (Windows) has switched to 64-bit MSI (actions/runner-images#8096). Runners will be updated gradually after 2023-8-25.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Azure CLI 64-bit on Windows
5 participants