Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Packaging] Support Windows ZIP package #27911

Merged
merged 21 commits into from
Jan 30, 2024
Merged

[Packaging] Support Windows ZIP package #27911

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
452a3dc
Init
bebound Nov 24, 2023
181d34b
Beta test
bebound Nov 24, 2023
2125e39
Minor fix
bebound Nov 27, 2023
74aa576
Add test
bebound Nov 27, 2023
fda9040
Minor fix
bebound Nov 28, 2023
33b2cf7
Add test
bebound Nov 28, 2023
79e3c18
Use 7z
bebound Nov 30, 2023
ef1f743
Add upgrade
bebound Nov 30, 2023
d926724
Minor fix
bebound Nov 30, 2023
72667ad
Minor fix
bebound Dec 4, 2023
deb94ac
Use LZMA
bebound Dec 6, 2023
7ed5497
Minor fix
bebound Dec 11, 2023
63207d3
Add azps_zip.ps1
bebound Jan 5, 2024
6fa7aa4
Remove x86
bebound Jan 15, 2024
0956882
Minor fix
bebound Jan 17, 2024
52552df
Minor fix
bebound Jan 17, 2024
3e6f1b5
Minor fix
bebound Jan 17, 2024
dd0f8bc
Merge branch 'dev' into zip
bebound Jan 17, 2024
95264cd
Remove root folder, use bin folder
bebound Jan 22, 2024
ac32b2d
MInor fix
bebound Jan 23, 2024
6fb6c8e
MInor fix
bebound Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ jobs:

- script: |
set ARCH=$(Platform)
set TARGET=msi
set /p CLI_VERSION=<$(System.ArtifactsDirectory)/metadata/version
set

Expand All @@ -228,6 +229,47 @@ jobs:
TargetPath: 'build_scripts/windows/out/'
ArtifactName: msi-$(Platform)

- job: BuildWindowsZIP
displayName: Build Windows ZIP
strategy:
matrix:
x86:
Platform: x86
x64:
Platform: x64

dependsOn: ExtractMetadata
condition: succeeded()
pool:
name: ${{ variables.windows_pool }}
steps:
- task: DownloadPipelineArtifact@1
displayName: 'Download Build Artifacts'
inputs:
TargetPath: '$(Build.ArtifactStagingDirectory)/metadata'
artifactName: metadata


- script: |
set ARCH=$(Platform)
set TARGET=zip
set /p CLI_VERSION=<$(System.ArtifactsDirectory)/metadata/version
set

build_scripts/windows/scripts/build.cmd
displayName: 'Build Windows ZIP'

- task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0
displayName: 'SBOM'
inputs:
BuildDropPath: 'build_scripts/windows/out/'

- task: PublishPipelineArtifact@0
displayName: 'Publish Artifact: ZIP'
inputs:
TargetPath: 'build_scripts/windows/out/'
ArtifactName: zip-$(Platform)

- job: TestMsiInstallation
displayName: Test MSI Installation
strategy:
Expand Down Expand Up @@ -259,6 +301,21 @@ jobs:
inputs:
filePath: build_scripts\windows\scripts\test_msi_installation.ps1

- job: TestZipInstallation
displayName: Test ZIP Installation
dependsOn: BuildWindowsZIP
strategy:
matrix:
x86:
Platform: x86
x64:
Platform: x64
condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule'))
pool:
name: ${{ variables.windows_pool }}
steps:
- task:

- job: BuildDockerImageAlpine
displayName: Build Docker Image Alpine
strategy:
Expand Down
12 changes: 12 additions & 0 deletions build_scripts/windows/scripts/az_zip.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
::
:: Microsoft Azure CLI - Windows Installer - Author file components script
:: Copyright (C) Microsoft Corporation. All Rights Reserved.
::

@IF EXIST "%~dp0\..\python.exe" (
SET AZ_INSTALLER=ZIP
"%~dp0\..\python.exe" -IBm azure.cli %*
) ELSE (
echo Failed to load python executable.
exit /b 1
)
65 changes: 42 additions & 23 deletions build_scripts/windows/scripts/build.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,15 @@ if "%CLI_VERSION%"=="" (
echo Please set the CLI_VERSION environment variable, e.g. 2.0.13
goto ERROR
)

@REM ARCH can be x86 or x64
if "%ARCH%"=="" (
set ARCH=x86
)
@REM TARGET can be msi or zip
if "%TARGET%"=="" (
set TARGET=msi
)

if "%ARCH%"=="x86" (
set PYTHON_ARCH=win32
) else if "%ARCH%"=="x64" (
Expand All @@ -43,6 +48,7 @@ set ARTIFACTS_DIR=%~dp0..\artifacts
mkdir %ARTIFACTS_DIR%
set TEMP_SCRATCH_FOLDER=%ARTIFACTS_DIR%\cli_scratch
set BUILDING_DIR=%ARTIFACTS_DIR%\cli
set ZIP_DIR=%ARTIFACTS_DIR%\Azure CLI
Copy link
Member

@jiasli jiasli Jan 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A folder name with space (Azure CLI) causes lots of trouble when the full path is being invoked from PowerShell.

For example, the MSI installs the Azure CLI executable at C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd.

It can't be directly executed because of the space in the path:

> C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd
C:\Program: The term 'C:\Program' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

If we quote it, PowerShell treats it as a string and echoes it:

> "C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd"
C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd

The correct way to invoke the executable with its full path is:

> & "C:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmd"

Welcome to Azure CLI!
---------------------
Use `az -h` to see available commands or go to https://aka.ms/cli.

Of course, we don't want such complex way of invoking az.cmd.

I suggest either

  1. renaming Azure CLI to azure-cli to align with other packages, such as PyPI, DEB, RPM, or
  2. putting Azure CLI files directly under the root of the ZIP

set WIX_DIR=%ARTIFACTS_DIR%\wix
set PYTHON_DIR=%ARTIFACTS_DIR%\Python

Expand Down Expand Up @@ -71,20 +77,22 @@ if exist %REPO_ROOT%\privates (
copy %REPO_ROOT%\privates\*.whl %TEMP_SCRATCH_FOLDER%
)

REM ensure wix is available
if exist %WIX_DIR% (
echo Using existing Wix at %WIX_DIR%
)
if not exist %WIX_DIR% (
mkdir %WIX_DIR%
pushd %WIX_DIR%
echo Downloading Wix.
curl --output wix-archive.zip %WIX_DOWNLOAD_URL%
unzip wix-archive.zip
if %errorlevel% neq 0 goto ERROR
del wix-archive.zip
echo Wix downloaded and extracted successfully.
popd
if "%TARGET%" == 'msi' (
REM ensure wix is available
if exist %WIX_DIR% (
echo Using existing Wix at %WIX_DIR%
)
if not exist %WIX_DIR% (
mkdir %WIX_DIR%
pushd %WIX_DIR%
echo Downloading Wix.
curl --output wix-archive.zip %WIX_DOWNLOAD_URL%
unzip wix-archive.zip
if %errorlevel% neq 0 goto ERROR
del wix-archive.zip
echo Wix downloaded and extracted successfully.
popd
)
)

REM ensure Python is available
Expand Down Expand Up @@ -122,7 +130,6 @@ set PYTHON_EXE=%PYTHON_DIR%\python.exe
robocopy %PYTHON_DIR% %BUILDING_DIR% /s /NFL /NDL

set CLI_SRC=%REPO_ROOT%\src
%BUILDING_DIR%\python.exe -m pip install --no-warn-script-location --force-reinstall pycparser==2.18
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pycparser==2.19 is installed later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azure-cli/build_scripts/windows/scripts/build.cmd

Line 138 in 6fa7aa4

%BUILDING_DIR%\python.exe -m pip install --no-warn-script-location --requirement %CLI_SRC%\azure-cli\requirements.py3.windows.txt

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also see #27196 (comment)

for %%a in (%CLI_SRC%\azure-cli %CLI_SRC%\azure-cli-core %CLI_SRC%\azure-cli-telemetry) do (
pushd %%a
%BUILDING_DIR%\python.exe -m pip install --no-warn-script-location --no-cache-dir --no-deps .
Expand All @@ -137,18 +144,25 @@ if %errorlevel% neq 0 goto ERROR

pushd %BUILDING_DIR%
%BUILDING_DIR%\python.exe %REPO_ROOT%\scripts\compact_aaz.py
if %errorlevel% neq 0 goto ERROR
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check the error code of last command.
compact_aaz.py may fails because of #27923

%BUILDING_DIR%\python.exe %~dp0\patch_models_v2.py
if %errorlevel% neq 0 goto ERROR
%BUILDING_DIR%\python.exe %REPO_ROOT%\scripts\trim_sdk.py
if %errorlevel% neq 0 goto ERROR
popd

REM Remove pywin32 help file to reduce size.
del %BUILDING_DIR%\Lib\site-packages\PyWin32.chm

echo Creating the wbin (Windows binaries) folder that will be added to the path...
mkdir %BUILDING_DIR%\wbin
copy %REPO_ROOT%\build_scripts\windows\scripts\az.cmd %BUILDING_DIR%\wbin\
copy %REPO_ROOT%\build_scripts\windows\scripts\azps.ps1 %BUILDING_DIR%\wbin\
copy %REPO_ROOT%\build_scripts\windows\scripts\az %BUILDING_DIR%\wbin\
if "%TARGET%"=="msi" (
copy %REPO_ROOT%\build_scripts\windows\scripts\az.cmd %BUILDING_DIR%\wbin\
copy %REPO_ROOT%\build_scripts\windows\scripts\azps.ps1 %BUILDING_DIR%\wbin\
copy %REPO_ROOT%\build_scripts\windows\scripts\az %BUILDING_DIR%\wbin\
) else (
copy %REPO_ROOT%\build_scripts\windows\scripts\az_zip.cmd %BUILDING_DIR%\wbin\az.cmd
Copy link
Contributor Author

@bebound bebound Nov 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azps.ps1 and az are not included in zip as they cause other problems. See #26682

Copy link
Member

@jiasli jiasli Jan 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other Microsoft-owned apps mentioned in #27911 (comment) put their executables such as Code.exe, pwsh.exe under the root folder. Perhaps we can do the same and put az.cmd under the root folder?

But this approach has its own problem: adding Azure CLI's folder to PATH will also put python.exe under PATH. If a user accidentally executes python.exe, that may corrupt Azure CLI's virtual environment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to maintain the current structure to avoid adding other executables into PATH.
PS: Other CLI tools also place their executables to dedicated folder. For example: C:\Program Files\Docker\Docker\resources\bin

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we use wbin, instead of bin?

Copy link
Member

@jiasli jiasli Jan 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. The explanation is right in the comment:

azure-cli/build_scripts/windows/scripts/build.cmd

Lines 147 to 148 in 04d82da

echo Creating the wbin (Windows binaries) folder that will be added to the path...
mkdir %BUILDING_DIR%\wbin

It was added by #2655.

I initially thought it means

A Unicode version with the letter "W" used to indicate "wide"
-- https://learn.microsoft.com/en-us/windows/win32/intl/unicode-in-the-windows-api

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this package is for Windows, w seems redundant.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other Microsoft-owned apps mentioned in #27911 (comment) put their executables such as Code.exe, pwsh.exe under the root folder. Perhaps we can do the same and put az.cmd under the root folder?

But this approach has its own problem: adding Azure CLI's folder to PATH will also put python.exe under PATH. If a user accidentally executes python.exe, that may corrupt Azure CLI's virtual environment.

Adding python in the path could also create instability if the platform (for example a custom build agent) is using python for other purposes.

I think that wbin will be confusing and the information it adds is not necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dcaro I can change the folder to bin in zip package.
Should we also do this in MSI? We never mention wbin in doc, and it's automatically added into PATH during MSI installation. This change should not affect users unless they use full path to run az.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's leave the folder name in the msi as is for now, as you mention, it is automatically added to path and transparent to users.

)
if %errorlevel% neq 0 goto ERROR
copy %REPO_ROOT%\build_scripts\windows\resources\CLI_LICENSE.rtf %BUILDING_DIR%
copy %REPO_ROOT%\build_scripts\windows\resources\ThirdPartyNotices.txt %BUILDING_DIR%
Expand Down Expand Up @@ -193,14 +207,19 @@ for /d %%d in ("azure*.dist-info") do (
)
popd

if %errorlevel% neq 0 goto ERROR

echo Building MSI...
msbuild /t:rebuild /p:Configuration=Release /p:Platform=%ARCH% %REPO_ROOT%\build_scripts\windows\azure-cli.wixproj
if "%TARGET%"=="msi" (
echo Building MSI...
msbuild /t:rebuild /p:Configuration=Release /p:Platform=%ARCH% %REPO_ROOT%\build_scripts\windows\azure-cli.wixproj
) else(
echo Building ZIP...
ren %BUILDING_DIR% "Azure CLI"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Azure CLI as root folder of zip package.
PS: The MSI installation path is C:\Program Files\Microsoft SDKs\Azure\CLI2.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

set BUILDING_DIR=%ARTIFACTS_DIR%\cli
set ZIP_DIR=%ARTIFACTS_DIR%\Azure CLI

ren %BUILDING_DIR% "Azure CLI" renames BUILDING_DIR to %ARTIFACTS_DIR%\Azure CLI. Let's not hard code Azure CLI but use a variable.

powershell Compress-Archive -Path '%ZIP_DIR%' -DestinationPath %OUTPUT_DIR%\azure-cli-%CLI_VERSION%.zip
)

if %errorlevel% neq 0 goto ERROR

echo %OUTPUT_DIR%
echo Output Dir: %OUTPUT_DIR%

goto END

Expand Down