Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{C} Use Azure Artifacts feeds in release branch #29866

Merged
merged 17 commits into from
Oct 16, 2024
Merged

{C} Use Azure Artifacts feeds in release branch #29866

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9530c8f
Init
bebound Sep 9, 2024
bf34a9b
Minor fix
bebound Sep 9, 2024
1c5b9d0
Add zip
bebound Sep 9, 2024
2391f5e
Minor fix
bebound Sep 9, 2024
8930bba
Add wheel
bebound Sep 9, 2024
b80bb6b
Minor fix
bebound Sep 9, 2024
dc7390c
Minor fix
bebound Sep 9, 2024
95a88e6
Change feed
bebound Sep 12, 2024
62602bd
Use condition
bebound Oct 11, 2024
892cf44
Merge branch 'dev' into cfs
bebound Oct 11, 2024
4e10196
Final
bebound Oct 12, 2024
491f7fe
Add cfs_feed variable
bebound Oct 14, 2024
d6214cc
Minor fix
bebound Oct 14, 2024
34aaaa1
Change variable name
bebound Oct 15, 2024
39d6240
Merge branch 'dev' into cfs
bebound Oct 15, 2024
31fa29f
Minor
bebound Oct 15, 2024
2c9ba0d
Minor fix
bebound Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ trigger:
branches:
include:
- '*'
exclude:
- 'release'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The public pipeline does not have the permission to run PipAuthenticate@1 task, so skip it. This also saves some resource during CLI release.


pr:
branches:
Expand Down Expand Up @@ -205,6 +207,11 @@ jobs:
TargetPath: '$(Build.ArtifactStagingDirectory)/metadata'
artifactName: metadata

- task: PipAuthenticate@1
condition: eq(variables['Build.SourceBranch'], 'refs/heads/release')
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: ${{ variables.AZURE_ARTIFACTS_FEEDS }}
Copy link
Member

@jiasli jiasli Oct 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other tasks use macro syntax as $(variables.AZURE_ARTIFACTS_FEEDS). Does template expression syntax work here?

My understanding is that ${{ variables.AZURE_ARTIFACTS_FEEDS }} only works for templates: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops&pivots=templates-includes#variable-reuse

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, the variable defined in ADO UI is empty when use ${{ variables.AZURE_ARTIFACTS_FEEDS }}. I've changed to $(AZURE_ARTIFACTS_FEEDS)


- script: |
set ARCH=$(Platform)
Expand Down Expand Up @@ -244,6 +251,11 @@ jobs:
TargetPath: '$(Build.ArtifactStagingDirectory)/metadata'
artifactName: metadata

- task: PipAuthenticate@1
condition: eq(variables['Build.SourceBranch'], 'refs/heads/release')
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: ${{ variables.AZURE_ARTIFACTS_FEEDS }}

- script: |
set ARCH=$(Platform)
Expand Down Expand Up @@ -429,6 +441,11 @@ jobs:
inputs:
versionSpec: 3.11

- task: PipAuthenticate@1
condition: eq(variables['Build.SourceBranch'], 'refs/heads/release')
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: ${{ variables.AZURE_ARTIFACTS_FEEDS }}

- script: |
if [[ "$(Build.Reason)" == "PullRequest" ]]; then
Expand Down Expand Up @@ -719,6 +736,12 @@ jobs:
- bash: ./scripts/ci/install_docker.sh
displayName: Install Docker

- task: PipAuthenticate@1
condition: eq(variables['Build.SourceBranch'], 'refs/heads/release')
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: ${{ variables.AZURE_ARTIFACTS_FEEDS }}

- task: Bash@3
displayName: 'Build Rpm Package: Azure Linux'
inputs:
Expand Down Expand Up @@ -809,6 +832,11 @@ jobs:
steps:
- bash: ./scripts/ci/install_docker.sh
displayName: Install Docker
- task: PipAuthenticate@1
condition: eq(variables['Build.SourceBranch'], 'refs/heads/release')
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: ${{ variables.AZURE_ARTIFACTS_FEEDS }}
- task: Bash@3
displayName: 'Build Rpm Package'
inputs:
Expand Down Expand Up @@ -926,6 +954,11 @@ jobs:
steps:
- bash: ./scripts/ci/install_docker.sh
displayName: Install Docker
- task: PipAuthenticate@1
condition: eq(variables['Build.SourceBranch'], 'refs/heads/release')
displayName: 'Pip Authenticate'
inputs:
artifactFeeds: ${{ variables.AZURE_ARTIFACTS_FEEDS }}
- task: Bash@3
displayName: 'Build $(deb_system) $(distro) $(arch) Package'
inputs:
Expand Down
1 change: 1 addition & 0 deletions scripts/release/debian/pipeline.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ docker run --rm \
-e CLI_VERSION=$CLI_VERSION \
-e CLI_VERSION_REVISION=1~$DISTRO \
-e DEBIAN_FRONTEND=noninteractive \
-e PIP_INDEX_URL=$PIP_INDEX_URL \
$DISTRO_BASE_IMAGE \
/mnt/repo/scripts/release/debian/build.sh
3 changes: 2 additions & 1 deletion scripts/release/rpm/azurelinux.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ COPY . .
# Azure Linux 3's python3 is 3.12, the rpm paths are
# /usr/src/azl/RPMS/x86_64/azure-cli-2.63.0-1.azl3.x86_64.rpm
# /usr/src/azl/RPMS/aarch64/azure-cli-2.63.0-1.azl3.aarch64.rpm
RUN dos2unix ./scripts/release/rpm/azure-cli.spec && \
RUN --mount=type=secret,id=PIP_INDEX_URL export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) && \
dos2unix ./scripts/release/rpm/azure-cli.spec && \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=python3 PYTHON_CMD=python3 \
rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec && \
cp /usr/src/*/RPMS/*/azure-cli-${cli_version}-1.*.rpm /azure-cli-dev.rpm && \
Expand Down
4 changes: 4 additions & 0 deletions scripts/release/rpm/pipeline.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,16 @@ set -exv

CLI_VERSION=`cat src/azure-cli/azure/cli/__main__.py | grep __version__ | sed s/' '//g | sed s/'__version__='// | sed s/\"//g`

# PIP_INDEX_URL env must exist in `docker build --secret`, use an empty string if it doesn't exist.
export PIP_INDEX_URL=${PIP_INDEX_URL}

# Create a container image that includes the source code and a built RPM using this file.
docker build \
--target build-env \
--build-arg cli_version=${CLI_VERSION} \
--build-arg image=${IMAGE} \
--build-arg python_package=${PYTHON_PACKAGE} \
--secret id=PIP_INDEX_URL \
-f ./scripts/release/rpm/${DOCKERFILE}.dockerfile \
-t azure/azure-cli:${DOCKERFILE}-builder \
.
Expand Down
4 changes: 4 additions & 0 deletions scripts/release/rpm/pipeline_azurelinux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,15 @@ set -exv

CLI_VERSION=`cat src/azure-cli/azure/cli/__main__.py | grep __version__ | sed s/' '//g | sed s/'__version__='// | sed s/\"//g`

# PIP_INDEX_URL env must exist in `docker build --secret`, use an empty string if it doesn't exist.
export PIP_INDEX_URL=${PIP_INDEX_URL}
Copy link
Contributor Author

@bebound bebound Oct 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error message without empty env during docker build: ERROR: failed to stat PIP_INDEX_URL: stat PIP_INDEX_URL: no such file or directory


# Create a container image that includes the source code and a built RPM using this file.
docker build \
--target build-env \
--build-arg cli_version=${CLI_VERSION} \
--build-arg image=${IMAGE} \
--secret id=PIP_INDEX_URL \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--build-arg also works, but I prefer --secret as the index url contains credential and secret does not record its value in image.

Ref: https://docs.docker.com/build/building/secrets/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is interesting that env can be omitted:

https://docs.docker.com/build/building/secrets/#sources

When you use secrets from environment variables, you can omit the env parameter to bind the secret to a file with the same name as the variable.

-f ./scripts/release/rpm/azurelinux.dockerfile \
-t azure/azure-cli:azurelinux-builder \
.
Expand Down
3 changes: 2 additions & 1 deletion scripts/release/rpm/ubi.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ COPY . .

# RHEL 8's 'python3' is Python 3.6. RHEL 9's 'python3' is Python 3.9.
# We have to explicitly specify 'python39' to install Python 3.9.
RUN dos2unix ./scripts/release/rpm/azure-cli.spec && \
RUN --mount=type=secret,id=PIP_INDEX_URL export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) && \
dos2unix ./scripts/release/rpm/azure-cli.spec && \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=python3.9 \
rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec && \
cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /azure-cli-dev.rpm && \
Expand Down