Skip to content
This repository has been archived by the owner on Aug 16, 2024. It is now read-only.

Added multiarch support by docker buildx #336

Merged
merged 1 commit into from
Mar 8, 2021

Conversation

bivasda1
Copy link
Contributor

Signed-off-by: Bivas Das bivasda1@in.ibm.com

What does this PR do?

Enables docker buildx support in GitHub Actions job to build and publish multiarch docker images. All the workflows (nightly build, PR check, and release) are updated to have multiarch support.

What issues does this PR fix or reference?

This refers to #321

Reviewers

@ericwill Could you please review this PR

@bivasda1
Copy link
Contributor Author

@ericwill ericwill requested a review from nickboldt February 10, 2021 19:39
Copy link
Contributor

@ericwill ericwill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few changes are required, please see comments inline. Also, please test to make sure that the regression from the last attempt doesn't happen this time.

@bivasda1
Copy link
Contributor Author

@ericwill Can you please review this PR, I have changed Accordingly

Below change requests are updated

  • Copyright header and also year is Updated
  • Enabled log into both quay.io and dockerhub
  • Enabled multiarch build using buildx for Both the happy path image and the base images
  • Added Platformfile

@bivasda1
Copy link
Contributor Author

@ericwill added Platform file in arbitrary-users-patch and arbitrary-users-patch/happy-path location, accordingly modified the script.

ericwill
ericwill previously approved these changes Feb 19, 2021
@ericwill
Copy link
Contributor

@bivasda1 please rebase the PR and let's merge it next week. There is a 7.26.2 release next week so that will be a good test run.

@nickboldt
Copy link
Contributor

@mkuznyetsov FYI ^

@bivasda1 if rebased and ready by EOD Tuesday, we can include in Wednesday's release. If not... will slip to the week after.

Note too that we've been having issues with qemu emulated s390x buildx builds, so you might want to only include ppc64le in this PR as the s390x build might make it harder to validate.

@bivasda1
Copy link
Contributor Author

@nickboldt I will remove s390x form Platform files and will rebase as quickly as possible.

@bivasda1
Copy link
Contributor Author

@nickboldt @ericwill I have rebased.

context: .
file: ./build/dockerfiles/Dockerfile
platforms: ${{ steps.prep.outputs.platforms }}
tags: quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ steps.prep.outputs.version }},quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ steps.prep.outputs.short_sha1 }},quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ github.event.inputs.version }}
Copy link
Contributor

@nickboldt nickboldt Feb 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're pushing the same image w/ 3 tags here:

  •       quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ steps.prep.outputs.version }},
    
  •       quay.io/eclipse/${{ steps.prep.outputs.image }}:${{  github.event.inputs.version }}
    
  •       quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ steps.prep.outputs.short_sha1 }},
    

should it be only two pushes for prep.outputs.version and short_sha1? Not sure why we would want the submitted version 7.y.z AND version from VERSION file AND the short sha from the tip of the checked out branch.

@mkuznyetsov WDYT?

Also there's an extra space in ${{ github.event.inputs.version }} which might cause error?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nickboldt

should it be only two pushes for prep.outputs.version and short_sha1?
Yes

prep.outputs.version-->7.27.0-SNAPSHOT
event.inputs.version-->7.27.0 

I was trying to release proper version, so that event.inputs.version I have added. Please let me know should I remove event.inputs.version?

I will remove spaces from ${{ github.event.inputs.version }}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't release -SNAPSHOT versions to quay.

https://quay.io/repository/eclipse/che-devfile-registry?tab=tags

Just nightly + sha tag and release (7.yy.z) + sha tag.

context: .
file: ./build/dockerfiles/Dockerfile
platforms: ${{ steps.prep.outputs.platforms }}
tags: quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ steps.prep.outputs.version }},quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ steps.prep.outputs.short_sha1 }}
Copy link
Contributor Author

@bivasda1 bivasda1 Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ericwill @nickboldt I have removed quay.io/eclipse/${{ steps.prep.outputs.image }}:${{ github.event.inputs.version }} and rebased

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

outputs.version-->7.27.0-SNAPSHOT

That's the wrong version to release. We would want 7.27.0, not the -SNAPSHOT version. Releases are for releases, nightly is for snapshots. :D

run: |
SHORT_SHA1=$(git rev-parse --short HEAD)
echo ::set-output name=short_sha1::${SHORT_SHA1}
VERSION=$(head -n 1 VERSION)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this value != inputs.version we should use inputs.version.

In fact I'm not sure why we need BOTH a computed value (7.27.0-SNAPSHOT) and an input value (7.27.0).

I would expect that if the input is missing, we compute it... but I believe part of the release process is to SET the value in the VERSION file, so there's no point reading the file before we set the value. :)

Instead I'd just take the value from the input as it should be the same as the VERSION file, once the release is complete.

Correct @mkuznyetsov ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@mkuznyetsov mkuznyetsov Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nickboldt as of now, make-release.sh is called first from another workflow, which makes changes to VERSION file, and then this workflow does the rest. Here it reads the VERSION file, in case it was triggered by the push to the release branch, which has no inputs

Because it was slightly confusing, I made a PR (about which I've forgot 😃 )
https://github.com/eclipse/che-devfile-registry/pull/338/files
it makes make-release.sh to be called from this workflow

Copy link
Contributor

@ericwill ericwill Mar 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So which approach are we taking? Right now as it stands, the 7.27.0-SNAPSHOT problem won't happen because the workflow is triggered after the file is changed. But we can have this PR wait until https://github.com/eclipse/che-devfile-registry/pull/338/files is merged and then adjust the approach. What do you guys think? cc @nickboldt @mkuznyetsov

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR 338 is ready to merge, after fixing all the shellcheck complaints. https://github.com/eclipse/che-devfile-registry/pull/338/files

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've merged it, @bivasda1 please adjust the PR to make sure -SNAPSHOT isn't pushed as part of the tag

@nickboldt
Copy link
Contributor

nickboldt commented Feb 23, 2021

do you expect ci-multiarch workflows should be formatted

Since I don't own this project, I'll let the TL comment... but my $0.02 is that it seems like from a project management perspective it might be better/more efficient to have:

  • 1 GH action that can handle all three (four) arches
    rather than
  • 1 GH action action for x64 and ppc64 (and aarch64), while a travis script separately handles s390x.

required: true
default: '7.y.z'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest having an empty default value, so we wouldn't have accidental workflow runs with 7.y.z taken as a default parameter.

@bivasda1
Copy link
Contributor Author

bivasda1 commented Feb 24, 2021

do you expect ci-multiarch workflows should be formatted https://github.com/eclipse/che-dashboard/blob/master/.github/workflows/ci-multiarch.yaml#L91-L115 referenced to eclipse-che/che-dashboard#119

Since I don't own this project, I'll let the TL comment... but my $0.02 is that it seems like from a project management perspective it might be better/more efficient to have:

  • 1 GH action that can handle all three (four) arches
    rather than
  • 1 GH action action for x64 and ppc64 (and aarch64), while a travis script separately handles s390x.

@ericwill Could you please share your thoughts @nickboldt suggestions

@bivasda1
Copy link
Contributor Author

bivasda1 commented Feb 24, 2021

@nickboldt Could you please review this PR,I think Versioning issue is resolved now.

@bivasda1 bivasda1 force-pushed the master branch 2 times, most recently from 0548984 to 953c109 Compare February 26, 2021 18:20
@bivasda1
Copy link
Contributor Author

Please remove this file: .github/workflows/.nightly-build-publish.yml.swp

Could u please review the PR I have deleted .github/workflows/.nightly-build-publish.yml.swp

@nickboldt
Copy link
Contributor

please rebase, you're out of date again :(

Signed-off-by: Bivas Das <bivasda1@in.ibm.com>
@bivasda1
Copy link
Contributor Author

bivasda1 commented Mar 3, 2021

@nickboldt Rebased and Added buildx support make-release.sh

Copy link
Contributor

@ericwill ericwill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's give it a shot

@nickboldt
Copy link
Contributor

seems legit for x and power arches but this won't work for s390x and will need to be refactored in future.

+1 for pushing this for now, with the caveat that we'll need something like eclipse-che/che-dashboard#177 to support pushing each per-arch to quay independently, then combining them into a single multi-arch digest/manifest.

@nickboldt nickboldt self-requested a review March 8, 2021 14:25
Copy link
Contributor

@nickboldt nickboldt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, but won't work for s390x.

@ericwill ericwill merged commit c6567ce into eclipse-che:master Mar 8, 2021
@che-bot che-bot added this to the 7.28 milestone Mar 8, 2021
@ArvinB
Copy link
Contributor

ArvinB commented Mar 8, 2021

Just commenting...here is a bash script very similar to what I do for building multi-arch images. As @nickboldt mentioned from eclipse-che/che-dashboard#177 that there is a risk of building for s390x. My issue primarily is that the CentOS repo's do not contain the packages for s390x. I don't have enough knowledge yet on which repo(s) to include, so for now I disable them all so that basically the yum and dnf updates all just pass on through.

registry=quay.io
repo=some/path
version=1.0.0
platforms=("amd64" "s390x")

tags=""
for platform in ${platforms[*]}
do
  echo "Building for linux/$platform"
  tag=$registry/$repo:${version}_$platform
  docker buildx build --load --platform linux/$platform -t $tag .
  docker push $tag
  tags="${tags} ${tag}"
done

docker manifest create $registry/$repo:$version $tags
docker manifest push $registry/$repo:$version
docker buildx imagetools inspect $registry/$repo:$version

FYI
@sleshchenko There is great interest in building multi-arch.

@ericwill
Copy link
Contributor

ericwill commented Mar 8, 2021

@bivasda1 so the build was successful, but please fix/update the following in a new PR:

  • the 7.28.0-SNAPSHOT tag shouldn't be there
  • a nightly tag hasn't been published/updated -- likely this is what the 7.28.0-SNAPSHOT tag should be

https://quay.io/repository/eclipse/che-devfile-registry?tab=tags

@nickboldt
Copy link
Contributor

There, I fixed it. :) https://quay.io/repository/eclipse/che-devfile-registry?tab=tags&tag=nightly

Also deleted the 7.28.0-SNAPSHOT tag

@nickboldt
Copy link
Contributor

But also... the image is not multiarch :(

#23 [linux/arm64 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
#23 sha256:3075bc83c5d59c32a9b1a25f5afd028722d2f71f259502bc813ddbcec8d90bf5
#23 0.092 /bin/sh: /tmp/install-editor-tooling.sh: not found
#23 ERROR: executor failed running [/bin/sh -c /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh]: exit code: 127

#12 [linux/amd64 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
#12 sha256:feb2a9698386ef81638fdeab39c0d81e0bbdad912bc578d7027b6a30d8144c40
#12 0.229 container process is already dead
#12 CANCELED

#17 [linux/ppc64le 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
#17 sha256:6d32b5832d93ea6889e549b5179b2c324f77d75fb4ce4be71c5176c98682b25b
#17 0.235 container process is already dead
#17 CANCELED
------
 > [linux/arm64 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh:
------
Dockerfile:10
--------------------
   8 |     # Install common terminal editors in container to aid development process
   9 |     COPY install-editor-tooling.sh /tmp
  10 | >>> RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
  11 |     
  12 |     USER 10001
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh]: exit code: 127
Pushing quay.io/eclipse/che-antora-2.3:nightly to remote registry
An image does not exist locally with the tag: quay.io/eclipse/che-antora-2.3
The push refers to repository [quay.io/eclipse/che-antora-2.3]
Building quay.io/eclipse/che-cpp-rhel7:nightly based on registry.access.redhat.com/devtools/llvm-toolset-rhel7 ...
time="2021-03-08T15:15:42Z" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"

-- https://github.com/eclipse/che-devfile-registry/runs/2057571221?check_suite_focus=true

and if I skopeo inspect I only get amd64:

$➔ skopeo --override-os ppc64le inspect docker://quay.io/eclipse/che-devfile-registry:nightly | jq -r .Architecture
amd64

Contrast with:

$➔ skopeo inspect docker://quay.io/eclipse/che-dashboard:multiarch-next --raw | jq -r '.manifests[].platform.architecture'
amd64
arm64
ppc64le
s390x

@ArvinB
Copy link
Contributor

ArvinB commented Mar 8, 2021

@nickboldt I am curious how this will get resolved, because this is similar to the issue I faced when trying to build registry images for s390x.

@nickboldt
Copy link
Contributor

with:
context: .
file: ./build/dockerfiles/Dockerfile
platforms: ${{ steps.package.outputs.content }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nickboldt Image is still single arch because I guess slight change is required here. Instead of platforms: ${{ steps.package.outputs.content }}, it needs to be platforms: ${{ steps.prep.outputs.platforms }} as defined on line 37.

@bivasda1
Copy link
Contributor Author

bivasda1 commented Mar 9, 2021

But also... the image is not multiarch :(

#23 [linux/arm64 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
#23 sha256:3075bc83c5d59c32a9b1a25f5afd028722d2f71f259502bc813ddbcec8d90bf5
#23 0.092 /bin/sh: /tmp/install-editor-tooling.sh: not found
#23 ERROR: executor failed running [/bin/sh -c /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh]: exit code: 127

#12 [linux/amd64 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
#12 sha256:feb2a9698386ef81638fdeab39c0d81e0bbdad912bc578d7027b6a30d8144c40
#12 0.229 container process is already dead
#12 CANCELED

#17 [linux/ppc64le 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
#17 sha256:6d32b5832d93ea6889e549b5179b2c324f77d75fb4ce4be71c5176c98682b25b
#17 0.235 container process is already dead
#17 CANCELED
------
 > [linux/arm64 5/6] RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh:
------
Dockerfile:10
--------------------
   8 |     # Install common terminal editors in container to aid development process
   9 |     COPY install-editor-tooling.sh /tmp
  10 | >>> RUN /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh
  11 |     
  12 |     USER 10001
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c /tmp/install-editor-tooling.sh && rm -f /tmp/install-editor-tooling.sh]: exit code: 127
Pushing quay.io/eclipse/che-antora-2.3:nightly to remote registry
An image does not exist locally with the tag: quay.io/eclipse/che-antora-2.3
The push refers to repository [quay.io/eclipse/che-antora-2.3]
Building quay.io/eclipse/che-cpp-rhel7:nightly based on registry.access.redhat.com/devtools/llvm-toolset-rhel7 ...
time="2021-03-08T15:15:42Z" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"

-- https://github.com/eclipse/che-devfile-registry/runs/2057571221?check_suite_focus=true

and if I skopeo inspect I only get amd64:

$➔ skopeo --override-os ppc64le inspect docker://quay.io/eclipse/che-devfile-registry:nightly | jq -r .Architecture
amd64

Contrast with:

$➔ skopeo inspect docker://quay.io/eclipse/che-dashboard:multiarch-next --raw | jq -r '.manifests[].platform.architecture'
amd64
arm64
ppc64le
s390x

@nickboldt @ericwill
there are these images which are not multiarch. So do we need to excluded from build for releasing multiarch?

che-antora-2.3          docker.io/antora/antora:2.3.3
che-dotnet-2.2          mcr.microsoft.com/dotnet/core/sdk:2.2-stretch
che-dotnet-3.1          mcr.microsoft.com/dotnet/core/sdk:3.1.301-buster
che-nodejs8-centos      registry.centos.org/che-stacks/centos-nodejs
che-php-7               quay.io/eclipse/che-php-base:7.4
che-python-3.6          centos/python-36-centos7:1

che-cpp-rhel7           registry.access.redhat.com/devtools/llvm-toolset-rhel7->not compatible for arm64
che-rust-1.39           rust:1.39.0-slim->not compatible for ppc64le

Even we can enable multiarch for these repos.

che-php-7               quay.io/eclipse/che-php-base:7.4->registry.access.redhat.com/ubi8/php-72
che-python-3.6          centos/python-36-centos7:1->registry.access.redhat.com/ubi8/python-36

@ericwill
Copy link
Contributor

ericwill commented Mar 9, 2021

@nickboldt @ericwill
there are these images which are not multiarch. So do we need to excluded from build for releasing multiarch?

che-antora-2.3          docker.io/antora/antora:2.3.3
che-dotnet-2.2          mcr.microsoft.com/dotnet/core/sdk:2.2-stretch
che-dotnet-3.1          mcr.microsoft.com/dotnet/core/sdk:3.1.301-buster
che-nodejs8-centos      registry.centos.org/che-stacks/centos-nodejs
che-php-7               quay.io/eclipse/che-php-base:7.4
che-python-3.6          centos/python-36-centos7:1

che-cpp-rhel7           registry.access.redhat.com/devtools/llvm-toolset-rhel7->not compatible for arm64
che-rust-1.39           rust:1.39.0-slim->not compatible for ppc64le

Even we can enable multiarch for these repos.

che-php-7               quay.io/eclipse/che-php-base:7.4->registry.access.redhat.com/ubi8/php-72
che-python-3.6          centos/python-36-centos7:1->registry.access.redhat.com/ubi8/python-36

Yes we need to exclude stacks that aren't available on multi-arch. For example if rust is x86 only, then it should be excluded from builds on other arches.

@nickboldt
Copy link
Contributor

nickboldt commented Mar 9, 2021

ericwill added a commit that referenced this pull request Apr 1, 2021
Partial revert of #336

Related to eclipse-che/che#19486

Signed-off-by: Eric Williams <ericwill@redhat.com>
nickboldt pushed a commit that referenced this pull request Apr 1, 2021
Partial revert of #336

Related to eclipse-che/che#19486

Signed-off-by: Eric Williams <ericwill@redhat.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants