Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build hybrid image #8476

Merged
merged 7 commits into from
Dec 10, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/e2e-test-kind.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ jobs:
- name: Build Velero Image
if: steps.image-cache.outputs.cache-hit != 'true'
run: |
IMAGE=velero VERSION=pr-test make container
docker save velero:pr-test -o ./velero.tar
IMAGE=velero VERSION=pr-test BUILD_OUTPUT_TYPE=docker make container
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At present, CI E2E supports Linux only, opened a separate issue #8477 for requirements of running it in Windows.

docker save velero:pr-test-linux-amd64 -o ./velero.tar
# Create json of k8s versions to test
# from guide: https://stackoverflow.com/a/65094398/4590470
setup-test-matrix:
Expand Down Expand Up @@ -130,7 +130,7 @@ jobs:
ADDITIONAL_BSL_CONFIG=region=minio,s3ForcePathStyle="true",s3Url=http://$(hostname -i):9000 \
ADDITIONAL_CREDS_FILE=/tmp/credential \
ADDITIONAL_BSL_BUCKET=additional-bucket \
VELERO_IMAGE=velero:pr-test \
VELERO_IMAGE=velero:pr-test-linux-amd64 \
GINKGO_LABELS="${{ matrix.labels }}" \
make -C test/ run-e2e
timeout-minutes: 30
Expand Down
55 changes: 55 additions & 0 deletions Dockerfile-Windows
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Copyright the Velero contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

ARG OS_VERSION=1809

# Velero binary build section
FROM --platform=$BUILDPLATFORM golang:1.22-bookworm AS velero-builder

ARG GOPROXY
ARG BIN
ARG PKG
ARG VERSION
ARG REGISTRY
ARG GIT_SHA
ARG GIT_TREE_STATE
ARG TARGETOS
ARG TARGETARCH
ARG TARGETVARIANT

ENV CGO_ENABLED=0 \
GO111MODULE=on \
GOPROXY=${GOPROXY} \
GOOS=${TARGETOS} \
GOARCH=${TARGETARCH} \
GOARM=${TARGETVARIANT} \
LDFLAGS="-X ${PKG}/pkg/buildinfo.Version=${VERSION} -X ${PKG}/pkg/buildinfo.GitSHA=${GIT_SHA} -X ${PKG}/pkg/buildinfo.GitTreeState=${GIT_TREE_STATE} -X ${PKG}/pkg/buildinfo.ImageRegistry=${REGISTRY}"

WORKDIR /go/src/github.com/vmware-tanzu/velero

COPY . /go/src/github.com/vmware-tanzu/velero

RUN mkdir -p /output/usr/bin && \
export GOARM=$( echo "${GOARM}" | cut -c2-) && \
go build -o /output/${BIN}.exe \
-ldflags "${LDFLAGS}" ${PKG}/cmd/${BIN} && \
go build -o /output/velero-helper.exe \
-ldflags "${LDFLAGS}" ${PKG}/cmd/velero-helper && \
go clean -modcache -cache

# Velero image packing section
FROM mcr.microsoft.com/windows/nanoserver:${OS_VERSION}
COPY --from=velero-builder /output /

USER ContainerUser
112 changes: 95 additions & 17 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ GCR_IMAGE ?= $(GCR_REGISTRY)/$(BIN)
# We allow the Dockerfile to be configurable to enable the use of custom Dockerfiles
# that pull base images from different registries.
VELERO_DOCKERFILE ?= Dockerfile
VELERO_DOCKERFILE_WINDOWS ?= Dockerfile-Windows
BUILDER_IMAGE_DOCKERFILE ?= hack/build-image/Dockerfile

# Calculate the realpath of the build-image Dockerfile as we `cd` into the hack/build
Expand Down Expand Up @@ -100,20 +101,31 @@ comma=,
RESTIC_VERSION ?= 0.15.0

CLI_PLATFORMS ?= linux-amd64 linux-arm linux-arm64 darwin-amd64 darwin-arm64 windows-amd64 linux-ppc64le
BUILDX_PLATFORMS ?= $(subst -,/,$(ARCH))
# Whether or not buildx should push the image to the registry, applies to when BUILDX_PLATFORMS has multiple platforms below.
# false by default because most people do not have credentials to $(REGISTRY) nor should they push from their local development machine.
# once you have set $(REGISTRY) or $(IMAGE) and have credentials to those registries, you can set BUILDX_PUSH=true
BUILDX_PUSH ?= false
# if BUILDX_PLATFORMS has multiple platforms, we need to use BUILDX_OUTPUT_TYPE=image and optionally BUILDX_OUTPUT_TYPE=image,push=true if BUILDX_PUSH is true
# The default image store in Docker Engine doesn't support loading multi-platform images. So set BUILDX_PUSH=true to push the image to the registry if you need to use the
# multi-platform output image. In the future, we may add containerd support for multi-platform images. https://docs.docker.com/engine/storage/containerd/
# if BUILDX_PLATFORMS has only one platform, we can use BUILDX_OUTPUT_TYPE=docker to import the image to the local docker daemon
ifeq ($(words $(subst $(comma), ,$(BUILDX_PLATFORMS))),1)
BUILDX_OUTPUT_TYPE ?= docker
BUILD_OUTPUT_TYPE ?= docker
BUILD_OS ?= linux
BUILD_ARCH ?= amd64
BUILD_TAG_GCR ?= false
BUILD_WINDOWS_VERSION ?= ltsc2022

ifeq ($(BUILD_OUTPUT_TYPE), docker)
ALL_OS = linux
ALL_ARCH.linux = $(word 2, $(subst -, ,$(shell go env GOOS)-$(shell go env GOARCH)))
else
BUILDX_OUTPUT_TYPE ?= image$(subst false,,$(subst true,$(comma)push=true,$(BUILDX_PUSH)))
ALL_OS = $(subst $(comma), ,$(BUILD_OS))
ALL_ARCH.linux = $(subst $(comma), ,$(BUILD_ARCH))
endif

ALL_ARCH.windows = $(if $(filter windows,$(ALL_OS)),amd64,)
ALL_OSVERSIONS.windows = $(if $(filter windows,$(ALL_OS)),$(BUILD_WINDOWS_VERSION),)
ALL_OS_ARCH.linux = $(foreach os, $(filter linux,$(ALL_OS)), $(foreach arch, ${ALL_ARCH.linux}, ${os}-$(arch)))
ALL_OS_ARCH.windows = $(foreach os, $(filter windows,$(ALL_OS)), $(foreach arch, $(ALL_ARCH.windows), $(foreach osversion, ${ALL_OSVERSIONS.windows}, ${os}-${osversion}-${arch})))
ALL_OS_ARCH = $(ALL_OS_ARCH.linux)$(ALL_OS_ARCH.windows)

ALL_IMAGE_TAGS = $(IMAGE_TAGS)
ifeq ($(BUILD_TAG_GCR), true)
ALL_IMAGE_TAGS += $(GCR_IMAGE_TAGS)
endif

# set git sha and tree state
GIT_SHA = $(shell git rev-parse HEAD)
ifneq ($(shell git status --porcelain 2> /dev/null),)
Expand Down Expand Up @@ -208,11 +220,31 @@ container:
ifneq ($(BUILDX_ENABLED), true)
$(error $(BUILDX_ERROR))
endif
-docker buildx rm velero-builder || true
@docker buildx create --use --name=velero-builder
Comment on lines +226 to +227
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This overrides my current buildx builder settings. ie. I already have a kubernetes docker builder I intended to use on a more powerful system.

Can we do it in currently available builders that our docker is already set to use?

I think this can be doc as a pre-step. It is also not compatible with podman for example.

~/git/velero build-hybrid-image 1m 44s
❯ podman buildx ls
Error: unrecognized command `podman buildx ls`
Try 'podman buildx --help' for more information

~/git/velero build-hybrid-image
❯ podman buildx rm
Error: unrecognized command `podman buildx rm`
Try 'podman buildx --help' for more information

~/git/velero build-hybrid-image
❯ podman buildx create
Error: unrecognized command `podman buildx create`
Try 'podman buildx --help' for more information

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BUILD_OUTPUT_TYPE=registry REGISTRY=xxx BUILD_OS=windows make container

other than that this did build.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have modified the code to accept specified buildx instance by BUILDX_INSTANCE input parameter.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the podman part, as far as my understanding, there is no conclusions/actions to support podman emulated docker purposefully. Looks like the current Makefile works, it may be a coincidence.
Anyway, with BUILDX_INSTANCE input parameter set, we don't add any new docker buildx call, hopefully this could ensure podman still work as is.
However, supporting podman is not something must-have with/without this PR. If podman is required after our evaluation, there should be separate issue and PR for the necessary changes based on this PR.

Copy link
Contributor Author

@Lyndon-Li Lyndon-Li Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the doc part, there is a separate issue for build-from-source doc changes related to Windows support #8475, so there will be a separate PR for the same which will include the changes mentioned here.


@mkdir -p _output

@for osarch in $(ALL_OS_ARCH); do \
$(MAKE) container-$${osarch}; \
done

ifeq ($(BUILD_OUTPUT_TYPE), registry)
@for tag in $(ALL_IMAGE_TAGS); do \
IMAGE_TAG=$${tag} $(MAKE) push-manifest; \
done
endif

container-linux-%:
@BUILDX_ARCH=$* $(MAKE) container-linux

container-linux:
@echo "building container: $(IMAGE):$(VERSION)-linux-$(BUILDX_ARCH)"

@docker buildx build --pull \
--output=type=$(BUILDX_OUTPUT_TYPE) \
--platform $(BUILDX_PLATFORMS) \
$(addprefix -t , $(IMAGE_TAGS)) \
$(addprefix -t , $(GCR_IMAGE_TAGS)) \
--output="type=$(BUILD_OUTPUT_TYPE)$(if $(findstring tar, $(BUILD_OUTPUT_TYPE)),$(comma)dest=_output/$(BIN)-$(VERSION)-linux-$(BUILDX_ARCH).tar,)" \
--platform="linux/$(BUILDX_ARCH)" \
$(addprefix -t , $(addsuffix "-linux-$(BUILDX_ARCH)",$(ALL_IMAGE_TAGS))) \
--build-arg=GOPROXY=$(GOPROXY) \
--build-arg=PKG=$(PKG) \
--build-arg=BIN=$(BIN) \
Expand All @@ -221,8 +253,54 @@ endif
--build-arg=GIT_TREE_STATE=$(GIT_TREE_STATE) \
--build-arg=REGISTRY=$(REGISTRY) \
--build-arg=RESTIC_VERSION=$(RESTIC_VERSION) \
--provenance=false \
--sbom=false \
-f $(VELERO_DOCKERFILE) .
@echo "container: $(IMAGE):$(VERSION)"

@echo "built container: $(IMAGE):$(VERSION)-linux-$(BUILDX_ARCH)"

container-windows-%:
@BUILDX_OSVERSION=$(firstword $(subst -, ,$*)) BUILDX_ARCH=$(lastword $(subst -, ,$*)) $(MAKE) container-windows

container-windows:
@echo "building container: $(IMAGE):$(VERSION)-windows-$(BUILDX_OSVERSION)-$(BUILDX_ARCH)"

@docker buildx build --pull \
--output="type=$(BUILD_OUTPUT_TYPE)$(if $(findstring tar, $(BUILD_OUTPUT_TYPE)),$(comma)dest=_output/$(BIN)-$(VERSION)-windows-$(BUILDX_OSVERSION)-$(BUILDX_ARCH).tar,)" \
--platform="windows/$(BUILDX_ARCH)" \
$(addprefix -t , $(addsuffix "-windows-$(BUILDX_OSVERSION)-$(BUILDX_ARCH)",$(ALL_IMAGE_TAGS))) \
--build-arg=GOPROXY=$(GOPROXY) \
--build-arg=PKG=$(PKG) \
--build-arg=BIN=$(BIN) \
--build-arg=VERSION=$(VERSION) \
--build-arg=OS_VERSION=$(BUILDX_OSVERSION) \
--build-arg=GIT_SHA=$(GIT_SHA) \
--build-arg=GIT_TREE_STATE=$(GIT_TREE_STATE) \
--build-arg=REGISTRY=$(REGISTRY) \
--provenance=false \
--sbom=false \
-f $(VELERO_DOCKERFILE_WINDOWS) .

@echo "built container: $(IMAGE):$(VERSION)-windows-$(BUILDX_OSVERSION)-$(BUILDX_ARCH)"

push-manifest:
@echo "building manifest: $(IMAGE_TAG) for $(foreach osarch, $(ALL_OS_ARCH), $(IMAGE_TAG)-${osarch})"
@docker manifest create --amend $(IMAGE_TAG) $(foreach osarch, $(ALL_OS_ARCH), $(IMAGE_TAG)-${osarch})

@set -x; \
for arch in $(ALL_ARCH.windows); do \
for osversion in $(ALL_OSVERSIONS.windows); do \
BASEIMAGE=mcr.microsoft.com/windows/nanoserver:$${osversion}; \
full_version=`docker manifest inspect $${BASEIMAGE} | jq -r '.manifests[0].platform["os.version"]'`; \
docker manifest annotate --os windows --arch $${arch} --os-version $${full_version} $(IMAGE_TAG) $(IMAGE_TAG)-windows-$${osversion}-$${arch}; \
done; \
done

@echo "pushing manifest $(IMAGE_TAG)"
@docker manifest push --purge $(IMAGE_TAG)

@echo "pushed manifest $(IMAGE_TAG):"
@docker manifest inspect $(IMAGE_TAG)

SKIP_TESTS ?=
test: build-dirs
Expand Down
1 change: 1 addition & 0 deletions changelogs/unreleased/8476-Lyndon-Li
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix issue #8415, implement multi-arch build and Windows build
8 changes: 4 additions & 4 deletions design/multiple-arch-build-with-windows.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ The built image could be listed by `docker image ls`.

**Local build for linux-amd64 and windows-amd64 to tar**
```
BUILDX_OUTPUT_TYPE=tar BUILD_OS=linux,windows make container
BUILD_OUTPUT_TYPE=tar BUILD_OS=linux,windows make container
```
Under `_output` directory, below files are generated:
```
Expand All @@ -86,7 +86,7 @@ velero-main-windows-ltsc2022-amd64.tar

**Local build for linux-amd64, linux-arm64 and windows-amd64 to tar**
```
BUILDX_OUTPUT_TYPE=tar BUILD_OS=linux,windows BUILD_ARCH=amd64,arm64 make container
BUILD_OUTPUT_TYPE=tar BUILD_OS=linux,windows BUILD_ARCH=amd64,arm64 make container
```
Under `_output` directory, below files are generated:
```
Expand All @@ -98,7 +98,7 @@ velero-main-windows-ltsc2022-amd64.tar
**Push build for linux-amd64 and windows-amd64**
Prerequisite: login to registry, e.g., through `docker login`
```
BUILDX_OUTPUT_TYPE=registry REGISTRY=<registry> BUILD_OS=linux,windows make container
BUILD_OUTPUT_TYPE=registry REGISTRY=<registry> BUILD_OS=linux,windows make container
```
Nothing is available locally, in the registry 3 tags are available:
```
Expand All @@ -110,7 +110,7 @@ velero/velero:main-linux-amd64
**Push build for linux-amd64, linux-arm64 and windows-amd64**
Prerequisite: login to registry, e.g., through `docker login`
```
BUILDX_OUTPUT_TYPE=registry REGISTRY=<registry> BUILD_OS=linux,windows BUILD_ARCH=amd64,arm64 make container
BUILD_OUTPUT_TYPE=registry REGISTRY=<registry> BUILD_OS=linux,windows BUILD_ARCH=amd64,arm64 make container
```
Nothing is available locally, in the registry 4 tags are available:
```
Expand Down
20 changes: 13 additions & 7 deletions hack/docker-push.sh
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ fi
if [[ -z $BRANCH && -z $TAG ]]; then
echo "Test Velero container build without pushing, when Dockerfile is changed by PR."
BRANCH="${GITHUB_BASE_REF}-container"
OUTPUT_TYPE="local,dest=."
OUTPUT_TYPE="tar"
else
OUTPUT_TYPE="registry"
fi
Expand All @@ -88,8 +88,12 @@ else
fi
fi

if [[ -z "$BUILDX_PLATFORMS" ]]; then
BUILDX_PLATFORMS="linux/amd64,linux/arm64"
if [[ -z "$BUILD_OS" ]]; then
BUILD_OS="linux,windows"
fi

if [[ -z "$BUILD_ARCH" ]]; then
BUILD_ARCH="amd64,arm64"
fi

# Debugging info
Expand All @@ -98,13 +102,15 @@ echo "BRANCH: $BRANCH"
echo "TAG: $TAG"
echo "TAG_LATEST: $TAG_LATEST"
echo "VERSION: $VERSION"
echo "BUILDX_PLATFORMS: $BUILDX_PLATFORMS"
echo "BUILD_OS: $BUILD_OS"
echo "BUILD_ARCH: $BUILD_ARCH"

echo "Building and pushing container images."


VERSION="$VERSION" \
TAG_LATEST="$TAG_LATEST" \
BUILDX_PLATFORMS="$BUILDX_PLATFORMS" \
BUILDX_OUTPUT_TYPE=$OUTPUT_TYPE \
make all-containers
BUILD_OS="$BUILD_OS" \
BUILD_ARCH="$BUILD_ARCH" \
BUILD_OUTPUT_TYPE=$OUTPUT_TYPE \
make all-containers
Loading