Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-3.0: update release-tools #685

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions release-tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ images. Building from master creates the main `canary` image.
Sharing and updating
--------------------

[`git subtree`](https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt)
[`git subtree`](https://github.com/git/git/blob/HEAD/contrib/subtree/git-subtree.txt)
is the recommended way of maintaining a copy of the rules inside the
`release-tools` directory of a project. This way, it is possible to make
changes also locally, test them and then push them back to the shared
Expand Down Expand Up @@ -89,7 +89,7 @@ main

All Kubernetes-CSI repos are expected to switch to Prow. For details
on what is enabled in Prow, see
https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi
https://github.com/kubernetes/test-infra/tree/HEAD/config/jobs/kubernetes-csi

Test results for periodic jobs are visible in
https://testgrid.k8s.io/sig-storage-csi-ci
Expand Down
2 changes: 1 addition & 1 deletion release-tools/SECURITY_CONTACTS
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# to for triaging and handling of incoming issues.
#
# The below names agree to abide by the
# [Embargo Policy](https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#embargo-policy)
# [Embargo Policy](https://github.com/kubernetes/sig-release/blob/HEAD/security-release-process-documentation/security-release-process.md#embargo-policy)
# and will be removed and replaced if they violate that agreement.
#
# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE
Expand Down
27 changes: 11 additions & 16 deletions release-tools/SIDECAR_RELEASE_PROCESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,8 @@ The release manager must:
* Be a member of the kubernetes-csi organization. Open an
[issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
kubernetes/org to request membership
* Be a top level approver for the repository. To become a top level approver,
the candidate must demonstrate ownership and deep knowledge of the repository
through active maintenance, responding to and fixing issues, reviewing PRs,
test triage.
* Be part of the maintainers or admin group for the repository. admin is a
superset of maintainers, only maintainers level is required for cutting a
release. Membership can be requested by submitting a PR to kubernetes/org.
* Be part of the maintainers group for the repository.
Membership can be requested by submitting a PR to kubernetes/org.
[Example](https://github.com/kubernetes/org/pull/1467)

## Updating CI Jobs
Expand All @@ -31,16 +26,16 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/HEAD/release-tools/prow.sh)
along with any overrides in
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/HEAD/.prow.sh)
to mirror the failing environment. Once e2e tests are passing (verify-unit tests
will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
1. Changes can then be updated in all the sidecar repos and hostpath driver repo
by following the [update
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/HEAD/README.md#sharing-and-updating).
1. New pull and CI jobs are configured by adding new K8s versions to the top of
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/HEAD/config/jobs/kubernetes-csi/gen-jobs.sh).
New pull jobs that have been unverified should be initially made optional by
setting the new K8s version as
[experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
Expand All @@ -52,7 +47,7 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Identify all issues and ongoing PRs that should go into the release, and
drive them to resolution.
1. Download v2.8+ [K8s release notes
generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
generator](https://github.com/kubernetes/release/tree/HEAD/cmd/release-notes)
1. Generate release notes for the release. Replace arguments with the relevant
information.
* Clean up old cached information (also needed if you are generating release
Expand Down Expand Up @@ -95,15 +90,15 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
the [k8s image
repo](https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io/images/k8s-staging-sig-storage),
repo](https://github.com/kubernetes/k8s.io/tree/HEAD/k8s.gcr.io/images/k8s-staging-sig-storage),
run `./generate.sh > images.yaml`, and send a PR with the updated images.
Once merged, the image promoter will copy the images from staging to prod.
1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
and feature pages with the new released version.
1. After all the sidecars have been released, update
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/HEAD/deploy)
and [k/k
in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
in-tree](https://github.com/kubernetes/kubernetes/tree/HEAD/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)

## Adding support for a new Kubernetes release

Expand Down Expand Up @@ -134,7 +129,7 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Once all sidecars for the new Kubernetes release are released,
either bump the version number of the images in the existing
[csi-driver-host-path
deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/HEAD/deploy)
and/or create a new deployment, depending on what Kubernetes
release an updated sidecar is compatible with. If no new deployment
is needed, then add a symlink to document that there intentionally
Expand Down
14 changes: 9 additions & 5 deletions release-tools/build.make
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ SHELL := /bin/bash
# set in main Makefile of a repository.
# CMDS=

# Normally, commands are expected in "cmd". That can be changed for a
# repository to something else by setting CMDS_DIR before including build.make.
CMDS_DIR ?= cmd

# This is the default. It can be overridden in the main Makefile after
# including build.make.
REGISTRY_NAME?=quay.io/k8scsi
Expand Down Expand Up @@ -90,15 +94,15 @@ $(CMDS:%=build-%): build-%: check-go-version-go
if ! [ $${#os_arch_seen_pre} = $${#os_arch_seen} ]; then \
continue; \
fi; \
if ! (set -x; CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '$(FULL_LDFLAGS)' -o "./bin/$*$$suffix" ./cmd/$*); then \
if ! (set -x; cd ./$(CMDS_DIR)/$* && CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '$(FULL_LDFLAGS)' -o "$(abspath ./bin)/$*$$suffix" .); then \
echo "Building $* for GOOS=$$os GOARCH=$$arch failed, see error(s) above."; \
exit 1; \
fi; \
os_arch_seen+=";$$os-$$arch"; \
done

$(CMDS:%=container-%): container-%: build-%
docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
docker build -t $*:latest -f $(shell if [ -e ./$(CMDS_DIR)/$*/Dockerfile ]; then echo ./$(CMDS_DIR)/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .

$(CMDS:%=push-%): push-%: container-%
set -ex; \
Expand Down Expand Up @@ -133,7 +137,7 @@ DOCKER_BUILDX_CREATE_ARGS ?=
# This target builds a multiarch image for one command using Moby BuildKit builder toolkit.
# Docker Buildx is included in Docker 19.03.
#
# ./cmd/<command>/Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows].
# ./$(CMDS_DIR)/<command>/Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows].
# It is currently optional: if no such file exists, Windows images are not included,
# even when Windows is listed in BUILD_PLATFORMS. That way, projects can test that
# Windows binaries can be built before adding a Dockerfile for it.
Expand All @@ -146,8 +150,8 @@ $(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
export DOCKER_CLI_EXPERIMENTAL=enabled; \
docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest; \
trap "docker buildx rm multiarchimage-buildertest" EXIT; \
dockerfile_linux=$$(if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi); \
dockerfile_windows=$$(if [ -e ./cmd/$*/Dockerfile.Windows ]; then echo ./cmd/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
dockerfile_linux=$$(if [ -e ./$(CMDS_DIR)/$*/Dockerfile ]; then echo ./$(CMDS_DIR)/$*/Dockerfile; else echo Dockerfile; fi); \
dockerfile_windows=$$(if [ -e ./$(CMDS_DIR)/$*/Dockerfile.Windows ]; then echo ./$(CMDS_DIR)/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \
if ! [ -f "$$dockerfile_windows" ]; then \
build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe *[^ ]* *[^ ]*//g' -e 's/; *;/;/g' -e 's/;[ ]*$$//')"; \
Expand Down
6 changes: 3 additions & 3 deletions release-tools/cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@
# because binaries will get built for different architectures and then
# get copied from the built host into the container image
#
# See https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md
# See https://github.com/kubernetes/test-infra/blob/HEAD/config/jobs/image-pushing/README.md
# for more details on image pushing process in Kubernetes.
#
# To promote release images, see https://github.com/kubernetes/k8s.io/tree/main/k8s.gcr.io/images/k8s-staging-sig-storage.
# To promote release images, see https://github.com/kubernetes/k8s.io/tree/HEAD/k8s.gcr.io/images/k8s-staging-sig-storage.

# This must be specified in seconds. If omitted, defaults to 600s (10 mins).
# Building three images in external-snapshotter takes roughly half an hour,
Expand All @@ -27,7 +27,7 @@ steps:
# The image must contain bash and curl. Ideally it should also contain
# the desired version of Go (currently defined in release-tools/prow.sh),
# but that just speeds up the build and is not required.
- name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20210331-c732583'
- name: 'gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55'
entrypoint: ./.cloudbuild.sh
env:
- GIT_TAG=${_GIT_TAG}
Expand Down
35 changes: 32 additions & 3 deletions release-tools/go-get-kubernetes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,43 @@ set -o pipefail
cmd=$0

function help () {
echo "$cmd <kubernetes version = x.y.z> - update all components from kubernetes/kubernetes to that version"
cat <<EOF
$cmd -p <kubernetes version = x.y.z>
Update all components from kubernetes/kubernetes to that version.
By default, replace statements are added for all Kubernetes packages,
whether they are used or not. This is useful when preparing a
repository for using k8s.io/kubernetes, because those replace
statements are needed to avoid "unknown revision v0.0.0" errors
(https://github.com/kubernetes/kubernetes/issues/79384).
With the optional -p flag, all unused replace statements are
pruned. This makes go.mod smaller, but isn't required.
The replace statements are needed for "go get -u ./..." which
otherwise ends up updating Kubernetes packages like client-go to
incompatible versions (in that case, a very old 1.x release which
happens to have a "higher" version number than the current
0.<Kubernetes minor version>.<Kubernetes patch version> numbers.
EOF
}

prune=false

while getopts "ph" o; do
case "$o" in
h) help; exit 0;;
p) prune=true;;
*) help; exit 1;;
esac
done
shift $((OPTIND-1))

if [ $# -ne 1 ]; then
help
exit 1
fi
case "$1" in -h|--help|help) help; exit 0;; esac

die () {
echo >&2 "$@"
Expand All @@ -55,7 +84,7 @@ mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubuserconten
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
) || die "failed to determine Kubernetes staging modules"
for mod in $mods; do
if ! (env GO111MODULE=on go mod graph) | grep "$mod@" > /dev/null; then
if $prune && ! (env GO111MODULE=on go mod graph) | grep "$mod@" > /dev/null; then
echo "Kubernetes module $mod is not used, skipping"
# Remove the module from go.mod "replace" that was added by an older version of this script.
(set -x; env GO111MODULE=on go mod edit "-dropreplace=$mod") || die "'go mod edit' failed"
Expand Down
40 changes: 26 additions & 14 deletions release-tools/prow.sh
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,15 @@ version_to_git () {
# the list of windows versions was matched from:
# - https://hub.docker.com/_/microsoft-windows-nanoserver
# - https://hub.docker.com/_/microsoft-windows-servercore
configvar CSI_PROW_BUILD_PLATFORMS "linux amd64; linux ppc64le -ppc64le; linux s390x -s390x; linux arm64 -arm64; windows amd64 .exe nanoserver:1809 servercore:ltsc2019; windows amd64 .exe nanoserver:1909 servercore:1909; windows amd64 .exe nanoserver:2004 servercore:2004; windows amd64 .exe nanoserver:20H2 servercore:20H2" "Go target platforms (= GOOS + GOARCH) and file suffix of the resulting binaries"
configvar CSI_PROW_BUILD_PLATFORMS "linux amd64; linux ppc64le -ppc64le; linux s390x -s390x; linux arm -arm; linux arm64 -arm64; windows amd64 .exe nanoserver:1809 servercore:ltsc2019; windows amd64 .exe nanoserver:1909 servercore:1909; windows amd64 .exe nanoserver:2004 servercore:2004; windows amd64 .exe nanoserver:20H2 servercore:20H2; windows amd64 .exe nanoserver:ltsc2022 servercore:ltsc2022" "Go target platforms (= GOOS + GOARCH) and file suffix of the resulting binaries"

# If we have a vendor directory, then use it. We must be careful to only
# use this for "make" invocations inside the project's repo itself because
# setting it globally can break other go usages (like "go get <some command>"
# which is disabled with GOFLAGS=-mod=vendor).
configvar GOFLAGS_VENDOR "$( [ -d vendor ] && echo '-mod=vendor' )" "Go flags for using the vendor directory"

configvar CSI_PROW_GO_VERSION_BUILD "1.16" "Go version for building the component" # depends on component's source code
configvar CSI_PROW_GO_VERSION_BUILD "1.17.3" "Go version for building the component" # depends on component's source code
configvar CSI_PROW_GO_VERSION_E2E "" "override Go version for building the Kubernetes E2E test suite" # normally doesn't need to be set, see install_e2e
configvar CSI_PROW_GO_VERSION_SANITY "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building the csi-sanity test suite" # depends on CSI_PROW_SANITY settings below
configvar CSI_PROW_GO_VERSION_KIND "${CSI_PROW_GO_VERSION_BUILD}" "Go version for building 'kind'" # depends on CSI_PROW_KIND_VERSION below
Expand Down Expand Up @@ -138,7 +138,7 @@ kind_version_default () {
latest|master)
echo main;;
*)
echo v0.11.0;;
echo v0.11.1;;
esac
}

Expand All @@ -149,14 +149,15 @@ configvar CSI_PROW_KIND_VERSION "$(kind_version_default)" "kind"

# kind images to use. Must match the kind version.
# The release notes of each kind release list the supported images.
configvar CSI_PROW_KIND_IMAGES "kindest/node:v1.21.1@sha256:fae9a58f17f18f06aeac9772ca8b5ac680ebbed985e266f711d936e91d113bad
kindest/node:v1.20.7@sha256:e645428988191fc824529fd0bb5c94244c12401cf5f5ea3bd875eb0a787f0fe9
kindest/node:v1.19.11@sha256:7664f21f9cb6ba2264437de0eb3fe99f201db7a3ac72329547ec4373ba5f5911
kindest/node:v1.18.19@sha256:530378628c7c518503ade70b1df698b5de5585dcdba4f349328d986b8849b1ee
kindest/node:v1.17.17@sha256:c581fbf67f720f70aaabc74b44c2332cc753df262b6c0bca5d26338492470c17
kindest/node:v1.16.15@sha256:430c03034cd856c1f1415d3e37faf35a3ea9c5aaa2812117b79e6903d1fc9651
kindest/node:v1.15.12@sha256:8d575f056493c7778935dd855ded0e95c48cb2fab90825792e8fc9af61536bf9
kindest/node:v1.14.10@sha256:6033e04bcfca7c5f2a9c4ce77551e1abf385bcd2709932ec2f6a9c8c0aff6d4f" "kind images"
configvar CSI_PROW_KIND_IMAGES "kindest/node:v1.22.0@sha256:b8bda84bb3a190e6e028b1760d277454a72267a5454b57db34437c34a588d047
kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6
kindest/node:v1.20.7@sha256:cbeaf907fc78ac97ce7b625e4bf0de16e3ea725daf6b04f930bd14c67c671ff9
kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f32729
kindest/node:v1.18.19@sha256:7af1492e19b3192a79f606e43c35fb741e520d195f96399284515f077b3b622c
kindest/node:v1.17.17@sha256:66f1d0d91a88b8a001811e2f1054af60eef3b669a9a74f9b6db871f2f1eeed00
kindest/node:v1.16.15@sha256:83067ed51bf2a3395b24687094e283a7c7c865ccc12a8b1d7aa673ba0c5e8861
kindest/node:v1.15.12@sha256:b920920e1eda689d9936dfcf7332701e80be12566999152626b2c9d730397a95
kindest/node:v1.14.10@sha256:f8a66ef82822ab4f7569e91a5bccaf27bceee135c1457c512e54de8c6f7219f8" "kind images"

# By default, this script tests sidecars with the CSI hostpath driver,
# using the install_csi_driver function. That function depends on
Expand Down Expand Up @@ -292,7 +293,7 @@ tests_need_alpha_cluster () {
tests_enabled "parallel-alpha" "serial-alpha"
}

# Enabling mock tests adds the "CSI mock volume" tests from https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/csi_mock_volume.go
# Enabling mock tests adds the "CSI mock volume" tests from https://github.com/kubernetes/kubernetes/blob/HEAD/test/e2e/storage/csi_mock_volume.go
# to the e2e.test invocations (serial, parallel, and the corresponding alpha variants).
# When testing canary images, those get used instead of the images specified
# in the e2e.test's normal YAML files.
Expand Down Expand Up @@ -795,7 +796,7 @@ install_snapshot_controller() {
kind load docker-image --name csi-prow ${NEW_IMG} || die "could not load the snapshot-controller:csiprow image into the kind cluster"

# deploy snapshot-controller
echo "Deploying snapshot-controller"
echo "Deploying snapshot-controller from ${SNAPSHOT_CONTROLLER_YAML} with $NEW_IMG."
# Replace image in SNAPSHOT_CONTROLLER_YAML with snapshot-controller:csiprow and deploy
# NOTE: This logic is similar to the logic here:
# https://github.com/kubernetes-csi/csi-driver-host-path/blob/v1.4.0/deploy/util/deploy-hostpath.sh#L155
Expand Down Expand Up @@ -832,8 +833,19 @@ install_snapshot_controller() {
echo "$modified"
exit 1
fi
echo "kubectl apply -f ${SNAPSHOT_CONTROLLER_YAML}(modified)"
done
elif [ "${CSI_PROW_DRIVER_CANARY}" = "canary" ]; then
echo "Deploying snapshot-controller from ${SNAPSHOT_CONTROLLER_YAML} with canary images."
yaml="$(kubectl apply --dry-run=client -o yaml -f "$SNAPSHOT_CONTROLLER_YAML")"
# Ignore: See if you can use ${variable//search/replace} instead.
# shellcheck disable=SC2001
modified="$(echo "$yaml" | sed -e "s;image: .*/\([^/:]*\):.*;image: ${CSI_PROW_DRIVER_CANARY_REGISTRY}/\1:canary;")"
diff <(echo "$yaml") <(echo "$modified")
if ! echo "$modified" | kubectl apply -f -; then
echo "modified version of $SNAPSHOT_CONTROLLER_YAML:"
echo "$modified"
exit 1
fi
else
echo "kubectl apply -f $SNAPSHOT_CONTROLLER_YAML"
kubectl apply -f "$SNAPSHOT_CONTROLLER_YAML"
Expand Down