Skip to content

Commit

Permalink
Merge commit '9259807b2334ff469d3018da685e5c91fa30faa8' into prow-upd…
Browse files Browse the repository at this point in the history
…ate-release-1.2
  • Loading branch information
pohly committed Nov 22, 2021
2 parents 45b15b7 + bbd32f6 commit 783ee65
Show file tree
Hide file tree
Showing 12 changed files with 354 additions and 122 deletions.
45 changes: 45 additions & 0 deletions release-tools/KUBERNETES_CSI_OWNERS_ALIASES
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md

aliases:

# SIG-Storage chairs and leads should always have approval rights in all repos.
# Others may be added as needed here or in each repo.
kubernetes-csi-approvers:
- jsafrane
- msau42
- saad-ali
- xing-yang

# Reviewers are automatically assigned to new PRs. The following
# reviewers will be active in all repos. Other reviewers can be
# added in each repo.
#
# Reviewers are encouraged to set the "Busy" flag in their GitHub status
# when they are temporarily unable to review PRs.
kubernetes-csi-reviewers:
- andyzhangx
- chrishenzie
- ggriffiths
- gnufied
- humblec
- j-griffith
- Jiawei0227
- jingxu97
- jsafrane
- pohly
- xing-yang

# This documents who previously contributed to Kubernetes-CSI
# as approver.
emeritus_approver:
- lpabon
- sbezverk
- vladimirvivien

# This documents who previously contributed to Kubernetes-CSI
# as reviewer.
emeritus_reviewer:
- lpabon
- saad-ali
- sbezverk
- vladimirvivien
7 changes: 2 additions & 5 deletions release-tools/OWNERS
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md

approvers:
- saad-ali
- msau42
- kubernetes-csi-approvers
- pohly

reviewers:
- saad-ali
- msau42
- pohly
- kubernetes-csi-reviewers
1 change: 1 addition & 0 deletions release-tools/OWNERS_ALIASES
4 changes: 2 additions & 2 deletions release-tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ images. Building from master creates the main `canary` image.
Sharing and updating
--------------------

[`git subtree`](https://github.com/git/git/blob/master/contrib/subtree/git-subtree.txt)
[`git subtree`](https://github.com/git/git/blob/HEAD/contrib/subtree/git-subtree.txt)
is the recommended way of maintaining a copy of the rules inside the
`release-tools` directory of a project. This way, it is possible to make
changes also locally, test them and then push them back to the shared
Expand Down Expand Up @@ -89,7 +89,7 @@ main

All Kubernetes-CSI repos are expected to switch to Prow. For details
on what is enabled in Prow, see
https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi
https://github.com/kubernetes/test-infra/tree/HEAD/config/jobs/kubernetes-csi

Test results for periodic jobs are visible in
https://testgrid.k8s.io/sig-storage-csi-ci
Expand Down
2 changes: 1 addition & 1 deletion release-tools/SECURITY_CONTACTS
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# to for triaging and handling of incoming issues.
#
# The below names agree to abide by the
# [Embargo Policy](https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md#embargo-policy)
# [Embargo Policy](https://github.com/kubernetes/sig-release/blob/HEAD/security-release-process-documentation/security-release-process.md#embargo-policy)
# and will be removed and replaced if they violate that agreement.
#
# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE
Expand Down
27 changes: 11 additions & 16 deletions release-tools/SIDECAR_RELEASE_PROCESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,8 @@ The release manager must:
* Be a member of the kubernetes-csi organization. Open an
[issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
kubernetes/org to request membership
* Be a top level approver for the repository. To become a top level approver,
the candidate must demonstrate ownership and deep knowledge of the repository
through active maintenance, responding to and fixing issues, reviewing PRs,
test triage.
* Be part of the maintainers or admin group for the repository. admin is a
superset of maintainers, only maintainers level is required for cutting a
release. Membership can be requested by submitting a PR to kubernetes/org.
* Be part of the maintainers group for the repository.
Membership can be requested by submitting a PR to kubernetes/org.
[Example](https://github.com/kubernetes/org/pull/1467)

## Updating CI Jobs
Expand All @@ -31,16 +26,16 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/HEAD/release-tools/prow.sh)
along with any overrides in
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/HEAD/.prow.sh)
to mirror the failing environment. Once e2e tests are passing (verify-unit tests
will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
1. Changes can then be updated in all the sidecar repos and hostpath driver repo
by following the [update
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/HEAD/README.md#sharing-and-updating).
1. New pull and CI jobs are configured by adding new K8s versions to the top of
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/HEAD/config/jobs/kubernetes-csi/gen-jobs.sh).
New pull jobs that have been unverified should be initially made optional by
setting the new K8s version as
[experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
Expand All @@ -52,7 +47,7 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Identify all issues and ongoing PRs that should go into the release, and
drive them to resolution.
1. Download v2.8+ [K8s release notes
generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
generator](https://github.com/kubernetes/release/tree/HEAD/cmd/release-notes)
1. Generate release notes for the release. Replace arguments with the relevant
information.
* Clean up old cached information (also needed if you are generating release
Expand Down Expand Up @@ -95,15 +90,15 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
the [k8s image
repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage),
repo](https://github.com/kubernetes/k8s.io/tree/HEAD/k8s.gcr.io/images/k8s-staging-sig-storage),
run `./generate.sh > images.yaml`, and send a PR with the updated images.
Once merged, the image promoter will copy the images from staging to prod.
1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
and feature pages with the new released version.
1. After all the sidecars have been released, update
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/HEAD/deploy)
and [k/k
in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
in-tree](https://github.com/kubernetes/kubernetes/tree/HEAD/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)

## Adding support for a new Kubernetes release

Expand Down Expand Up @@ -134,7 +129,7 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Once all sidecars for the new Kubernetes release are released,
either bump the version number of the images in the existing
[csi-driver-host-path
deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/HEAD/deploy)
and/or create a new deployment, depending on what Kubernetes
release an updated sidecar is compatible with. If no new deployment
is needed, then add a symlink to document that there intentionally
Expand Down
63 changes: 47 additions & 16 deletions release-tools/build.make
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# force the usage of /bin/bash instead of /bin/sh
SHELL := /bin/bash

.PHONY: build-% build container-% container push-% push clean test

# A space-separated list of all commands in the repository, must be
# set in main Makefile of a repository.
# CMDS=

# Normally, commands are expected in "cmd". That can be changed for a
# repository to something else by setting CMDS_DIR before including build.make.
CMDS_DIR ?= cmd

# This is the default. It can be overridden in the main Makefile after
# including build.make.
REGISTRY_NAME?=quay.io/k8scsi
Expand Down Expand Up @@ -63,30 +70,39 @@ endif
# Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables
# to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below.

# BUILD_PLATFORMS contains a set of <os> <arch> <suffix> triplets,
# BUILD_PLATFORMS contains a set of tuples [os arch suffix base_image addon_image]
# separated by semicolon. An empty variable or empty entry (= just a
# semicolon) builds for the default platform of the current Go
# toolchain.
BUILD_PLATFORMS =

# Add go ldflags using LDFLAGS at the time of compilation.
IMPORTPATH_LDFLAGS = -X main.version=$(REV)
IMPORTPATH_LDFLAGS = -X main.version=$(REV)
EXT_LDFLAGS = -extldflags "-static"
LDFLAGS =
LDFLAGS =
FULL_LDFLAGS = $(LDFLAGS) $(IMPORTPATH_LDFLAGS) $(EXT_LDFLAGS)
# This builds each command (= the sub-directories of ./cmd) for the target platform(s)
# defined by BUILD_PLATFORMS.
$(CMDS:%=build-%): build-%: check-go-version-go
mkdir -p bin
echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix; do \
if ! (set -x; CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '$(FULL_LDFLAGS)' -o "./bin/$*$$suffix" ./cmd/$*); then \
# os_arch_seen captures all of the $$os-$$arch seen for the current binary
# that we want to build, if we've seen an $$os-$$arch before it means that
# we don't need to build it again, this is done to avoid building
# the windows binary multiple times (see the default value of $$BUILD_PLATFORMS)
export os_arch_seen="" && echo '$(BUILD_PLATFORMS)' | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
os_arch_seen_pre=$${os_arch_seen%%$$os-$$arch*}; \
if ! [ $${#os_arch_seen_pre} = $${#os_arch_seen} ]; then \
continue; \
fi; \
if ! (set -x; cd ./$(CMDS_DIR)/$* && CGO_ENABLED=0 GOOS="$$os" GOARCH="$$arch" go build $(GOFLAGS_VENDOR) -a -ldflags '$(FULL_LDFLAGS)' -o "$(abspath ./bin)/$*$$suffix" .); then \
echo "Building $* for GOOS=$$os GOARCH=$$arch failed, see error(s) above."; \
exit 1; \
fi; \
os_arch_seen+=";$$os-$$arch"; \
done

$(CMDS:%=container-%): container-%: build-%
docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
docker build -t $*:latest -f $(shell if [ -e ./$(CMDS_DIR)/$*/Dockerfile ]; then echo ./$(CMDS_DIR)/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .

$(CMDS:%=push-%): push-%: container-%
set -ex; \
Expand Down Expand Up @@ -121,7 +137,7 @@ DOCKER_BUILDX_CREATE_ARGS ?=
# This target builds a multiarch image for one command using Moby BuildKit builder toolkit.
# Docker Buildx is included in Docker 19.03.
#
# ./cmd/<command>/Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows].
# ./$(CMDS_DIR)/<command>/Dockerfile[.Windows] is used if found, otherwise Dockerfile[.Windows].
# It is currently optional: if no such file exists, Windows images are not included,
# even when Windows is listed in BUILD_PLATFORMS. That way, projects can test that
# Windows binaries can be built before adding a Dockerfile for it.
Expand All @@ -131,30 +147,46 @@ DOCKER_BUILDX_CREATE_ARGS ?=
# the tag for the resulting multiarch image.
$(CMDS:%=push-multiarch-%): push-multiarch-%: check-pull-base-ref build-%
set -ex; \
DOCKER_CLI_EXPERIMENTAL=enabled; \
export DOCKER_CLI_EXPERIMENTAL; \
export DOCKER_CLI_EXPERIMENTAL=enabled; \
docker buildx create $(DOCKER_BUILDX_CREATE_ARGS) --use --name multiarchimage-buildertest; \
trap "docker buildx rm multiarchimage-buildertest" EXIT; \
dockerfile_linux=$$(if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi); \
dockerfile_windows=$$(if [ -e ./cmd/$*/Dockerfile.Windows ]; then echo ./cmd/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
dockerfile_linux=$$(if [ -e ./$(CMDS_DIR)/$*/Dockerfile ]; then echo ./$(CMDS_DIR)/$*/Dockerfile; else echo Dockerfile; fi); \
dockerfile_windows=$$(if [ -e ./$(CMDS_DIR)/$*/Dockerfile.Windows ]; then echo ./$(CMDS_DIR)/$*/Dockerfile.Windows; else echo Dockerfile.Windows; fi); \
if [ '$(BUILD_PLATFORMS)' ]; then build_platforms='$(BUILD_PLATFORMS)'; else build_platforms="linux amd64"; fi; \
if ! [ -f "$$dockerfile_windows" ]; then \
build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe//g' -e 's/; *;/;/g')"; \
build_platforms="$$(echo "$$build_platforms" | sed -e 's/windows *[^ ]* *.exe *[^ ]* *[^ ]*//g' -e 's/; *;/;/g' -e 's/;[ ]*$$//')"; \
fi; \
pushMultiArch () { \
tag=$$1; \
echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do \
echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
escaped_base_image=$${base_image/:/-}; \
if ! [ -z $$escaped_base_image ]; then escaped_base_image+="-"; fi; \
docker buildx build --push \
--tag $(IMAGE_NAME):$$arch-$$os-$$tag \
--tag $(IMAGE_NAME):$$arch-$$os-$$escaped_base_image$$tag \
--platform=$$os/$$arch \
--file $$(eval echo \$${dockerfile_$$os}) \
--build-arg binary=./bin/$*$$suffix \
--build-arg ARCH=$$arch \
--build-arg BASE_IMAGE=$$base_image \
--build-arg ADDON_IMAGE=$$addon_image \
--label revision=$(REV) \
.; \
done; \
images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix; do echo $(IMAGE_NAME):$$arch-$$os-$$tag; done); \
images=$$(echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
escaped_base_image=$${base_image/:/-}; \
if ! [ -z $$escaped_base_image ]; then escaped_base_image+="-"; fi; \
echo $(IMAGE_NAME):$$arch-$$os-$$escaped_base_image$$tag; \
done); \
docker manifest create --amend $(IMAGE_NAME):$$tag $$images; \
echo "$$build_platforms" | tr ';' '\n' | while read -r os arch suffix base_image addon_image; do \
if [ $$os = "windows" ]; then \
escaped_base_image=$${base_image/:/-}; \
if ! [ -z $$escaped_base_image ]; then escaped_base_image+="-"; fi; \
image=$(IMAGE_NAME):$$arch-$$os-$$escaped_base_image$$tag; \
os_version=$$(docker manifest inspect mcr.microsoft.com/windows/$${base_image} | grep "os.version" | head -n 1 | awk '{print $$2}' | sed -e 's/"//g') || true; \
docker manifest annotate --os-version $$os_version $(IMAGE_NAME):$$tag $$image; \
fi; \
done; \
docker manifest push -p $(IMAGE_NAME):$$tag; \
}; \
if [ $(PULL_BASE_REF) = "master" ]; then \
Expand Down Expand Up @@ -288,4 +320,3 @@ test-spelling:
test-boilerplate:
@ echo; echo "### $@:"
@ ./release-tools/verify-boilerplate.sh "$(pwd)"

6 changes: 3 additions & 3 deletions release-tools/cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@
# because binaries will get built for different architectures and then
# get copied from the built host into the container image
#
# See https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md
# See https://github.com/kubernetes/test-infra/blob/HEAD/config/jobs/image-pushing/README.md
# for more details on image pushing process in Kubernetes.
#
# To promote release images, see https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage.
# To promote release images, see https://github.com/kubernetes/k8s.io/tree/HEAD/k8s.gcr.io/images/k8s-staging-sig-storage.

# This must be specified in seconds. If omitted, defaults to 600s (10 mins).
# Building three images in external-snapshotter takes roughly half an hour,
Expand All @@ -27,7 +27,7 @@ steps:
# The image must contain bash and curl. Ideally it should also contain
# the desired version of Go (currently defined in release-tools/prow.sh),
# but that just speeds up the build and is not required.
- name: 'gcr.io/k8s-testimages/gcb-docker-gcloud:v20200421-a2bf5f8'
- name: 'gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20210917-12df099d55'
entrypoint: ./.cloudbuild.sh
env:
- GIT_TAG=${_GIT_TAG}
Expand Down
35 changes: 32 additions & 3 deletions release-tools/go-get-kubernetes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,43 @@ set -o pipefail
cmd=$0

function help () {
echo "$cmd <kubernetes version = x.y.z> - update all components from kubernetes/kubernetes to that version"
cat <<EOF
$cmd -p <kubernetes version = x.y.z>
Update all components from kubernetes/kubernetes to that version.
By default, replace statements are added for all Kubernetes packages,
whether they are used or not. This is useful when preparing a
repository for using k8s.io/kubernetes, because those replace
statements are needed to avoid "unknown revision v0.0.0" errors
(https://github.com/kubernetes/kubernetes/issues/79384).
With the optional -p flag, all unused replace statements are
pruned. This makes go.mod smaller, but isn't required.
The replace statements are needed for "go get -u ./..." which
otherwise ends up updating Kubernetes packages like client-go to
incompatible versions (in that case, a very old 1.x release which
happens to have a "higher" version number than the current
0.<Kubernetes minor version>.<Kubernetes patch version> numbers.
EOF
}

prune=false

while getopts "ph" o; do
case "$o" in
h) help; exit 0;;
p) prune=true;;
*) help; exit 1;;
esac
done
shift $((OPTIND-1))

if [ $# -ne 1 ]; then
help
exit 1
fi
case "$1" in -h|--help|help) help; exit 0;; esac

die () {
echo >&2 "$@"
Expand All @@ -55,7 +84,7 @@ mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubuserconten
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
) || die "failed to determine Kubernetes staging modules"
for mod in $mods; do
if ! (env GO111MODULE=on go mod graph) | grep "$mod@" > /dev/null; then
if $prune && ! (env GO111MODULE=on go mod graph) | grep "$mod@" > /dev/null; then
echo "Kubernetes module $mod is not used, skipping"
# Remove the module from go.mod "replace" that was added by an older version of this script.
(set -x; env GO111MODULE=on go mod edit "-dropreplace=$mod") || die "'go mod edit' failed"
Expand Down
Loading

0 comments on commit 783ee65

Please sign in to comment.