release-2.0: update release-tools

release-tools/.prow.sh

@@ -1,7 +1,23 @@
 #! /bin/bash -e
+
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
 #
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 # This is for testing csi-release-tools itself in Prow. All other
 # repos use prow.sh for that, but as csi-release-tools isn't a normal
 # repo with some Go code in it, it has a custom Prow test script.
 
 ./verify-shellcheck.sh "$(pwd)"
+./verify-spelling.sh "$(pwd)"
+./verify-boilerplate.sh "$(pwd)"

release-tools/README.md

@@ -21,7 +21,11 @@ The expected repository layout is:
    Dockerfile in the root when only building a single command
  - `Makefile` - includes `release-tools/build.make` and sets
    configuration variables
- - `.travis.yml` - a symlink to `release-tools/.travis.yml`
+ - `.prow.sh` script which imports `release-tools/prow.sh`
+   and may contain further customization
+ - `.cloudbuild.sh` and `cloudbuild.yaml` as symlinks to
+   the corresponding files in `release-tools` or (if necessary)
+   as custom files
 
 To create a release, tag a certain revision with a name that
 starts with `v`, for example `v1.0.0`, then `make push`
@@ -44,10 +48,17 @@ is the recommended way of maintaining a copy of the rules inside the
 changes also locally, test them and then push them back to the shared
 repository at a later time.
 
+We no longer care about importing the full commit history, so `--squash` should be used
+when submitting a `release-tools` update. Also make sure that the PR for that
+contains the automatically generated commit message in the PR description.
+It contains the list of individual commits that were squashed. The script from
+https://github.com/kubernetes-csi/csi-release-tools/issues/7 can create such
+PRs automatically.
+
 Cheat sheet:
 
-- `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
-- `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
+- `git subtree add --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
+- `git subtree pull --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
 - edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:<user>/csi-release-tools.git <my-new-or-existing-branch>` - push to a new branch before submitting a PR
 
 verify-shellcheck.sh
@@ -81,7 +92,7 @@ on what is enabled in Prow, see
 https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi
 
 Test results for periodic jobs are visible in
-https://testgrid.k8s.io/sig-storage-csi
+https://testgrid.k8s.io/sig-storage-csi-ci
 
 It is possible to reproduce the Prow testing locally on a suitable machine:
 - Linux host

release-tools/SIDECAR_RELEASE_PROCESS.md

@@ -0,0 +1,150 @@
+# Sidecar Release Process
+
+This page describes the process for releasing a kubernetes-csi sidecar.
+
+## Prerequisites
+
+The release manager must:
+
+* Be a member of the kubernetes-csi organization. Open an
+  [issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
+  kubernetes/org to request membership
+* Be a top level approver for the repository. To become a top level approver,
+  the candidate must demonstrate ownership and deep knowledge of the repository
+  through active maintenance, responding to and fixing issues, reviewing PRs,
+  test triage.
+* Be part of the maintainers or admin group for the repository. admin is a
+  superset of maintainers, only maintainers level is required for cutting a
+  release.  Membership can be requested by submitting a PR to kubernetes/org.
+  [Example](https://github.com/kubernetes/org/pull/1467)
+
+## Updating CI Jobs
+Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs
+must be updated.
+
+[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the
+naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
+
+1. Jobs should be actively monitored to find and fix failures in sidecars and
+   infrastructure changes early in the development cycle. Test failures are sent
+   to kubernetes-sig-storage-test-failures@googlegroups.com.
+1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
+1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
+   repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
+   [prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
+   along with any overrides in
+   [.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
+   to mirror the failing environment. Once e2e tests are passing (verify-unit tests
+   will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
+1. Changes can then be updated in all the sidecar repos and hostpath driver repo
+   by following the [update
+   instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
+1. New pull and CI jobs are configured by adding new K8s versions to the top of
+   [gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
+   New pull jobs that have been unverified should be initially made optional by
+   setting the new K8s version as
+   [experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
+1. Once new pull and CI jobs have been verified, and the new Kubernetes version
+   is released, we can make the optional jobs required, and also remove the
+   Kubernetes versions that are no longer supported.
+
+## Release Process
+1. Identify all issues and ongoing PRs that should go into the release, and
+  drive them to resolution.
+1. Download v2.8+ [K8s release notes
+  generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
+1. Generate release notes for the release. Replace arguments with the relevant
+  information.
+    * Clean up old cached information (also needed if you are generating release
+      notes for multiple repos)
+      ```bash
+      rm -rf /tmp/k8s-repo
+      ```
+    * For new minor releases on master:
+        ```bash
+        GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest
+        --github-org=kubernetes-csi --github-repo=external-provisioner
+        --required-author="" --output out.md
+        ```
+    * For new patch releases on a release branch:
+        ```bash
+        GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1
+        --github-org=kubernetes-csi --github-repo=external-provisioner
+        --required-author="" --output out.md
+        ```
+1. Compare the generated output to the new commits for the release to check if
+   any notable change missed a release note.
+1. Reword release notes as needed. Make sure to check notes for breaking
+   changes and deprecations.
+1. If release is a new major/minor version, create a new `CHANGELOG-<major>.<minor>.md`
+   file. Otherwise, add the release notes to the top of the existing CHANGELOG
+   file for that minor version.
+1. Submit a PR for the CHANGELOG changes.
+1. Submit a PR for README changes, in particular, Compatibility, Feature status,
+   and any other sections that may need updating.
+1. Check that all [canary CI
+  jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing,
+  and that test coverage is adequate for the changes that are going into the release.
+1. Make sure that no new PRs have merged in the meantime, and no PRs are in
+   flight and soon to be merged.
+1. Create a new release following a previous release as a template. Be sure to select the correct
+   branch. This requires Github release permissions as required by the prerequisites.
+   [external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new)
+1. If release was a new major/minor version, create a new `release-<minor>`
+   branch at that commit.
+1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
+1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
+   the [k8s image
+   repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage),
+   run `./generate.sh > images.yaml`, and send a PR with the updated images.
+   Once merged, the image promoter will copy the images from staging to prod.
+1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
+   and feature pages with the new released version.
+1. After all the sidecars have been released, update
+   CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
+   and [k/k
+   in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
+
+## Adding support for a new Kubernetes release
+
+1. Add the new release to `k8s_versions` in
+   https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L25
+   to enable generating a job for it. Set `experimental_k8s_version`
+   in
+   https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L40
+   to ensure that the new jobs aren't run for PRs unless explicitly
+   requested. Generate and submit the new jobs.
+1. Create a test PR to try out the new job in some repo with `/test
+   pull-kubernetes-csi-<repo>-<x.y>-on-kubernetes-<x.y>` where x.y
+   matches the Kubernetes release. Alternatively, run .prow.sh in that
+   repo locally with `CSI_PROW_KUBERNETES_VERSION=x.y.z`.
+1. Optional: update to a [new
+   release](https://github.com/kubernetes-sigs/kind/tags) of kind with
+   pre-built images for the new Kubernetes release. This is optional
+   if the current version of kind is able to build images for the new
+   Kubernetes release. However, jobs require less resources when they
+   don't need to build those images from the Kubernetes source code.
+   This change needs to be tried out in a PR against a component
+   first, then get submitted against csi-release-tools.
+1. Optional: propagate the updated csi-release-tools to all components
+   with the script from
+   https://github.com/kubernetes-csi/csi-release-tools/issues/7#issuecomment-707025402
+1. Once it is likely to work in all components, unset
+   `experimental_k8s_version` and submit the updated jobs.
+1. Once all sidecars for the new Kubernetes release are released,
+   either bump the version number of the images in the existing
+   [csi-driver-host-path
+   deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
+   and/or create a new deployment, depending on what Kubernetes
+   release an updated sidecar is compatible with. If no new deployment
+   is needed, then add a symlink to document that there intentionally
+   isn't a separate deployment. This symlink is not needed for Prow
+   testing because that will use "kubernetes-latest" as fallback.
+   Update that link when creating a new deployment.
+1. Create a new csi-driver-host-path release.
+1. Bump `CSI_PROW_DRIVER_VERSION` in prow.sh to that new release and
+   (eventually) roll that change out to all repos by updating
+   `release-tools` in them. This is used when testing manually. The
+   Prow jobs override that value, so also update
+   `hostpath_driver_version` in
+   https://github.com/kubernetes/test-infra/blob/91b04e6af3a40a9bcff25aa030850a4721e2dd2b/config/jobs/kubernetes-csi/gen-jobs.sh#L46-L47

release-tools/boilerplate/boilerplate.Dockerfile.txt

@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

release-tools/boilerplate/boilerplate.Makefile.txt

@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

release-tools/boilerplate/boilerplate.bzl.txt

@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

release-tools/boilerplate/boilerplate.go.txt

@@ -0,0 +1,15 @@
+/*
+Copyright YEAR The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/