Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update Go mod support #34

Merged
merged 2 commits into from
Oct 8, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,3 +106,57 @@ Kubernetes releases:

CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh
CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh

Dependencies and vendoring
--------------------------

Most projects will (eventually) use `go mod` to manage
dependencies. `dep` is also still supported by `csi-release-tools`,
but not documented here because it's not recommended anymore.

The usual instructions for using [go
modules](https://github.com/golang/go/wiki/Modules) apply. Here's a cheat sheet
for some of the relevant commands:
- list available updates: `GO111MODULE=on go list -u -m all`
- update or add a single dependency: `GO111MODULE=on go get <package>`
- update all dependencies to their next minor or patch release:
`GO111MODULE=on go get ./...` (add `-u=patch` to limit to patch
releases)
- lock onto a specific version: `GO111MODULE=on go get <package>@<version>`
- clean up `go.mod`: `GO111MODULE=on go mod tidy`
- update vendor directory: `GO111MODULE=on go mod vendor`

`GO111MODULE=on` can be left out when using Go >= 1.13 or when the
source is checked out outside of `$GOPATH`.

`go mod tidy` must be used to ensure that the listed dependencies are
really still needed. Changing import statements or a tentative `go
get` can result in stale dependencies.

The `test-vendor` verifies that it was used when run locally or in a
pre-merge CI job. If a `vendor` directory is present, it will also
verify that it's content is up-to-date.

The `vendor` directory is optional. It is still present in projects
because it avoids downloading sources during CI builds. If this is no
longer deemed necessary, then a project can also remove the directory.

When using packages that are part of the Kubernetes source code, the
commands above are not enough because the [lack of semantic
versioning](https://github.com/kubernetes/kubernetes/issues/72638)
prevents `go mod` from finding newer releases. Importing directly from
`kubernetes/kubernetes` also needs `replace` statements to override
the fake `v0.0.0` versions
(https://github.com/kubernetes/kubernetes/issues/79384). The
`go-get-kubernetes.sh` script can be used to update all packages in
lockstep to a different Kubernetes version. It takes a single version
number like "1.16.0".

Conversion of a repository that uses `dep` to `go mod` can be done with:

GO111MODULE=on go mod init
release-tools/go-get-kubernetes.sh <current Kubernetes version from Gopkg.toml>
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
git rm -f Gopkg.toml Gopkg.lock
git add go.mod go.sum vendor
55 changes: 45 additions & 10 deletions build.make
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,26 @@ test-fmt:
# - the fabricated merge commit leaves go.mod, go.sum and vendor dir unchanged
# - release-tools also didn't change (changing rules or Go version might lead to
# a different result and thus must be tested)
# - import statements not changed (because if they change, go.mod might have to be updated)
#
# "git diff" is intelligent enough to annotate changes inside the "import" block in
# the start of the diff hunk:
#
# diff --git a/rpc/common.go b/rpc/common.go
# index bb4a5c4..5fa4271 100644
# --- a/rpc/common.go
# +++ b/rpc/common.go
# @@ -21,7 +21,6 @@ import (
# "fmt"
# "time"
#
# - "google.golang.org/grpc"
# "google.golang.org/grpc/codes"
# "google.golang.org/grpc/status"
#
# We rely on that to find such changes.
#
# Vendoring is optional when using go.mod.
.PHONY: test-vendor
test: test-vendor
test-vendor:
Expand All @@ -135,22 +155,37 @@ test-vendor:
*v0.[56789]*) dep check && echo "vendor up-to-date" || false;; \
*) echo "skipping check, dep >= 0.5 required";; \
esac; \
else \
echo "Repo uses 'go mod' for vendoring."; \
elif [ -f go.mod ]; then \
echo "Repo uses 'go mod'."; \
if [ "$${JOB_NAME}" ] && \
( [ "$${JOB_TYPE}" != "presubmit" ] || \
[ $$(git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools | wc -l) -eq 0 ] ); then \
echo "Skipping vendor check because the Prow pre-submit job does not change vendoring."; \
elif ! GO111MODULE=on go mod vendor; then \
[ $$( (git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools; \
git diff "${PULL_BASE_SHA}..HEAD" | grep -e '^@@.*@@ import (' -e '^[+-]import') | \
wc -l) -eq 0 ] ); then \
echo "Skipping vendor check because the Prow pre-submit job does not affect dependencies."; \
elif ! GO111MODULE=on go mod tidy; then \
echo "ERROR: vendor check failed."; \
false; \
elif [ $$(git status --porcelain -- vendor | wc -l) -gt 0 ]; then \
echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"; \
git status -- vendor; \
git diff -- vendor; \
elif [ $$(git status --porcelain -- go.mod go.sum | wc -l) -gt 0 ]; then \
echo "ERROR: go module files *not* up-to-date, they did get modified by 'GO111MODULE=on go mod tidy':"; \
git diff -- go.mod go.sum; \
false; \
elif [ -d vendor ]; then \
if ! GO111MODULE=on go mod vendor; then \
echo "ERROR: vendor check failed."; \
false; \
elif [ $$(git status --porcelain -- vendor | wc -l) -gt 0 ]; then \
echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"; \
git status -- vendor; \
git diff -- vendor; \
false; \
else \
echo "Go dependencies and vendor directory up-to-date."; \
fi; \
else \
echo "Go dependencies up-to-date."; \
fi; \
fi;
fi

.PHONY: test-subtree
test: test-subtree
Expand Down
104 changes: 104 additions & 0 deletions go-get-kubernetes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
#!/usr/bin/env bash

# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script can be used while converting a repo from "dep" to "go mod"
# by calling it after "go mod init" or to update the Kubernetes packages
# in a repo that has already been converted. Only packages that are
# part of kubernetes/kubernetes and thus part of a Kubernetes release
# are modified. Other k8.io packages (like k8s.io/klog, k8s.io/utils)
# need to be updated separately.

set -o pipefail

cmd=$0

function help () {
echo "$cmd <kubernetes version = x.y.z> - update all components from kubernetes/kubernetes to that version"
}

if [ $# -ne 1 ]; then
help
exit 1
fi
case "$1" in -h|--help|help) help; exit 0;; esac

die () {
echo >&2 "$@"
exit 1
}

k8s="$1"

# If the repo imports k8s.io/kubernetes (directly or indirectly), then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually have repos that do this? I checked csi-lib-utils and external-provisioner, and didn't find any kubernetes/kubernetes dependencies.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k8s.io/kubernetes does get pulled in at the moment:

For the driver it is the mount code which still hasn't been moved into a staging repo (there is work under way, I just don't remember how far it got). Not sure for external-provisioner, haven't checked.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mount dependency is expected and will soon be removed. In general, we should start trying to cleanup and avoid any k8s.io/kubernetes dependencies. But I guess it's fine to have this script in the meantime.

Does this help with the non-k8s.io/kubernetes repos (like k8s.io/api)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k8s.io/api is from k/k and gets updated by the script.

k8s.io/util isn't, but doesn't tag releases. k8s.io/klog has separate releases and shows up in go.mod with a proper version number (k8s.io/klog v0.3.1). I think "go get" picks the version specified by some other go.mod file for those modules, just as it does for other modules outside of Kubernetes.

# "go mod" will try to find "v0.0.0" versions because
# k8s.io/kubernetes has those in it's go.mod file
# (https://github.com/kubernetes/kubernetes/blob/2bd9643cee5b3b3a5ecbd3af49d09018f0773c77/go.mod#L146-L157).
# (https://github.com/kubernetes/kubernetes/issues/79384).
#
# We need to replicate the replace statements to override those fake
# versions also in our go.mod file (idea and some code from
# https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597).
mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubusercontent.com/kubernetes/kubernetes/v${k8s}/go.mod") |
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
) || die "failed to determine Kubernetes staging modules"
for mod in $mods; do
# The presence of a potentially incomplete go.mod file affects this command,
# so move elsewhere.
modinfo=$(set -x; cd /; env GO111MODULE=on go mod download -json "$mod@kubernetes-${k8s}") ||
die "failed to determine version of $mod: $modinfo"
v=$(echo "$modinfo" | sed -n 's|.*"Version": "\(.*\)".*|\1|p')
(set -x; env GO111MODULE=on go mod edit "-replace=$mod=$mod@$v") || die "'go mod edit' failed"
done

packages=

# Beware that we have to work with packages, not modules (i.e. no -m
# flag), because some modules trigger a "no Go code except tests"
# error. Getting their packages works.
if ! packages=$( (set -x; env GO111MODULE=on go list all) | grep ^k8s.io/ | sed -e 's; *;;'); then
cat >&2 <<EOF

Warning: "GO111MODULE=on go list all" failed, trying individual packages instead.

EOF
if ! packages=$( (set -x; env GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./...) | grep ^k8s.io/); then
cat >&2 <<EOF

ERROR: could not obtain package list, both of these commands failed:
GO111MODULE=on go list all
GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./pkg/...
EOF
exit 1
fi
fi

deps=
for package in $packages; do
# Some k8s.io packages do not come from Kubernetes staging and
# thus have different versioning (or none at all...). We need to
# skip those. We know what packages are from staging because we
# now have "replace" statements for them in go.mod.
#
# shellcheck disable=SC2001
module=$(echo "$package" | sed -e 's;k8s.io/\([^/]*\)/.*;k8s.io/\1;')
if grep -q -w "$module *=>" go.mod; then
deps="$deps $(echo "$package" | sed -e "s;\$;@kubernetes-$k8s;" -e 's;^k8s.io/kubernetes\(/.*\)@kubernetes-;k8s.io/kubernetes\1@v;')"
fi
done

# shellcheck disable=SC2086
(set -x; env GO111MODULE=on go get $deps 2>&1) || die "go get failed"
echo "SUCCESS"