Add hack/update package to automate PR generation

[prezha]

fixes: #4392
fixes: #9264
fixes: #9420
etc.

continued work on scripting and automating various update processes:

proposal to reuse the same approach as in pr #9384 also for issues eg, #9264 [update go version to be 1.15 in go mod], #9420 [automate pushing kicbase image from snapshot to stable after release], and potentially others that have similar requirements (ie, fully/semi-automated patching of specific files + additional actions)

the idea is that we just need to define what we want to update and with which values - as simple as possible, and leave implementation details to available functions

i have restructured the code accordingly and also expanded to include functionalities for above-mentioned issues

so, currently, with this, we can: track new kubernetes stable & latest versions, new goland stable versions, update any local fs files, update any files on github, automatically create pr for those changes (if one does not exists already), update all three remote container registries we use here (ie, gcr, docker, and github), and, if that makes sense, we can further expand

examples of automatically created (test) PRs:

• https://github.com/triplepoint-tech/minikube/pull/8
• https://github.com/triplepoint-tech/minikube/pull/9
• https://github.com/triplepoint-tech/minikube/pull/10

example command outputs:

• update go version:

❯ go run update_golang_version.go
I1013 23:32:12.326377    7595 update_golang_version.go:114] Golang stable version: 1.17.7
I1013 23:32:12.327221    7595 patch.go:111] The Plan:
{
  ".github/workflows/iso.yml": {
    "replace": {
      "go-version: '.*": "go-version: '1.17.7'"
    }
  },
  ".github/workflows/kic_image.yml": {
    "replace": {
      "go-version: '.*": "go-version: '1.17.7'"
    }
  },
  ".github/workflows/master.yml": {
    "replace": {
      "go-version: '.*": "go-version: '1.17.7'"
    }
  },
  ".github/workflows/pr.yml": {
    "replace": {
      "go-version: '.*": "go-version: '1.17.7'"
    }
  },
  ".travis.yml": {
    "replace": {
      "go: .*": "go: 1.17.7",
      "go:\\n  - .*": "go:\n  - 1.17.7"
    }
  },
  "Makefile": {
    "replace": {
      "GO_VERSION \\?= .*": "GO_VERSION ?= 1.17.7"
    }
  },
  "go.mod": {
    "replace": {
      "(?m)^go .*": "go 1.17"
    }
  },
  "hack/jenkins/common.sh": {
    "replace": {
      "sudo \\.\\/installers\\/check_install_golang\\.sh \\\".*\\\" \\\"\\/usr\\/local\\\"": "sudo ./installers/check_install_golang.sh \"1.17.7\" \"/usr/local\""
    }
  }
}
I1013 23:32:12.336384    7595 patch.go:120] Local repo updated
I1013 23:32:22.568153    7595 github.go:112] PR commit '03522ce09a056032e0c96fc36e5449ff9ca5b9a4' created: https://github.com/prezha/minikube/commit/03522ce09a056032e0c96fc36e5449ff9ca5b9a4
I1013 23:32:23.248829    7595 github.go:126] PR branch 'update-golang-version_03522ce' created: https://api.github.com/repos/prezha/minikube/git/refs/heads/update-golang-version_03522ce
I1013 23:32:24.821191    7595 patch.go:147] PR created: https://github.com/triplepoint-tech/minikube/pull/8

• update kic base image version:

❯ go run update_kicbase_version.go
I1013 23:46:57.013402   29415 update_kicbase_version.go:104] kic base image versions: 'current' is v0.0.13 and 'stable' would be v0.0.13
E1013 23:46:59.965634   29415 patch.go:185] Temporary error running '/usr/bin/docker pull gcr.io/minikube-292219/kicbase:v0.0.13' (will retry in 656.519254ms): exit status 1: Error response from daemon: manifest for gcr.io/minikube-292219/kicbase:v0.0.13 not found: manifest unknown: Failed to fetch "v0.0.13" from request "/v2/minikube-292219/kicbase/manifests/v0.0.13".
E1013 23:47:01.622627   29415 patch.go:185] Temporary error running '/usr/bin/docker pull gcr.io/minikube-292219/kicbase:v0.0.13' (will retry in 3.203647442s): exec: already started:
E1013 23:47:12.826714   29415 patch.go:185] Temporary error running '/usr/bin/docker pull gcr.io/minikube-292219/kicbase:v0.0.13' (will retry in 21.018239376s): exec: already started:
E1013 23:47:56.845340   29415 patch.go:185] Temporary error running '/usr/bin/docker pull gcr.io/minikube-292219/kicbase:v0.0.13' (will retry in 52.839431358s): exec: already started:
I1013 23:49:08.579679   29415 update_kicbase_version.go:111] local kic base reference image: docker.io/prezha/kicbase
I1013 23:49:08.971285   29415 registry.go:98] successfully logged in to Google Cloud Container Registry
I1013 23:49:09.046466   29415 registry.go:104] successfully tagged gcr.io/minikube-292219/kicbase:v0.0.13 for Google Cloud Container Registry
I1013 23:49:26.391649   29415 registry.go:110] successfully pushed gcr.io/minikube-292219/kicbase:v0.0.13 to Google Cloud Container Registry
I1013 23:49:26.391686   29415 registry.go:122] successfully updated Google Cloud Container Registry
I1013 23:49:27.821724   29415 registry.go:98] successfully logged in to Docker Hub Container Registry
I1013 23:49:27.892438   29415 registry.go:104] successfully tagged docker.io/prezha/kicbase:v0.0.13 for Docker Hub Container Registry
I1013 23:49:33.470330   29415 registry.go:110] successfully pushed docker.io/prezha/kicbase:v0.0.13 to Docker Hub Container Registry
I1013 23:49:33.470342   29415 registry.go:122] successfully updated Docker Hub Container Registry
I1013 23:49:34.077895   29415 registry.go:98] successfully logged in to GitHub Packages Registry
I1013 23:49:34.145706   29415 registry.go:104] successfully tagged docker.pkg.github.com/triplepoint-tech/minikube/kicbase:v0.0.13 for GitHub Packages Registry
I1013 23:49:38.614636   29415 registry.go:110] successfully pushed docker.pkg.github.com/triplepoint-tech/minikube/kicbase:v0.0.13 to GitHub Packages Registry
I1013 23:49:38.614673   29415 registry.go:122] successfully updated GitHub Packages Registry
I1013 23:49:38.615047   29415 patch.go:111] The Plan:
{
  "pkg/drivers/kic/types.go": {
    "replace": {
      "Version = \".*\"": "Version = \"v0.0.13\""
    }
  }
}
I1013 23:49:38.615652   29415 patch.go:118] Local repo update skipped: nothing changed
I1013 23:49:46.673485   29415 github.go:112] PR commit '1fcf74f0700889c782926c737b2f3176d28009b8' created: https://github.com/prezha/minikube/commit/1fcf74f0700889c782926c737b2f3176d28009b8
I1013 23:49:47.389907   29415 github.go:126] PR branch 'update-kicbase-version_1fcf74f' created: https://api.github.com/repos/prezha/minikube/git/refs/heads/update-kicbase-version_1fcf74f
I1013 23:49:49.067492   29415 patch.go:147] PR created: https://github.com/triplepoint-tech/minikube/pull/9

• update kubernetes version:

❯ go run update_kubernetes_version.go
I1013 23:57:30.229903    7908 update_kubernetes_version.go:82] Kubernetes versions: 'stable' is v1.20.1 and 'latest' is v1.23.4-rc.5
I1013 23:57:30.230121    7908 patch.go:111] The Plan:
{
  "pkg/minikube/constants/constants.go": {
    "replace": {
      "DefaultKubernetesVersion = \".*": "DefaultKubernetesVersion = \"v1.20.1\"",
      "NewestKubernetesVersion = \".*": "NewestKubernetesVersion = \"v1.23.4-rc.5\""
    }
  },
  "site/content/en/docs/commands/start.md": {
    "replace": {
      "'latest' for .*\\)": "'latest' for v1.23.4-rc.5)",
      "'stable' for .*,": "'stable' for v1.20.1,"
    }
  }
}
I1013 23:57:30.230658    7908 patch.go:120] Local repo updated
I1013 23:57:38.547231    7908 github.go:112] PR commit '40a13056567e6258dd6fbd3c878033c74eeabbcc' created: https://github.com/prezha/minikube/commit/40a13056567e6258dd6fbd3c878033c74eeabbcc
I1013 23:57:39.368093    7908 github.go:126] PR branch 'update-kubernetes-version_40a1305' created: https://api.github.com/repos/prezha/minikube/git/refs/heads/update-kubernetes-version_40a1305
I1013 23:57:40.702064    7908 patch.go:147] PR created: https://github.com/triplepoint-tech/minikube/pull/10

[k8s-ci-robot]

Hi @prezha. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[minikube-bot]

Can one of the admins verify this patch?

[codecov-io]

Codecov Report

Merging #9464 into master will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9464      +/-   ##
==========================================
- Coverage   29.20%   29.05%   -0.16%     
==========================================
  Files         171      171              
  Lines       10447    10443       -4     
==========================================
- Hits         3051     3034      -17     
- Misses       6972     6987      +15     
+ Partials      424      422       -2     

Impacted Files	Coverage Δ
cmd/minikube/cmd/root.go	47.28% <0.00%> (-10.81%)	⬇️
cmd/minikube/cmd/start.go	9.38% <0.00%> (-0.03%)	⬇️
pkg/drivers/common.go	8.00% <0.00%> (ø)
pkg/util/lock/lock.go	46.15% <0.00%> (ø)
pkg/minikube/out/out.go	51.88% <0.00%> (ø)
pkg/util/retry/retry.go	0.00% <0.00%> (ø)
cmd/minikube/cmd/stop.go	4.00% <0.00%> (ø)
pkg/minikube/perf/logs.go	73.91% <0.00%> (ø)
pkg/drivers/kic/oci/oci.go	0.00% <0.00%> (ø)
pkg/minikube/machine/ssh.go	0.00% <0.00%> (ø)
... and 15 more

[TravisBuddy]

Travis tests have failed

Hey @prezha,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test

PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.14.6.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.3/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.14.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.14.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="52273c1dc832d589db05e4b8b82c788e74d644a2" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v3" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.30.0'
golangci/golangci-lint info found version: 1.30.0 for v1.30.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
ok
= go mod ================================================================
ok
= boilerplate ===========================================================
boilerplate missing: /home/travis/gopath/src/github.com/kubernetes/minikube/hack/kubernetes_version/update_kubernetes_version.go
consider running: /home/travis/gopath/src/github.com/kubernetes/minikube/hack/boilerplate/fix.sh
Makefile:316: recipe for target 'test' failed
make: *** [test] Error 8

TravisBuddy Request Identifier: 8768a940-100b-11eb-8acb-27a1e4e1eb88

[tstromberg]

Hey @prezha - this PR is doing a lot. Do you mind splitting up the PR's to automate the various steps? It's fine if you want to just send one at a time.

[prezha]

hey @tstromberg, yes, of course: i've made some additional minor changes (mostly for consistency) and then split this pr into four parts/prs, each having their own references to other relevant issues/prs - in summary:

• hack/update: base library #9526 hack/update - base library/package (this one should go first, and the rest can go in any order):

includes:
- hack/update/update.go: package init, base struct w/ method, and functions
- hack/update/filesystem.go: update local filesystem repo
- hack/update/github.go: get greatest stable and latest releases, find/create PR, and update remote GitHub repo
- hack/update/registry.go: manage images (eg, pull, tag, etc.) and update container registries

• hack/update: kubernetes version #9529 hack/update - kubernetes version:

- includes: hack/update/kubernetes_version/update_kubernetes_version.go
- removes: hack/kubernetes_version/update_kubernetes_version.go

• hack/update: golang version #9531 hack/update - golang version:

- includes: hack/update/golang_version/update_golang_version.go

• hack/update: kicbase version #9532 hack/update - kicbase version:

- includes: hack/update/kicbase_version/update_kicbase_version.go

if this is ok, it's ready for review, and if for a more 'synchronous' conversation, i'm also available on k8s slack - happy to jump in

[tstromberg]

Thank you!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: prezha, tstromberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tstromberg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment