From 821c50343265daee83091e617440bfe19f35d8c3 Mon Sep 17 00:00:00 2001 From: Andy Goldstein Date: Tue, 9 May 2023 17:16:40 -0400 Subject: [PATCH] Add rebase documentation Signed-off-by: Andy Goldstein --- .../content/developers/rebasing-kubernetes.md | 245 ++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 docs/content/developers/rebasing-kubernetes.md diff --git a/docs/content/developers/rebasing-kubernetes.md b/docs/content/developers/rebasing-kubernetes.md new file mode 100644 index 00000000000..a83cf6ad921 --- /dev/null +++ b/docs/content/developers/rebasing-kubernetes.md @@ -0,0 +1,245 @@ +# Rebasing Kubernetes + +This describes the process of rebasing kcp onto a new Kubernetes version. For the examples below, we'll be rebasing +onto v1.26.3. + +# 1. Update kcp-dev/apimachinery + +1. Create a new branch for the update, such as `1.26-prep`. +2. Update go.mod: + 1. You may need to change the go version at the top of the file to match what's in go.mod in the root of the + Kubernetes repo. This is not required, but probably a good idea. + 2. Update the primary Kubernetes dependencies: + ``` + go get -u k8s.io/api@v0.26.3 k8s.io/apimachinery@v0.26.3 k8s.io/client-go@v0.26.3 + ``` +3. Manually review the code that is upstream from `third_party` and make the same/similar edits to anything that + changed upstream. For example, we maintain a slightly modified copy of the + [shared informer code](https://github.com/kubernetes/kubernetes/blob/v1.26.3/staging/src/k8s.io/client-go/tools/cache/shared_informer.go). + In between 1.24 (the previous Kubernetes version upon which kcp was based) and 1.26, the `AddEventHandler` + function was modified to return a `ResourceEventHandlerRegistration` and an `error`. To rebase + kcp-dev/apimachinery onto Kubernetes v1.26.3, we had to make our copy do the same thing. +4. Run `make lint test`. Fix any issues you encounter. +5. Commit your changes. +6. Push to your fork. +7. Open a PR; get it reviewed and merged. + +# 2. Update kcp-dev/code-generator + +1. Create a new branch for the update, such as `1.26-prep`. +2. Update go.mod: + 1. You may need to change the go version at the top of the file to match what's in go.mod in the root of the + Kubernetes repo. This is not required, but probably a good idea. + 2. Update the primary Kubernetes dependencies: + ``` + go get -u k8s.io/apimachinery@v0.26.3 k8s.io/code-generator@v0.26.3 + ``` +3. Update Kubernetes tooling versions in the `Makefile`: + 1. `KUBE_CLIENT_GEN_VER := v0.26.3` + 2. `KUBE_LISTER_GEN_VER := v0.26.3` + 3. `KUBE_INFORMER_GEN_VER := v0.26.3` + 4. `KUBE_APPLYCONFIGURATION_GEN_VER := v0.26.3` +4. Update generators, if needed: + 1. Manually review upstream to check for any changes to these generators in the `kubernetes` repository: + 1. staging/src/k8s.io/code-generator/cmd/client-gen + 2. staging/src/k8s.io/code-generator/cmd/informer-gen + 3. staging/src/k8s.io/code-generator/cmd/lister-gen + 2. If there were any substantive changes, make the corresponding edits to our generators. For example, in between + 1.24 and 1.26, upstream Kubernetes added shutdown support to shared informer factories. We had to add the same + thing to our generated shared informer factories. + 3. You'll probably want to commit your changes at this point. +5. Run `make codegen`. You'll probably want to commit these changes as a standalone commit. +6. Run `make lint test build` and fix any issues you encounter. +7. Commit any remaining changes. +8. Push to your fork. +9. Open a PR; get it reviewed and merged. + +# 3. Update kcp-dev/client-go +1. Create a new branch for the update, such as `1.26-prep`. +2. Update go.mod: + 1. You may need to change the go version at the top of the file to match what's in go.mod in the root of the + Kubernetes repo. This is not required, but probably a good idea. + 2. Update the `kcp-dev/apimachinery` dependency: + ``` + go get -u github.com/kcp-dev/apimachinery@main + ``` + 3. That should have updated the primary Kubernetes dependencies, but in case it didn't, you can do so manually: + ``` + go get -u k8s.io/api@v0.26.3 k8s.io/apimachinery@v0.26.3 k8s.io/client-go@v0.26.3 + ``` + 4. Update the kcp code-generator tooling version in the `Makefile`: + 1. Look up the latest commit from the main branch in https://github.com/kcp-dev/code-generator. + 2. Adjust `CODE_GENERATOR_VER` to match accordingly. +3. Manually review the code that is upstream from `third_party` and make the same/similar edits to anything that + changed upstream. +4. Run `make codegen`. You'll probably want to commit these changes as a standalone commit. +5. Run `make lint` and fix any issues you encounter. +6. Commit any remaining changes. +7. Push to your fork. +8. Open a PR; get it reviewed and merged. + +# 4. Update kcp-dev/kubernetes + +1. First and foremost, take notes of what worked/didn't work well. Update this guide based on your experiences! +2. Remember, if you mess up, `git rebase --abort` and `git reflog` are your very good friends! +3. Terminology: + - "Old" version: the current version of Kubernetes that kcp is using + - "New" version: the new Kubernetes version on top of which you are rebasing +4. The `upstream` in e.g. `upstream/kcp-feature-logical-cluster-1.24-v3` is the name of the git remote that points + to github.com/kubernetes/kubernetes. Please adjust the commands below if your remote is named differently. +5. Prepare the old branch for rebasing: + 1. In this example, the old version is 1.24, and the new version is 1.26. + 2. Create a new temporary branch to clean up the accumulated commits on the old branch: + ``` + git checkout -b kcp-1.24-clean upstream/kcp-feature-logical-cluster-1.24-v3 + ``` + 3. Start by doing a soft git reset so your working directory contains all the changes from the branch in an + uncommitted state: + ``` + git reset v1.24.3 # this must be whatever upstream commit was the starting point for the old branch + ``` + 4. Revert the following types of changes: + - go.mod/go.sum + - Generated clients, listers, informers, applyconfigurations + - vendor/* + 5. Create a single baseline commit: + ``` + git add . + git commit -m 'UPSTREAM: : baseline for kcp 1.24' + ``` +6. Create a baseline branch that starts from the new version. We'll use this to cherry-pick any commits from + upstream that we need to continue carrying (i.e., they have not yet merged into the new version). + ``` + git checkout -b kcp-1.26-baseline v1.26.3 + ``` +7. Review the `git log` from the old version branch (this will typically also be the default branch in GitHub). Look + for commit messages of the format + `UPSTREAM: 12345: ...`. The number indicates the upstream pull request. You'll need to inspect each pull request + to determine when it was merged. Anything that merged **before** the rebase version can be ignored and omitted + from the baseline branch. Anything that merged **after** the rebase version as well as anything not yet merged + must be cherry-picked to the baseline branch. For example, in 1.24, we have `UPSTREAM: 111898: Reflector: support + logging Unstructured type`. This was added in 1.27, and therefore needs to be cherry-picked to the + `kcp-1.26-baseline` branch. +8. Create a branch where you'll attempt to do the rebase. This will start from the cleaned up old version branch: + ``` + git checkout -b kcp-1.26-rebase kcp-1.24-clean + ``` +9. Rebase onto the baseline branch you created above: + ``` + git rebase kcp-1.26-baseline + ``` + + You will likely encounter numerous conflicts. This is 100% normal! Go through each file and resolve them as best + as you can. Make notes of any particular changes you have questions about, so you don't forget them. +10. Update kcp dependencies, for all kcp repositories that changed. For example: + ``` + hack/pin-dependency.sh github.com/kcp-dev/logicalcluster/v3 v3.0.4 + hack/pin-dependency.sh github.com/kcp-dev/apimachinery/v2 767ac05aebce82530dee46e9dab8c7bb47f1c823 + hack/pin-dependency.sh github.com/kcp-dev/client-go 654321d8cac56f9944e8cf801dc15e37b3a582f3 + ``` +11. Review changes to the origins of pkg/genericcontrolplane to see if similar changes are needed in the new version: + + | Destination | Source(s) | + |------------------------------------------------------------------|------------------------------------------------------| + | pkg/genericcontrolplane | cmd/kube-apiserver/app, pkg/kubeapiserver/admission | + | pkg/genericcontrolplane/apis/apis.go | pkg/controlplane/instance.go | + | pkg/genericcontrolplane/options.go | cmd/kube-apiserver/app/options/options.go | + | pkg/api/genericcontrolplane/scheme.go | pkg/api/legacyscheme/scheme.go | + | pkg/apis/core/install/genericcontrolplane/install.go | pkg/apis/core/install/install.go | + | pkg/apis/core/register_generic_control_plane.go | pkg/apis/core/register.go | + | pkg/apis/core/v1/register_generic_control_plane.go | pkg/apis/core/v1/register.go | + | staging/src/k8s.io/api/core/v1/register_generic_control_plane.go | staging/src/k8s.io/api/core/v1/register.go | + | pkg/registry/core/rest/genericcontrolplane/storage_core.go | pkg/registry/core/rest/storage_core.go | + | pkg/kubeapiserver/legacy_storage_factory_builder.go | pkg/kubeapiserver/default_storage_factory_builder.go | + +12. Update generated clients, informers, listers, etc. because we generate logical cluster aware versions of these + for Kubernetes to use: + ``` + hack/update-codegen.sh + ``` + +13. Check if any new controllers were added to Kubernetes. If so, determine if they are relevant to the control + plane, or if they're specific to workloads (anything related to pods/nodes/etc.). If they're for the control + plane and you think they should be enabled in kcp, you have 1 of 2 choices: + + 1. Modify the controller in Kubernetes to be logical cluster aware + 2. Add code to kcp to spawn a new controller instance scoped to a single logical cluster + + For option 1, follow what we did in pkg/controller/namespace/namespace_controller.go. + + For option 2, follow what we did in kcp in either the garbage collector or quota controllers. + +14. Check if any new admission plugins were added to Kubernetes. Decide if we need them in kcp. If so, make a note + of them, and we'll add them to kcp below. + +15. Push your commits to your fork of Kubernetes in GitHub. + +16. Open a pull request for review **against the baseline branch, e.g. kcp-1.26-baseline**, but mark it `WIP` and maybe + even open it in draft mode - you don't want to merge anything just yet. + +# 5. Update kcp-dev/kcp + +1. At this point, you're ready to try to integrate the updates into kcp proper. There is still likely a good + amount of work to do, so don't get discouraged if you encounter dozens or hundreds of compilation issues at + this point. That's totally normal! +2. Update go.mod: + 1. You'll need to adjust the versions of kcp-dev/apimachinery, client-go, and logicalcluster to match any changes + you made in previous steps. Make sure the versions that are in the Kubernetes go.mod match the versions used here. + 2. If any new staging repositories were added in between the old and new Kubernetes versions, you'll need to add + those in the `replace` directive at the bottom of go.mod. For example, in the 1.26 rebase, we had to add 4: + - k8s.io/dynamic-resource-allocation + - k8s.io/kms + - k8s.io/sample-cli-plugin + - k8s.io/sample-controller + 3. Go ahead and make a commit here, as the next change we'll be making is to point kcp at your local checkout of + Kubernetes. +3. Point kcp at your local checkout of Kubernetes: + ``` + # Change KUBE per your local setup + KUBE=../../../go/src/k8s.io/kubernetes + gsed -i "s,k8s.io/\(.*\) => .*/kubernetes/.*,k8s.io/\1 => $KUBE/vendor/k8s.io/\1,;s,k8s.io/kubernetes => .*,k8s.io/kubernetes => $KUBE," go.mod + ``` + !!! warning + Don't commit your changes to go.mod/go.sum. They point to your local file system. +4. Resolve any conflicts +5. Run `make modules` +6. Run `make codegen` +7. Keep iterating to get all the code to compile +8. Get the `lint` and `test` make targets to pass +9. Get the `e2e-*` make targets to pass. + +# 6. Test CI + +1. Undo your changes to go.mod and go.sum that point to your local checkout: + ``` + git checkout -- go.mod go.sum + ``` +2. Update go.mod to point to your rebase branch in your Kubernetes fork: + ``` + GITHUB_USER= BRANCH=kcp-1.26-rebase hack/bump-k8s.sh + ``` +3. Commit this change with a message such as `UNDO: my Kubernetes`. We won't be using this commit when it's time to + merge, so we make it easy to find. +4. Open a pull request in kcp. Wait for CI results. The `deps` job will always fail at this point because it's + pointing to your fork of Kubernetes. This is expected, so don't worry. Your job at this point is to get all the + other CI jobs to pass. + +# 7. Get it merged! + +1. Once CI is passing (except for the `deps` job, as expected), we're ready to merge! +2. Coordinate with another project member - show them the test results, then get them to approve your rebase PR in + kcp-dev/kubernetes. Get the PR merged (this is something that currently must be done manually - ask someone with + merge rights to do it for you if you need help). +3. Rename/create a new branch in kcp-dev/kubernetes based off what you just merged into kcp-1.26-baseline - this is + the actual rebase! The new branch could be named something like `kcp-1.26`. +4. Back in your local checkout of kcp, update to the kcp-dev/kubernetes rebase branch: + ``` + BRANCH=kcp-1.26 hack/bump-k8s.sh + ``` +5. Commit this change. You can either squash this with your previous `UNDO` commit (reword it so it's not an `UNDO` + any more), or drop the `UNDO` commit and replace it with this one. +6. Check on CI. Hopefully everything is green. If not, keep iterating on it. + +# 7. Update the default branch in kcp-dev/kubernetes + +1. Change it to your new rebase branch, e.g. `kcp-1.26`