Upgrade Go version/Knative/Spark Operator/Kubernetes Client/Docker Compose for dev environment.

[ashwinath]

In lieu of keeping all our libraries and dependencies up to date, in this PR, we are updating everything to allow compatibility with the following:

• Kubernetes 1.22
• Istio 12.5
• Knative 1.0
• Spark Operator v1beta2-1.3.3-3.1.1

I would not suggest using this in our clusters at the moment as the version difference is very large. For instance, kubectl supports only maximum of 1 skew version. Merlin is also taking the same approach here: caraml-dev/merlin#221. Should we merge this into a separate branch instead of the main branch?

This also requires the cluster to have the networking/v1beta1 API installed for the Istio APIs. The other API versions remains unchanged; Spark and Knative components are using the same version.

Also, Go version has been upgraded to 1.16 to prevent compilation issues with the newer libraries. I have kept it the same as what Knative and Istio uses.

Trying to upgrade beyond 1.16 yields the following error which I'm not sure how to solve:

github.com/gojek/turing/api/turing/service imports
        golang.org/x/oauth2/google imports
        cloud.google.com/go/compute/metadata: ambiguous import: found package cloud.google.com/go/compute/metadata in multiple modules:
        cloud.google.com/go v0.97.0 (${HOME}/go/pkg/mod/cloud.google.com/go@v0.97.0/compute/metadata)
        cloud.google.com/go/compute v1.1.0 (${HOME}/go/pkg/mod/cloud.google.com/go/compute@v1.1.0/metadata)

EDIT:

Found a way to upgrade to Go 1.18. Turing API, Experiment and Router has been upgraded to Go 1.18. The golangci-linter has been upgraded to the latest version to support the newer language features.

I have also changed the CI to allow the Github Actions to run when a PR targets any branch (previously only main).

[romanwozniak]

Should we merge this into a separate branch instead of the main branch?

Yes, this makes sense.

I would not suggest using this in our clusters at the moment as the version difference is very large

Could you elaborate on what's the proposed plan then?

1. Branch from main and prepare the new release that is guaranteed to work with the upgraded knative/istio
2. Upgrade cluster dependencies
3. Deploy the new release into the upgraded cluster
4. Merge the branch with gojek:main

Is this right?

Trying to upgrade beyond 1.16 yields the following error which I'm not sure how to solve

What have you tried so far? Any analysis on the root cause?

[ashwinath]

Could you elaborate on what's the proposed plan then?

Originally I was thinking along the lines of these:

For our current deployments with the old clusters, things should remain the same. But for the new deployments, a separate pipeline should mimic the old pipeline, with the difference of the target Kubernetes clusters and different tagged versions. We should tag the new branch and build separately from the main branch. In summary it should look something like this:

Old pipeline:

1. Tag new version on main branch.
2. Deploy to current Kubernetes clusters.

New pipeline:

1. Tag new version on new branch.
2. Deploy to new Kubernetes cluster.
3. Rebase any new changes on main and test again (repeat this step as many times as needed).

Reconciliation:

1. Merge to main
2. Tag new version
3. Target new clusters.

However @krithika369 was suggesting merging to main directly after the 1.4 release (correct me if I am wrong here). Any updates required should be merged to the 1.4 branch and tagged again. It will still require a new pipeline for testing on the new cluster. In summary it should look something like this:

Old pipeline:

1. Branch out release 1.4 and tag it.
2. Deploy to current Kubernetes clusters.

New pipeline:

1. Tag new version on main.
2. Deploy to new Kubernetes cluster.
3. Repeat steps 1 - 2 as many times as needed

No reconciliation needed because releases will be done periodically with the new branching method. I prefer @krithika369's method.

What have you tried so far? Any analysis on the root cause?

I will attend to this later as I am currently testing the batch ensembling part as of now.

[ashwinath]

Some clarifications to the above comment:

This branch will only be merged into main after the feature is fully completed and after release 1.4.

[ashwinath]

Trying to upgrade beyond 1.16 yields the following error which I'm not sure how to solve

What have you tried so far? Any analysis on the root cause?

Managed to find a way to use Go 1.18. Someone had an issue with upgrading from 1.16 as well: aws-observability/aws-otel-collector#926. I upgraded experiments and router to Go 1.18 as well.

The other upgrade was the golangci-linter as well, upgraded to the latest version 1.45.2 to support Go 1.18.

[romanwozniak]

Thanks, it lgtm. Development documentation should also be updated to reflect the changes to the Golang version

[ashwinath]

Thanks, it lgtm. Development documentation should also be updated to reflect the changes to the Golang version

I have updated the Go versions as well as all the dependency versions such as Knative/Istio/Spark etc and the new ports that is required by Istio's mutating webhook.