Define supported Juju version for CKF 1.8 and 1.9

[DnPlas]

Context

In the past months, the team has hit different issues with certain juju versions. As we are approaching the CKF 1.9 release and to provide better support for users and customers, we have to define which version of juju we'll be using for testing and eventually communicate it as the supported one.

What needs to get done

1. Look into issues that are affecting the bundle, both on 3.5 and 3.4
2. Perform some testing to identify more potential issues
3. Decide which version of juju to go with
4. Based on 3, we may need to change the CI of all repositories if we decide to go with juju 3.4 (as we are using 3.5 now)
5. Create a discourse post or some sort of communication to let everybody know which version is supported
6. Update the Supported versions entry in public docs

Definition of Done

There is a communication with the juju version to use and it is guaranteed that it works.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5887.

This message was autogenerated

[DnPlas]

Identified issues

Juju 3.4.2

• https://bugs.launchpad.net/juju/+bug/2060943

Juju 3.5.0

• https://bugs.launchpad.net/juju/+bug/2066517
• https://bugs.launchpad.net/juju/+bug/2065284 - this issue is affecting mysql-k8s 8.0/stabe, which is used by CKF

Juju 3.5.1

• https://bugs.launchpad.net/juju/+bug/2067636 - Pebble Error cannot create directory: permission denied when pushing file to container on Juju 3.5.1 #921
• https://bugs.launchpad.net/juju/+bug/2065284 - this issue is affecting mysql-k8s 8.0/stabe, which is used by CKF

Workarounds

1. Because our CI is using juju 3.5/stable, we had to pin the agent version to 3.5.0. This workaround works as long as we DO NOT integrate rocks. See ci: bootstrap juju controller with agent version 3.5.0 notebook-operators#374 and ci: bootstrap juju controller with agent version 3.5.0 kfp-operators#499 for reference.
2. We had to revert the oci-image of the kubeflow-dashboard to use the upstream image instead of the rock, see Revert "chore: use charmedkubeflow/kubeflow-central-dashboard as oci-image (#178)" kubeflow-dashboard-operator#191 and Revert "chore: use charmedkubeflow/kubeflow-central-dashboard as oci-image (#178)" kubeflow-dashboard-operator#192 for reference
3. The mysql-k8s charm has applied more strict assumes rules for avoiding using the buggy versions, see Fix assumes (k8s only) mysql-k8s-operator#431 for reference

[DnPlas]

Tests

Juju 3.4.3

I have deployed CKF 1.8/stable and I can confirm that https://bugs.launchpad.net/juju/+bug/2060943 and none of the issues present in 3.5.x are happening.

Likewise, I deployed CKF latest/edge and it seems like there are no issues.

Juju 3.5.1

I have deployed CKF 1.8/stable and as expected, some charms are failing with the mentioned errors. I was only able to run on this version because 3.5.2 hasn't been released.

In this case latest/edge will also fail.

Resolution options

The majority of the issues should be solved once juju 3.5.2 is released, which will potentially happen on the week of June 24th. Historically, the release has delayed in a few occassions (e.g. 3.4.3 delayed for more than a month), though. That being said,

Option 1: use juju 3.4.3 in all CI

With this option we could guarantee some stability as juju 3.4 has been around for a while and we know it works for deploying CKF latest/edge and 1.8/stable.

Pros:

• More stable version
• Already released
• Whenever the mysql-k8s charm decides to promote 8.0/edge -> 8.0/stable with the stricter assumes, we'll be ready
• The juju version will not be a blocker for the upcoming release

Cons:

• We will have to change all the CIs to use 3.4/stable (3.4.3)
• We will have to make the bump soon to 3.5/stable because bugs support for 3.4 will end in August 2024. See here.
• We won't be testing this version in our CI until we bump - this can be mitigated if we request the QA team to test with juju 3.5/stable on a regular basis up until we are ready to bump the juju version

Option 2: use juju 3.5/stable

With this option we guarantee the juju version is newer and we are constantly testing it.

Pros:

• Testing regularly on our CI with the latest version

Cons:

• We will have to add workarounds everywhere in our CI, which can potentially add blind spots for actual issues
• We are not sure if the upcoming patch release (3.5.2) will introduce more unknown issues
• We do not know when 3.5.2 is going to release, which can cause delays in our CKF 1.9 release

[DnPlas]

Conclusion

1. If mysql-k8s is bumped before juju 3.5.2 is released, CKF is not going to work. This is going to be a release blocker.
2. We do not know if 3.5.2 is going to be introducing new issues.
3. We should avoid using juju "edge"
4. It looks like we only have option 1
5. We should ask juju to test our product before releasing to avoid breaking
6. The team will be using 3.4/stable instead of 3.5/stable

Work from here

1. Change the juju version to 3.4/stable in all CIs
2. Let's have a testing plan with SQA for testing future versions of juju
3. ci(aks): Katib UAT fail on EKS juju 3.5.0 #942 and ci(aks+eks): bootstrap juju controller with agent version 3.5.0 #927 we can close these issue, and we have to update those scripts to download the 3.4 binary.
4. Generate a list of files that we have to update (for future reference)

[DnPlas]

Conclusion: support juju 3.4/stable >=3.4.3 for both 1.8 and 1.9 releases.
Closing this issue as the decision has been made, #944 will be used for tracking changes.