Upgrade Python version to 3.11

[tenzen-y]

What this PR does / why we need it:
I upgraded the Python version to 3.11 since Python 3.10 no longer fixes bugs, according to this.

Also, I upgraded some libraries to switch Python 3.11 supported version:

• numpy 1.25.2: https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table

Additionally, we keep the library and Python version for the skopt since some dependencies error occurs due to stopped maintenance: #2280.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

/hold

We need to upgrade other library versions first.

[tenzen-y]

After I upgraded the Python version, I faced OOM killed in the ENAS suggestion service.

[tenzen-y]

Ready for the review. PTAL.

/assign @andreyvelich @johnugeorge

[tenzen-y]

/hold cancel

[andreyvelich]

Thank you @tenzen-y!
A few questions:

1. Do we want to add support for Python 3.11 in Katib SDK: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/setup.py#L61 ?
2. Do we want to add this change before the next Katib release ?

[tenzen-y]

Thank you @tenzen-y! A few questions:

1. Do we want to add support for Python 3.11 in Katib SDK: https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/setup.py#L61 ?
2. Do we want to add this change before the next Katib release ?

@andreyvelich It's a good question.

1. I agree with you. If so, should we remove Python 3.7 from the list as Python 3.7 is no longer supported?
2. Adding support in this PR would be better since we can verify if our SDK with Python 3.11 can work for a while by adding CI:

   katib/.github/workflows/e2e-test-darts-cifar10.yaml

   Line 24 in 679e6fb

   python-version: "3.7"

[andreyvelich]

I agree with you. If so, should we remove Python 3.7 from the list as Python 3.7 is no longer supported?

How strict we want to be with our SDK versions since it is user facing interface and sometimes users might be on the out-of-date environments ?
For example, Kubernetes Python client still supports Python 3.6 and 3.7: https://github.com/kubernetes-client/python/blob/master/setup.py#L80.

However, I prefer approach which Jupyter Lab and Jupyter Server follow where they add minimal Python version that is supported by today.

Could you give your feedback on this @kubeflow/wg-training-leads @droctothorpe @shipengcheng1230 @andreafehrman @harrisonfritz

[tenzen-y]

How strict we want to be with our SDK versions since it is user facing interface and sometimes users might be on the out-of-date environments ? For example, Kubernetes Python client still supports Python 3.6 and 3.7: https://github.com/kubernetes-client/python/blob/master/setup.py#L80.

However, I prefer approach which Jupyter Lab and Jupyter Server follow where they add minimal Python version that is supported by today.

I'm ok with either way, but I believe that we should define deprecation or support policies for SDK with Python version.
For example, we decided to support 3 Kubernets minor versions.

So, should we keep supported Python versions in this PR, and then open a dedicated issue and discuss the Python version?
This means that I should not add Python 3.11 to SDK and should not remove Python 3.7 from SDK.

@andreyvelich WDYt?

[andreyvelich]

So, should we keep supported Python versions in this PR, and then open a dedicated issue and discuss the Python version?
This means that I should not add Python 3.11 to SDK and should not remove Python 3.7 from SDK.

That's fine I believe. Let's create dedicated issue to discuss long-term support for SDK Python versions.

[tenzen-y]

So, should we keep supported Python versions in this PR, and then open a dedicated issue and discuss the Python version?
This means that I should not add Python 3.11 to SDK and should not remove Python 3.7 from SDK.

That's fine I believe. Let's create dedicated issue to discuss long-term support for SDK Python versions.

I created: #2283

[tenzen-y]

@andreyvelich So, can we merge this?

[andreyvelich]

Yeah, let's merge it!
/lgtm