Bump Python version to 3.9

[tenzen-y]

What this PR does / why we need it:

Upgrade the Python version and the dependent libraries version with that fix some training codes that stopped working after upgrading them and fix some dockerfile for arm64 and ppc64le that were not buildable.

Also, it allows the TFEventMetricsCollector to collect metrics generated by TF 2.X and, stop supporting TF <= 1.X.

Furthermore, add the task to kubeflow-katib-presubmit to verify that tfevent-metrics-collector can collect metrics output by training code using TF2.0.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1730

Checklist:

• Docs included if any changes are user facing

[coveralls]

Coverage decreased (-0.1%) to 74.138% when pulling d428ddc on tenzen-y:issue-1730-bump-python-to-3.7 into 7be1f0a on kubeflow:master.

[tenzen-y]

I added a new trial image as discussed in this comment but could not push it to ECR in CI.

Do I need to create PR as this in kubeflow/testing repository? @kubeflow/wg-automl-leads

[andreyvelich]

I added a new trial image as discussed in this comment but could not push it to ECR in CI.

Do I need to create PR as this in kubeflow/testing repository? @kubeflow/wg-automl-leads

Yes, please can you create PR with the new image:

"katib-trial-tf-mnist-with-summaries": "katib/v1beta1/trial-tf-mnist-with-summaries"

[tenzen-y]

Thank you for letting me know. @andreyvelich

[tenzen-y]

I have rebased this PR and resolved conflicts.

[tenzen-y]

/retest

[gaocegege]

LGTM

[andreyvelich]

@tenzen-y Please can you also update update-images.sh script with your new Trial image name.

[tenzen-y]

@tenzen-y Please can you also update update-images.sh script with your new Trial image name.

Sure.

[tenzen-y]

I have modified update-images.sh.
Can you double-check the changes? @andreyvelich

[andreyvelich]

Thank you for driving this @tenzen-y!
/lgtm
/retest

[tenzen-y]

E2E sometimes fails in this PR by timeout.
I guess that when the suggestion service proposes a large batch size in this, the training Pod will be killed by OOM, and the run-e2e-experiment will fail. So I reduced batch size in the example for TFJob.

• https://argo.kubeflow-testing.com/workflows/kubeflow-test-infra/kubeflow-katib-presubmit-e2e-v1beta1-1731-b63b40e-3200-0e17?tab=workflow&nodeId=kubeflow-katib-presubmit-e2e-v1beta1-1731-b63b40e-3200-0e17-1824442209&sidePanel=logs%3Akubeflow-katib-presubmit-e2e-v1beta1-1731-b63b40e-3200-0e17-1824442209%3Amain
• https://argo.kubeflow-testing.com/workflows/kubeflow-test-infra/kubeflow-katib-presubmit-e2e-v1beta1-1731-8553312-6368-49d0?tab=workflow&nodeId=kubeflow-katib-presubmit-e2e-v1beta1-1731-8553312-6368-49d0-491296182&sidePanel=logs%3Akubeflow-katib-presubmit-e2e-v1beta1-1731-8553312-6368-49d0-491296182%3Amain

2021/12/10 11:07:56 Waiting for Experiment tfjob-mnist-with-summaries to finish
2021/12/10 11:07:56 Experiment is running: 2 Trials, 0 Pending Trials, 1 Running Trials, 1 Succeeded Trials, 0 Failed Trials
2021/12/10 11:07:56 Current optimal Trial: {tfjob-mnist-with-summaries-mzdtdxhr [{learning_rate 0.014925334116019179} {batch_size 118}] {[{accuracy 0.9707 0.9707 0.9707}]}}
2021/12/10 11:07:56 Experiment conditions: [{Created True ExperimentCreated Experiment is created 2021-12-10 10:18:12 +0000 UTC 2021-12-10 10:18:12 +0000 UTC} {Running True Experi
mentRunning Experiment is running 2021-12-10 10:18:26 +0000 UTC 2021-12-10 10:18:26 +0000 UTC}]
2021/12/10 11:08:16 Deleting Experiment tfjob-mnist-with-summaries
2021/12/10 11:08:16 Wait Experiment finish failed: Experiment run timed out

[tenzen-y]

It looks like run-e2e-experiment-tfjob was improved.
Could you please take another look? @andreyvelich

https://argo.kubeflow-testing.com/workflows/kubeflow-test-infra/kubeflow-katib-presubmit-e2e-v1beta1-1731-d428ddc-6416-735e?tab=workflow&nodeId=kubeflow-katib-presubmit-e2e-v1beta1-1731-d428ddc-6416-735e-2332995225

[andreyvelich]

Should be good now, thanks @tenzen-y!
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment