[PyOV] Enable Python 3.10 on Azure CI #12578

p-wysocki · 2022-08-16T09:55:45Z

Details:

Enable Python 3.10 on Azure CI
Enable building openvino_dev wheel on Python 3.10

Tickets:

89605

…into drop_python3.6

…into enable_python3.10

ilya-lavrenov · 2022-11-05T19:55:07Z

.ci/azure/linux_onnxruntime.yml

+      tar -xzf Python-"$PYTHON_VERSION".tgz
+      cd Python-"$PYTHON_VERSION" || exit
+      ./configure --enable-optimizations --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
+      make python


here and maybe in other places we are just wasting machine time with building python each time.
I still suppose we need to enable python 3.10 only on Jenkins, while on Azure we should use default python version for the system (for Windows we can install 3.10 since Windows does not have python by default)

Answered in an internal channel.

…vino into enable_python3.10

ilya-lavrenov · 2022-11-06T06:49:22Z

Please, have a look at pip conflicts issue:

[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,189] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Installing ninja-1.11.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl to /home/jenkins/agent/workspace/private-ci/ie/e2e-tests-linux-ubuntu20-pip-conflicts/b/tmp/check_pkgs_conflicts_venv_3.9_pip_22.2.2/lib/python3.9/site-packages
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,224] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Adding ninja 1.11.1 to easy-install.pth file
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,231] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Installing ninja script to /home/jenkins/agent/workspace/private-ci/ie/e2e-tests-linux-ubuntu20-pip-conflicts/b/tmp/check_pkgs_conflicts_venv_3.9_pip_22.2.2/bin
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,231] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: 
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,231] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Installed /home/jenkins/agent/workspace/private-ci/ie/e2e-tests-linux-ubuntu20-pip-conflicts/b/tmp/check_pkgs_conflicts_venv_3.9_pip_22.2.2/lib/python3.9/site-packages/ninja-1.11.1-py3.9-linux-x86_64.egg
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,270] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: error: numpy 1.23.1 is installed but numpy<1.23.0,>=1.16.5 is required by {'scipy'}
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,295] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: ========== Completed. Exit code: 1, elapsed time: 9.486 sec, 0:00:09
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,296] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: 
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,305] [140019797665536] install 3.9_pip_22.2.2 nncf ERROR: ========== ERRORS were found in the output:
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,305] [140019797665536] install 3.9_pip_22.2.2 nncf ERROR: error: numpy 1.23.1 is installed but numpy<1.23.0,>=1.16.5 is required by {'scipy'}
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,305] [140019797665536] install 3.9_pip_22.2.2 nncf WARNING: ========== See full log for details regarding the errors
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,306] [140019844339520] TaskTreeExec ERROR: Task 'install 3.9_pip_22.2.2 nncf' generated an exception: Task 'install 3.9_pip_22.2.2 nncf' - FAILED. Exit code: 1
[2022-11-06T00:14:29.375Z] Traceback (most recent call last):

And resolve merge conflicts.

p-wysocki · 2022-11-06T10:49:05Z

The pip conflicts issue is fixed by bumping NNCF commit on CI, the PR is currently waiting for review and merge. I'll resolve conflicts later to not waste CI's time.

eaidova · 2022-11-06T11:15:27Z

src/bindings/python/src/compatibility/openvino/requirements-dev.txt

-attrs==19.1.0
+pytest==4.0.1; python_version < '3.10'
+pytest==7.2.0; python_version >= '3.10'
+py>=1.9.0


are you sure that py package requires to install separately?
as I know pytest 7.2.0 removes dependency on it
pytest-dev/pytest#10396
P.S. this package has vulnerability issue, from product perspective, it can lead to problems if we will require it

We need to support a range of pytest versions due to multiple different Python versions used (and other repositories' dependencies). The reason it's added is there were CIs failing due to this package not being installed in the environment. Adding it helped with the issue.

@eaidova BTW it is still open how to score this problem and other additional packages are still using py if you look at some comments like: pytest-dev/py#287 (comment) or here tox-dev/tox#2524

It is still on-going effort to drop it everywhere and some of checks still show the issue.
I think we should be dealing with it in separate issue - not here.

I do not insist, just highlight potential problem which these changes bring

However if py is a problem, project will face it eventually, even without this change. This package is already a dependency of others... probably a new task is coming.

Thanks for noticing it!

tests/stress_tests/scripts/requirements.txt

tools/pot/setup.py

p-wysocki and others added 30 commits July 25, 2022 18:57

Fix mypy for numpy==1.21.6

90238ca

Merge branch 'drop_python3.6' of https://github.com/p-wysocki/openvino …

ca60f1d

…into drop_python3.6

Merge branch 'master' into drop_python3.6

47c12c5

Apply CR changes

1a504b8

Merge branch 'drop_python3.6' of https://github.com/p-wysocki/openvino …

2940cd8

…into drop_python3.6

Allow Python 3.6 for open source

7f9f5f0

Attempt to silence mypy

ed0fd6f

Revert to 3.6 in setup.pys

97e5021

Fix merge conflict

324a771

Fix merge conflict

4c118cb

Add newline

c63f202

Fix Python version assert

824d984

Test Python 3.10 CI

ad74f2c

Try bumping up pybind to 2.10

0899b94

Link default python to python 3.10

5492999

Add sudo to changing default python

2a6d6fb

Merge branch 'master' into drop_python3.6

2af72d0

Bump python version in setupvars

74dd405

Merge branch 'drop_python3.6' of https://github.com/p-wysocki/openvino …

b9d596d

…into drop_python3.6

Fix exception in test

f2e63e6

Do the same for compatibility

e55b20e

Apply same CI changes to other pipelines

5ee5c1b

Attempt to fix arm and windows CIs

2d4463d

Fix win py lib version and links?

dd0452b

Merge branch 'master' into drop_python3.6

bac83cd

Try to fix CI

d415e93

Merge branch 'drop_python3.6' of https://github.com/p-wysocki/openvino …

4296710

…into drop_python3.6

Fix win python path

5a29b32

Update win path

a940820

Test disable PDPD

dea4659

p-wysocki added 2 commits November 4, 2022 18:43

Merge remote-tracking branch 'jiwaszki/jiwaszki/fix_wheels_platform' …

bdd2931

…into enable_python3.10

Merge branch 'master' into enable_python3.10

26bf9ff

ilya-lavrenov reviewed Nov 5, 2022

View reviewed changes

p-wysocki added 3 commits November 5, 2022 22:48

Merge branch 'master' into enable_python3.10

c2952c8

Merge remote-tracking branch 'upstream/master' into enable_python3.10

1b29f24

Merge branch 'enable_python3.10' of https://github.com/p-wysocki/open…

3e787de

…vino into enable_python3.10

ilya-lavrenov requested a review from eaidova November 6, 2022 06:51

eaidova reviewed Nov 6, 2022

View reviewed changes

tests/stress_tests/scripts/requirements.txt Show resolved Hide resolved

eaidova reviewed Nov 6, 2022

View reviewed changes

tools/pot/setup.py Show resolved Hide resolved

eaidova reviewed Nov 6, 2022

View reviewed changes

tools/pot/setup.py Show resolved Hide resolved

p-wysocki added 2 commits November 6, 2022 12:33

Merge master

09dd822

Specify CPU plugin versions

5a05421

p-wysocki mentioned this pull request Nov 7, 2022

Limit pyparsing module openvinotoolkit/open_model_zoo#3589

Merged

AlexKoff88 approved these changes Nov 7, 2022

View reviewed changes

p-wysocki added 2 commits November 7, 2022 10:45

Bump OMZ version

d55de6a

Merge branch 'master' into enable_python3.10

33b1a08

eaidova approved these changes Nov 7, 2022

View reviewed changes

p-wysocki added 6 commits November 7, 2022 18:17

Merge branch 'master' into enable_python3.10

1f27a29

Merge branch 'master' into enable_python3.10

e74bade

Fix linter

659eaab

Fix linter 2

ec037fb

Fix linter 3

678275f

Disable linter

d7dd13f

ilya-lavrenov merged commit d2871fd into openvinotoolkit:master Nov 8, 2022

jiwaszki mentioned this pull request Nov 27, 2022

DeepSparse and OpenVINO are not working with python 3.10 nebuly-ai/optimate#103

Closed

p-wysocki deleted the enable_python3.10 branch March 31, 2023 12:18

p-wysocki mentioned this pull request Aug 8, 2024

Enable python3.12 in GHA #25964

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PyOV] Enable Python 3.10 on Azure CI #12578

[PyOV] Enable Python 3.10 on Azure CI #12578

p-wysocki commented Aug 16, 2022 •

edited

Loading

ilya-lavrenov Nov 5, 2022

p-wysocki Nov 5, 2022

ilya-lavrenov commented Nov 6, 2022 •

edited

Loading

p-wysocki commented Nov 6, 2022

eaidova Nov 6, 2022 •

edited

Loading

p-wysocki Nov 6, 2022

jiwaszki Nov 6, 2022 •

edited

Loading

eaidova Nov 6, 2022

jiwaszki Nov 6, 2022

[PyOV] Enable Python 3.10 on Azure CI #12578

[PyOV] Enable Python 3.10 on Azure CI #12578

Conversation

p-wysocki commented Aug 16, 2022 • edited Loading

Details:

Tickets:

ilya-lavrenov Nov 5, 2022

Choose a reason for hiding this comment

p-wysocki Nov 5, 2022

Choose a reason for hiding this comment

ilya-lavrenov commented Nov 6, 2022 • edited Loading

p-wysocki commented Nov 6, 2022

eaidova Nov 6, 2022 • edited Loading

Choose a reason for hiding this comment

p-wysocki Nov 6, 2022

Choose a reason for hiding this comment

jiwaszki Nov 6, 2022 • edited Loading

Choose a reason for hiding this comment

eaidova Nov 6, 2022

Choose a reason for hiding this comment

jiwaszki Nov 6, 2022

Choose a reason for hiding this comment

p-wysocki commented Aug 16, 2022 •

edited

Loading

ilya-lavrenov commented Nov 6, 2022 •

edited

Loading

eaidova Nov 6, 2022 •

edited

Loading

jiwaszki Nov 6, 2022 •

edited

Loading