Support for mutliple installed versions - 3.13 #3605

antirotor · 2022-08-01T16:33:05Z

Feature

This PR is adding support for multiple installed versions (builds) on the same system. It will allow easy rollback and testing on farms and without disrupting existing running sessions.

On Deadline, it is now passing OpenPype version as OPENPYPE_VERSION to the job. If this environment variable is found on the job, Deadline will try to find compatible version build to use for launching that version of OpenPype. So if job was submitted from OpenPype version 3.12.2 Deadline will try to find any build that matches 3.12 and will use the latest available. Deadline will try to find it in directories specified in OpenPypeInstallationDirs configuration option.

It relies on functionality implemented here #3445.

Deployment Notes

Since #3445 you can have multiple versions of OpenPype installed on Windows. This was possible even before with Linux, but it was missing logic that will load only versions from the update zips/repository compatible with the currently running build.

So with this PR, running build will "see" only versions compatible with it and will use only them. For Deadline, where this process is reversed - you know what version was used to create the submission and you need to find correct build to launch it, you need to put builds under some common directory do OpenPype can detect them. On Windows, this is by default C:\Program Files (x86)\OpenPype but it can be any directory following logic that its direct subdirectories should contain separate build available. So with Deadline, you need to point it to the directory with following structure: /some/op/dir/3.12.2-nightly.2, /some/op/dir/3.13.0 and so on.

Testing

For proper testing, you need to create few "mock" build versions of OpenPype. It should be enough to change openpype/version.py and running tools/build.ps1|sh and after each run store produced version somewhere.

Visibility of compatible versions

OpenPype should by default see only versions compatible with the running build. So issuing command like:

.\.poetry\bin\poetry.exe run python start.py --list-versions

or

& "C:\Program Files (x86)\OpenPype\3.12.2-nightly.3\openpype_console.exe" --list-versions

should show only versions compatible with that particular build.

Deadline

When testing, don't forget to update Deadline plugins in Deadline repository.

You need to configure OpenPype Deadline plugin:

Add path to OpenPype Installation Directories. This is path-per-line, first found is used.

Jobs submitted to Deadline should have set OPENPYPE_VERSION to correct one. In job log, you should find messages about version resolution.

Implementation steps:

change Inno Setup to allow multiple versions intalled on Windows Windows installer: Clean old files and add version subfolder #3445
implement compatibility checks to functions detecting OP versions
change unzipping and tools to look into version folders in user data folder
mark as deprecated looking for versions in common folder
fix unit tests
implement version selector in the Deadline global event / OP plugin¹

🧑‍🏭 TODO: find out how to deal with multiple installations on mac.

Resolves #3415
Resolves #3416
Resolves #3439

This would go over all specified path to OpenPype in Deadline and try to find requested version among them. ↩

ynbot · 2022-08-01T16:33:09Z

Task linked: OP-3458 Support for multiple build versions

antirotor · 2022-08-01T17:45:42Z

This is re-opened version of #3456 based on correct branch to simplify review.

when finding local version, stop crashing if directory to search doesn't exist - this will allow to just use build version

and fixing the error message

m-u-r-p-h-y

there is just a small issue with log output of available versions as 3.9.1 should not be mentioned as the Latest available

2022-08-04 13:26:08:  0: PYTHON: >>> Injecting OpenPype environments ...
2022-08-04 13:26:08:  0: PYTHON: >>> Getting OpenPype executable ...
2022-08-04 13:26:08:  0: PYTHON: >>> Scanning for compatible requested version 3.12.2-nightly.3
2022-08-04 13:26:08:  0: PYTHON: --- Looking for OpenPype at: d:\REPO\
2022-08-04 13:26:08:  0: PYTHON:   - found: 3.12.2 - d:\REPO\build_3.12.2-nightly.3
2022-08-04 13:26:08:  0: PYTHON:   - found: 3.9.1 - d:\REPO\build_3.9.1
2022-08-04 13:26:08:  0: PYTHON:    ! path is not a build: d:\REPO\OpenPype
2022-08-04 13:26:08:  0: PYTHON:    ! path is not a build: d:\REPO\OpenPypeDefault
2022-08-04 13:26:08:  0: PYTHON:   - found: 3.9.0 - d:\REPO\OpenPypeInstal
2022-08-04 13:26:08:  0: PYTHON: Looking for OpenPype at: D:\utils\customOpenPypeCommands
2022-08-04 13:26:08:  0: PYTHON: *** Latest available version found is 3.9.1
2022-08-04 13:26:08:  0: PYTHON: *** Latest compatible version found is 3.12.2
2022-08-04 13:26:08:  0: PYTHON: --- OpenPype executable: d:\REPO\build_3.12.2-nightly.3\openpype_console.exe

other than that, it is working correctly

m-u-r-p-h-y

works as expected.

versions in log output are properly recognized now.

BigRoy · 2022-08-10T21:15:50Z

I'm running into issues after this PR.

Could you tell me the difference now between OpenPype executables and the OpenPype installation directories? Why do I need to set both? Doesn't that conflict each other?

antirotor · 2022-08-11T07:40:59Z

It works like this:

ZIP updates are installed as usual to %LOCALAPPDATA%\pypeclub\openpype but there is added one level with {major}.{minor}. It is backwards compatible in a way the it still looks for version one level higher.

So ZIPs are coming as usual from update repository (there is no change there) but only zips matching {major}.{minor} of currently running builds are considered (because change in {minor} usually resulted in incompatibility between build and ZIP update).

So that's how updates work, but for builds, we need reverse logic for Deadline.

Windows installer now installs OpenPype build to subfolder based on its versions - so before it was (on Windows) C:\Program Files (x86)\OpenPype but now it is C:\Program Files (x86)\OpenPype\3.13.3-nightly.2 for example. This allows multiple build versions to be installed at the same time.

Now with the Deadline, you need to tell it where it can find those builds (because they can be installe anywhere). Job on Deadline will request 3.13.2 and Deadline needs to find build that is compatible with it.

What is the issue you are running into?

BigRoy · 2022-08-11T08:10:40Z

What is the issue you are running into?

Latest develop broke Deadline jobs on our end when:

The installed version (using the installer) != used zip version (updated via OpenPype deployment control > version repository in Studio Settings)
The job is submitted with an OPENPYPE_VERSION that e.g. matches the latest zip version with major.minor
The Deadline worker has OpenPype running/open on their machine (e.g. is a workstation) - not sure if required but it's the case on our end.
Render job fails due to it trying to resolve the latest version and somehow trying to "force reinstall" the latest version/zip, which tries to delete it first which fails because it's in use.

Some logs:

2022-08-10 18:07:34:  0: Traceback (most recent call last):
2022-08-10 18:07:34:  0:   File "P:\pipeline\openpype\build\exe.win-amd64-3.7\igniter\bootstrap_repos.py", line 1496, in install_version
2022-08-10 18:07:34:  0:     shutil.rmtree(destination)
2022-08-10 18:07:34:  0:   File "C:\Users\Roy\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 516, in rmtree
2022-08-10 18:07:34:  0:   File "C:\Users\Roy\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 395, in _rmtree_unsafe
2022-08-10 18:07:34:  0:   File "C:\Users\Roy\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 395, in _rmtree_unsafe
2022-08-10 18:07:34:  0:   File "C:\Users\Roy\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 395, in _rmtree_unsafe
2022-08-10 18:07:34:  0:   [Previous line repeated 1 more time]
2022-08-10 18:07:34:  0:   File "C:\Users\Roy\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 400, in _rmtree_unsafe
2022-08-10 18:07:34:  0:   File "C:\Users\Roy\AppData\Local\Programs\Python\Python37\lib\shutil.py", line 398, in _rmtree_unsafe
2022-08-10 18:07:34:  0: PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Maqina-02\\AppData\\Local\\pypeclub\\openpype\\3.13\\openpype-v3.13.1-colorbleed.4\\openpype\\style\\fonts\\Noto_Sans\\NotoSans-Bold.ttf'
2022-08-10 18:07:34:  0: The above exception was the direct cause of the following exception:
2022-08-10 18:07:34:  0: Traceback (most recent call last):
2022-08-10 18:07:34:  0:   File "S:\openpype\OpenPype\.venv\Lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 113, in run
2022-08-10 18:07:34:  0:   File "S:\openpype\OpenPype\.venv\Lib\site-packages\cx_Freeze\initscripts\Console.py", line 15, in run
2022-08-10 18:07:34:  0:   File "start.py", line 1197, in <module>
2022-08-10 18:07:34:  0:   File "start.py", line 1056, in boot
2022-08-10 18:07:34:  0:   File "start.py", line 770, in _find_frozen_openpype
2022-08-10 18:07:34:  0:   File "P:\pipeline\openpype\build\exe.win-amd64-3.7\igniter\bootstrap_repos.py", line 1502, in install_version
2022-08-10 18:07:34:  0:     f"cannot remove existing {destination}") from e
2022-08-10 18:07:34:  0: igniter.bootstrap_repos.OpenPypeVersionIOError: cannot remove existing C:\Users\Maqina-02\AppData\Local\pypeclub\openpype\3.13\openpype-v3.13.1-colorbleed.4

The deadline logic also doesn't find the installed appdata zip version it seems:

2022-08-10 23:19:26:  0: PYTHON: >>> Getting OpenPype executable ...
2022-08-10 23:19:26:  0: PYTHON: >>> Scanning for compatible requested version 3.13.1-colorbleed.4
2022-08-10 23:19:26:  0: PYTHON: --- Looking for OpenPype at: C:\Program Files (x86)\OpenPype
2022-08-10 23:19:26:  0: PYTHON:   - found: 3.13.1 - C:\Program Files (x86)\OpenPype\3.13.1-colorbleed.1
2022-08-10 23:19:26:  0: PYTHON: Looking for OpenPype at: C:\Program Files (x86)\OpenPype\3.13.1-colorbleed.1
2022-08-10 23:19:26:  0: PYTHON:   - found: 3.13.1 - C:\Program Files (x86)\OpenPype\3.13.1-colorbleed.1
2022-08-10 23:19:26:  0: PYTHON: *** Latest available version found is 3.13.1
2022-08-10 23:19:26:  0: PYTHON: *** Latest compatible version found is 3.13.1

Anyhow, the issue is basically that it tries to force reinstall and breaks the OpenPype installations by deleting half the files until it hits the permission error. And it thus fails all Deadline jobs.

Only workaround I was able to put in place is to put this in GlobalJobPreLoad.py:

# in GlobalJobPreLoad.py -> inject_openpype_environment...

        # TODO: Hardcode disable OpenPype Version!
        print("DISABLING OPENPYPE_VERSION ENVIRONMENT KEY!")
        os.environ.pop("OPENPYPE_VERSION", None)
        requested_version = None

Originally reported on discord

antirotor · 2022-08-11T09:47:20Z

Ok, reinstalling is strange, I'll investigate.

The deadline logic also doesn't find the installed appdata zip version it seems

this is listing only compatible builds it can find, once it finds the correct one, it will start it and then it will automatically elevate to the zip version (the build is use only to bootstrap)

BigRoy · 2022-08-11T09:51:37Z

Ok, reinstalling is strange, I'll investigate.

The reinstalling happens here as mentioned on Discord.

BigRoy · 2022-08-12T08:30:41Z

@antirotor any chance you have pointers to fix this? :) The workaround I mentioned works for 99% of our cases but somehow we still occasionally get the issue with Publish jobs if the user is running OpenPype locally too. Probably because that one still (logically) gets OPENPYPE_VERSION from the job itself instead of local environment.

It's getting really annoying to restore the workstations by killing all running openpype instances on the machine, force deleting the folders and then forcing the correct update/reinstall each time because Deadline ends up "deleting half the files" with a force reinstall of an existing version that's in use.

antirotor · 2022-08-12T11:03:28Z

I think this is fixing it:

#3656

antirotor added 5 commits August 1, 2022 18:00

🚨 f-strings and cosmetic issues

bb10fdd

♻️ add openpype version env var to deadline job

a9f910a

♻️ handle multiple versions

0e126a2

♻️ changes in bootstrapping for multiple versions

9205d4b

♻️ deadline plugin support for job specific OP versions

de70521

antirotor added Bump Minor Pull requests that update a dependency file type: feature Larger, user affecting changes and completely new things labels Aug 1, 2022

antirotor self-assigned this Aug 1, 2022

antirotor added 3 commits August 2, 2022 11:26

♻️ skip non-existent local path

3d7e195

when finding local version, stop crashing if directory to search doesn't exist - this will allow to just use build version

🐛 filter out non-build versions

9ed329a

and fixing the error message

🚨 I hate you Hound so much

c64925f

antirotor requested review from m-u-r-p-h-y, iLLiCiTiT and 64qam August 3, 2022 14:59

antirotor added 5 commits August 4, 2022 11:07

♻️ change macos installer

3137644

🔨 add more verbose info to Deadline

633c7a5

🐛 fix inverted condition

b9703f3

🐛 fix executable detection on platforms

03c648c

🚨 unify output messages

53877eb

m-u-r-p-h-y suggested changes Aug 4, 2022

View reviewed changes

antirotor added 3 commits August 4, 2022 14:35

♻️ natural sort versions

097638c

🚨 fix invalid sequence warning

7de8c33

🚨 fix hound 🐶

52eba35

antirotor requested a review from m-u-r-p-h-y August 4, 2022 12:46

m-u-r-p-h-y approved these changes Aug 4, 2022

View reviewed changes

antirotor merged commit 278c8e9 into release/3.13.x Aug 4, 2022

antirotor deleted the feature/OP-3458_support-for-mutiple-build-versions_2 branch August 4, 2022 13:24

antirotor mentioned this pull request Aug 4, 2022

Deadline: support for version specific job #3416

Closed

This was referenced Aug 4, 2022

Deadline: add information about used OP version to Job #3439

Closed

Fix for multi-version build on Mac #3622

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for mutliple installed versions - 3.13 #3605

Support for mutliple installed versions - 3.13 #3605

antirotor commented Aug 1, 2022

ynbot commented Aug 1, 2022

antirotor commented Aug 1, 2022

m-u-r-p-h-y left a comment

m-u-r-p-h-y left a comment

BigRoy commented Aug 10, 2022

antirotor commented Aug 11, 2022

BigRoy commented Aug 11, 2022 •

edited

Loading

antirotor commented Aug 11, 2022

BigRoy commented Aug 11, 2022

BigRoy commented Aug 12, 2022

antirotor commented Aug 12, 2022

Support for mutliple installed versions - 3.13 #3605

Support for mutliple installed versions - 3.13 #3605

Conversation

antirotor commented Aug 1, 2022

Feature

Deployment Notes

Testing

Visibility of compatible versions

Deadline

Implementation steps:

Footnotes

ynbot commented Aug 1, 2022

antirotor commented Aug 1, 2022

m-u-r-p-h-y left a comment

Choose a reason for hiding this comment

m-u-r-p-h-y left a comment

Choose a reason for hiding this comment

BigRoy commented Aug 10, 2022

antirotor commented Aug 11, 2022

BigRoy commented Aug 11, 2022 • edited Loading

antirotor commented Aug 11, 2022

BigRoy commented Aug 11, 2022

BigRoy commented Aug 12, 2022

antirotor commented Aug 12, 2022

BigRoy commented Aug 11, 2022 •

edited

Loading