Pyblish QML failing to load from Maya after loading Houdin Engine plug-in

[BigRoy]

Issue

Under Windows 10 it seems that whenever the current Maya session has loaded the Houdini Engine plug-in that Pyblish QML fails to run afterwards. This is due to a DLL error on loading PyQt5. Even after unloading the Houdini Engine plug-in the problem remains for that Maya session, whenever Houdini has been active Pyblish QML will fail to run.

# Running in Maya after Houdini Engine loaded...
import pyblish_qml
pyblish_qml.show()

# Output below:
Using Python @ 'D:/test/python36\python.exe'
Using PyQt5 @ 'None'
Targets: default
Success. QML server available as pyblish_qml.api.current_server()
Event filter has been installed.
  File "D:\test\python36\lib\runpy.py", line 193, in _run_module_as_main
Traceback (most recent call last):
ImportError: DLL load failed: The specified procedure could not be found.
    from PyQt5 import QtCore, QtGui, QtQuick, QtTest
  File "D:\git_pipeline\avalon\development\git\pyblish-qml\pyblish_qml\app.py", line 12, in <module>
    from . import app
  File "D:\git_pipeline\avalon\development\git\pyblish-qml\pyblish_qml\__main__.py", line 6, in <module>
    exec(code, run_globals)
  File "D:\test\python36\lib\runpy.py", line 85, in _run_code
    "__main__", mod_spec)

To reproduce

Reproducible on: Windows 10 + Maya 2018.5 + Houdini 17.0.416

We have been unable to reproduce the error on Windows 7 with the same Maya and Houdini - thus it seems to be related to something in Windows 10!

The basis of the error is also reproducible by doing:

import subprocess
subprocess.Popen("python")

In the opened python console (assuming that's also the Python interpreted used to run the QML in background) if you just do:

from PyQt5 import QtWidgets
# DLL error

What I've tried..

Trying to strip the environment of anything related to Houdini/Qt/Maya as I run the subprocess command that reproduces the DLL error. Didn't work for me.

Here's some example code I tried:

import os
import subprocess
from subprocess import PIPE
import pprint

# Clean the environment for the subprocess to be safe... (doesn't help)
# Note that loading Houdini Engine does not change the Maya environment (before/after is exactly the same, perfect match)
env = os.environ.copy()
for key in env.copy():
    if any(key.lower().startswith(x) for x in ["qt", "maya", "houdini", "vray", "xgen", "redshift", "pyblish", "mtoa", "bifrost", "avalon", "atf", "arnold", "adsk", "colorbleed", "deadline", "substance", "pymel", "mash", "fbx", "ffmpeg", "com", "session"]):
        env.pop(key)
        continue
    
    value = env[key]
    for name in ["maya", "houdini", "python27", "mtoa"]:
        if name in value.lower():
            paths = value.split(";")
            paths = [p for p in paths if name not in p.lower()]
            value = ";".join(paths)
            env[key] = value
        
# Remove some random variables if they are there - just to be sure and keep as clean as possible env
env.pop("__APP__", None)
env.pop("__EXE__", None)
env.pop("__COMPAT_LAYER", None)
env.pop("PROMPT", None)
env.pop("ONEDRIVE", None)
env.pop("SCHEMA", None)

# Debug print the environment
pprint.pprint(env)

# Running it with this environment, doesn't work. 
subprocess.Popen("python", env=env)

# Or (other test) 
# Even if I overkill detach the process and run Python through shell - same issue.
# Using Process Explorer this way it's not a 'child' of the original Maya process
# so I thought maybe that would behave different, but didn't fix it either.
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
subprocess.Popen("python",
                 env=env, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE,
                 creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)

Or instead of stripping from current environment, just take the SYSTEMROOT (so Python can actually run)

import os
import subprocess
from subprocess import PIPE
import pprint

# Take only speciifc environment keys
env = {}
env["SYSTEMROOT"] = os.environ["SYSTEMROOT"]
env["PATH"] = ""  # need at least empty PATH in environment for PyQt5 to import (see __init__.py of PyQt5)
    
# Debug print the environment
pprint.pprint(env)

# Running it with this environment, doesn't work. 
subprocess.Popen("python", env=env)

# Or (other test) 
# Even if I overkill detach the process and run Python through shell - same issue.
# Using Process Explorer this way it's not a 'child' of the original Maya process
# so I thought maybe that would behave different, but didn't fix it either.
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
subprocess.Popen("python",
                 env=env, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE,
                 creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)

# Or using close_fds = True
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
subprocess.Popen("python",
                 env=env,
                 close_fds=True,
                 creationflags=CREATE_NEW_PROCESS_GROUP)

Again, no luck.

Whatever does fix it (manually)

However! when I open Process Explorer, right click the python.exe process and press "Restart" it reopens the Python with the exact same environment... and voila, PyQt5 can import correctly.

It seems as if somehow Windows 10 is trying to use the DLL from Memory that was loaded in connection with the Houdini (in Maya) process making it fail to load the actual correct one for the Python process.

Any ideas on how this could be resolved?

---

Reference discussion on Pyblish Gitter:

• https://gitter.im/pyblish/pyblish?at=5c3382c9aac7082e6feb1117
• https://gitter.im/pyblish/pyblish?at=5c2c88cadb5b5c68834396ef
• https://gitter.im/pyblish/pyblish?at=5c34adf7357dd7655d377726

Thanks to @davidlatwe for also confirming the issue on Windows 10 and being unable to reproduce it on Windows 7. Appreciate the time you took to investigate, cheers!

[mottosso]

Thanks for the detailed summary @BigRoy, this seems like a hard one to debug!

I think step one might be to confirm that it does happen on another machine; can you confirm?

[BigRoy]

I think step one might be to confirm that it does happen on another machine; can you confirm?

I reproduced the error on multiple Windows 10 machines here - and @davidlatwe was able to reproduce on Windows 10 (though he was unable to reproduce on Windows 7 - I am unable to test that myself though.).

[mottosso]

Keeping conversations in sync.

• https://gitter.im/pyblish/pyblish?at=5c35dbcf1491b27876e5033c

[BigRoy]

We're still facing this issue whenever Houdini Engine is loaded in Maya (or has been loaded at least once during the current Maya session). It's odd behavior but I have no clue how to work towards a fix.

Anyone else who is able to reproduce this or has some additional pointers on how to debug/resolve this? It is reproducable on multiple machines as stated above.

[davidlatwe]

I have encountered #301 today, I was using Nuke 11.3 (comercial version) and I believe this issue is related to #301.

And what I have found were pretty much like what @BigRoy had above. It's really hard to solve starting from subprocess.

Then I thought, what if we change PyQt5 into Qt5.py ? So I get PySide2 installed in my environment, and Yes ! the DLL load fail error was gone and QML window pops-up. But it crashed pretty soon since all the code was written for PyQt5 which wasn't easy to tweak.

So before going deeper, what do you think of changing PyQt5 into Qt5.py ?

[mottosso]

Sounds good. The only reason PySide2 wasn't supported from the start was because the QML support there wasn't finished. I haven't kept up to date with it, but I would imagine it's in a good position to switch now.

The main difference should be those pyqtSignal, and there are probably some other minor PyQt-specific things in there. But other than that, it should be good to go.

Once there, it'll also enable use of mayapy and friends as the external Python interpreter, in place of the one you install "manually". Would simplify distribution significantly, as it already comes with PySide2 pre-installed. 👍