Unity process crashing silently on headless Ubuntu 18.04

[jzhanson]

EDIT: I think the command line Unity process working fine earlier was because I was running it without arguments — if I run it with the same arguments in the Python code it crashes silently as well. The issue text is updated, and I'm linking this issue's sibling issue which has more to do with the ALFRED code.

When running example_agent.py in the Docker container, it seems like the Unity process crashes immediately without giving any information on why. If I run the Unity executable by itself in the container's bash shell, it also crashes, e.g. if I run $ /root/.ai2thor/releases/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240/thor-Linux64- 8db5080010a07f037367ad6be0fd83d8f5f75240 -screen-fullscreen 0 -screen-quality 7 -screen-width 300 - screen-height 300 it prints out Found path: /root/.ai2thor/releases/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240 and then crashes

Below is the output from example_agent.py

X.Org X Server 1.19.6
Release Date: 2017-12-20
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.15.0-115-generic x86_64 Ubuntu
Current Operating System: Linux 7ec1dfd17635 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=UUID=fd242869-5ed6-4010-a7c4-9171df38a426 ro quiet splash vt.handoff=1
Build Date: 04 September 2020  03:34:39PM
xorg-server 2:1.19.6-1ubuntu4.6 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.34.0
        Before reporting problems, check http://wiki.x.org
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sun Sep 27 13:39:10 2020
(++) Using config file: "/tmp/tmpjabi_9yd"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
PlayerPrefs - Creating folder: /root/.config/unity3d/unknown
PlayerPrefs - Creating folder: /root/.config/unity3d/unknown/unknown
Unable to load player prefs
Found path: /root/.ai2thor/releases/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ai2thor/wsgi_server.py", line 39, in queue_get
    res = que.get(block=True, timeout=0.5)
  File "/usr/lib/python3.6/queue.py", line 172, in get
    raise Empty
queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example_agent.py", line 8, in <module>
    controller = ai2thor.controller.Controller(scene='FloorPlan28')
  File "/usr/local/lib/python3.6/dist-packages/ai2thor/controller.py", line 426, in __init__
    host=host
  File "/usr/local/lib/python3.6/dist-packages/ai2thor/controller.py", line 929, in start
    self.last_event = self.server.receive()
  File "/usr/local/lib/python3.6/dist-packages/ai2thor/wsgi_server.py", line 212, in receive
    return queue_get(self.request_queue, self.unity_proc)
  File "/usr/local/lib/python3.6/dist-packages/ai2thor/wsgi_server.py", line 45, in queue_get
    raise Exception("Unity process exited %s" % unity_proc.returncode)
Exception: Unity process exited 1

and the Unity prefs at /root/.config/unity3d/unknown/unknown/prefs read

<unity_prefs version_major="1" version_minor="1">
        <pref name="Screenmanager Fullscreen mode" type="int">3</pref>
</unity_prefs>

while the Player.log at /root/.config/unity3d/Allen\ Institute\ for\ Artificial\ Intelligence/AI2-Thor/Player.log reads

Mono path[0] = '/root/.ai2thor/releases/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240_Data/Managed'
Mono config path = '/root/.ai2thor/releases/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240/thor-Linux64-8db5080010a07f037367ad6be0fd83d8f5f75240_Data/MonoBleedingEdge/etc'
Display 0 'Smart Cable': 1024x768 (primary device).
Desktop is 1024 x 768 @ 60 Hz
Unable to find a supported OpenGL core profile
Failed to create valid graphics context: please ensure you meet the minimum requirements
E.g. OpenGL core profile 3.2 or later for OpenGL Core renderer
Vulkan detection: 0
No supported renderers found, exiting
(Filename:  Line: 618)

The Docker container is built with CUDA Version: 11.0, NVIDIA Driver: 450.57. With the startx.py script running inside the docker container and setting export DISPLAY=:0 I get from $ glxgears | grep version that OpenGL core profile version string: 4.6.0 NVIDIA 450.57 and the like for OpenGL shading language. It does say

server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4

for what it's worth.

Additional information

I'm trying to get headless running — some of the other change I've made are adding RUN pip install --upgrade pip to the Dockerfile before RUN pip3 install -r requirements.txt && ... as advised by the second FAQ on the opencv-python pypl page because I was running into the error

...
ModuleNotFoundError: No module named 'skbuild'

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-pniw64kt/opencv-python/
The command '/bin/sh -c pip3 install -r requirements.txt && python3 -c "import ai2thor.controller;   ai2thor.controller.Controller(download_only=True)"' returned a non-zero code: 1

I also added libgl1-mesa-glx and mesa-utils to the apt-get call, but that was mostly for running the startx.py script and glxinfo.

Also, there is one /usr/lib/xorg/Xorg process already running on this cluster machine — one process, on the four GPUs — which means that my Xorg process from startx.py is the second extant one — its parent process is gdm-xsession, and its great-grandparent process is gdm3, so I'm pretty sure that has to do with the desktop GUI <-> GPU interface.

[NoodlesAwaken]

Any luck? Same issue here

[aishwaryap]

Might be having the same issue.

Follow up question, for either of you does the unity crash result in the Xorg process also crashing (i.e. ps -aux | grep Xorg before the crash shows an Xorg process but ps -aux | grep Xorg shows no Xorg process)?