ptyprocess fails on OpenIndiana 151a9 and Python 2.7.11 when the process lacks a controlling terminal

[fazalmajid]

pexpect is failing in ptyprocess.spawn on OpenIndiana 151a9 (a distro of Illumos/OpenSolaris). pexpect is 4.1.0, ptyprocess is 0.5.1:

Traceback (most recent call last):
  File "/home/majid/bin/upload_file.py", line 26, in <module>
    upload(fn)
  File "/home/majid/bin/upload_file.py", line 7, in upload
    timeout=14400)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 198, in __init__
    self._spawn(command, args, preexec_fn, dimensions)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 298, in _spawn
    cwd=self.cwd, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 309, in _spawnpty
    return ptyprocess.PtyProcess.spawn(args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 223, in spawn
    pid, fd = _fork_pty.fork_pty()
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/_fork_pty.py", line 30, in fork_pty
    pty_make_controlling_tty(child_fd)
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/_fork_pty.py", line 76, in pty_make_controlling_tty
    fd = os.open("/dev/tty", os.O_WRONLY)
OSError: [Errno 6] No such device or address: '/dev/tty'

The script is run under the control of at(1), and the job itself is queued from a crontab. It looks like what is happening is that when the parent process, which has no controlling terminal, calls os.openpty(), it acquires the pty as a controlling terminal. The problem seems to occur when posix_openpty pushes the ptem STREAMS module onto the slave pty fd, the simple call ioctl(slave_fd, I_PUSH, "ptem") causes the parent process to acquire the pty as its controlling terminal, and thus the child, which is in a different session, is unable to acquire it and the failure at line 76 in _fork_pty().

I diffed posixmodule.c on Python 3.5 vs 2.7.11, the implementation of posix_openpty is not much different.

Older versions of pexpect had a different implementation _svr4_openpty(), but those look very similar to the Python posixmodule.c code.

[fazalmajid]

Here's a minimal reproduction case:

#!/usr/local/bin/python
import sys, os, time, pexpect

# fork and setsid to disassociate from the controlling terminal
x = os.fork()
if x != 0:
  sys.exit(0)
os.setsid()

s = pexpect.spawn('/usr/bin/cat')
s.sendline('foo')
s.expect('foo')
s.close()

if you comment out the first block (os.fork to os.setsid), it works. Otherwise, you get:

Traceback (most recent call last):
  File "test_pty.py", line 11, in <module>
    s = pexpect.spawn('/usr/bin/cat')
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 198, in __init__
    self._spawn(command, args, preexec_fn, dimensions)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 298, in _spawn
    cwd=self.cwd, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 309, in _spawnpty
    return ptyprocess.PtyProcess.spawn(args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 223, in spawn
    pid, fd = _fork_pty.fork_pty()
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/_fork_pty.py", line 30, in fork_pty
    pty_make_controlling_tty(child_fd)
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/_fork_pty.py", line 76, in pty_make_controlling_tty
    fd = os.open("/dev/tty", os.O_WRONLY)
OSError: [Errno 6] No such device or address: '/dev/tty'

[fazalmajid]

My workaround is to issue os.openpty() just before the call to pexpect.spawn() so the parent process gets a bogus controlling terminal and does not attempt to steal its' child's, but that is obviously quite ugly.

[fazalmajid]

Opened an issue with Illumos: https://www.illumos.org/issues/6995
Not sure if this occurs in Oracle's Solaris.

[takluyver]

It sounds like the issue is occuring inside the call to openpty(), and I don't think there's a lower-level call we can do from Python, unless we do stuff with ctypes, which I'd rather avoid.

[jquast]

Hello,

I'm familiar with Illumos/OpenSolaris, and have attempt to provide build stability for this OS in the past -- the cost of joyent containers was too high to maintain for the long term. If only travis-ci could provide OpenSolaris support :)

The core Python project itself has similar issue, they are not supporting Solaris very well any longer due to lack of build infrastructure and core contributors with solaris experience. From what I recall about stagnated pty.fork() fixes in the bug tracker, there is not interest in resolving pty.fork() for Solaris, they do not have the resources to test it.

This issue smells very familiar to a previous one:

pexpect/pexpect#44

Which from the various related PR's and Issues, we thought was addressed.

Just a memory dump:

• Standard python's pty.fork() is no good on Solaris. Works fine on other OS's.
• And for this reason, ptyprocess provides this custom wrapper conditional executed only for Solaris.
• I attempted to use ctypes to use the C library fork routines to correct this, as pty C libraries work perfectly fine on Solaris, only Python's os.fork() is not correct for it.
• Unfortunately, this also failed, because ctypes were broken on SmartOS at the time, I submitted bugfix which was never accepted, http://bugs.python.org/issue20664

(edit: i kept saying os.fork() but meant pty.fork() of course!)

[fazalmajid]

@jquast
I work around the ctypes issues in my build process for Python using:

# ctypes.util.find_library() uses the 32-bit instead of 64-bit crle path
        gsed -i -e 's/is64 = False/is64 = True/g' Python-$(PY_VER)/Lib/ctypes/util.py
# ctypes.util.find_library uses /usr/ccs/bin/dump on OpenIndiana
# or /usr/bin/dump on SmartOS
ifeq ($(DUMP_EXE),/usr/bin/dump)
        gsed -i -e 's@/usr/ccs/bin/dump@/usr/bin/dump@g' Python-$(PY_VER)/Lib/ctypes/util.py
endif

What are the "pty C libraries" that work perfectly fine on Solaris you are referring to? I could look into how they avoid the issue os.openpty() is experiencing and see if that could be backported into Python.

[dimpase]

As we are trying to resurrect Solaris port of Sagemath, more precisely a SPARC Solaris 11 --- it used to work back in 2011 or so--- it appears that pexpect/ptyprocess is a stumbling block. We see at lot of setecho() may not be called on this platform. errors in tests.

How hard would be to work around this pexpect limitation? (We could patch Python if needed...)

[jdemeyer]

@dimpase The setecho() issue looks unrelated to this ticket. It seems that Solaris simply does not support (un)setting terminal echo from the master process after the child is started. It is only possible to (un)set terminal echo when starting the child. In pexpect, this is using echo=False when initializing pexpect.spawn.

[dimpase]

Perhaps a better error message would be to add something like Use echo=... while initializing pexpect.spawn.

[livelace]

We have the same problem (/dev/tty) on Solaris 10 :(
Any decision ?

[dimpase]

basically, we just don't call setecho() once the child is started any more. One can set echo at the startup; this is supported by pexpect, and works for us on Solaris 11.

[livelace]

@dimpase

s = pexpect.spawn('/usr/bin/cat', echo=False)
s.sendline('foo')
s.expect('foo')
s.close()

Have you meant "echo=False" ? If so - it doesn't work on Solaris 10.

[dimpase]

For the latter, I see echo both on Linux and on Solaris 11. So it is trickier than this anyway. Perhaps @jdemeyer - who wrote the Solaris fix here, could comment.

[jdemeyer]

We have the same problem (/dev/tty) on Solaris 10 :(
Any decision ?

Can you please state exactly which problem you have.

[jdemeyer]

it doesn't work

See https://www.chiark.greenend.org.uk/~sgtatham/bugs.html

[livelace]

@jdemeyer

I have exactly the same problem as @fazalmajid had and his example gives the same result on Solaris 10 (it can be reproduced in any time).

"OSError: [Errno 6] No such device or address: '/dev/tty'"

That is why I wrote in brackets "/dev/tty".

• I'm trying to find out how it is at the current time (bug was opened in 2016). If it's hard to fix (nobody wants, Solaris 10!), then ... I could make a workaround by myself.

[jdemeyer]

Traceback please...

[jdemeyer]

it can be reproduced in any time

If you have access to a Solaris 10 system, which I do not have. If you want me to help you, then you will need to give more information.

[jdemeyer]

Sorry for the mess. This ticket is about two things and I got totally confused. Because of the comments by @dimpase I was thinking that it was about setecho() which is totally not the case.