"filedescriptor out of range in select()" error in _ready_for_reading

[QuiteClose]

On a machine with more than 1024 open file-descriptors, the _ready_for_reading function (eventloop/posix.py) will always raise a "ValueError: filedescriptor out of range in select()".

This is because of the use of the select.select function to poll file-descriptors. A hard-limit of 1024 on any file-descriptor passed to select.select is built into Python at compile-time using the FD_SETSIZE value. On most platforms, this is 1024. An administrator may allow more file-descriptors to be opened on busy machines making the select function unreliable in posix environments.

On posix environments, the select.poll function will provide the same functionality as select in this context. It is also not limited by the FD_SETSIZE value. May I recommend switching to this function on posix environments or otherwise catching the ValueError in your _select wrapper?

This is currently preventing us from using IPython v5, as we often use machines with more than 1024 open file-descriptors. Below is a traceback from one such IPython session:

In [1]: Exception in thread Thread-97:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/prompt_toolkit/eventloop/inputhook.py", line 66, in thread
    input_is_ready_func(wait=True)
  File "/usr/local/lib/python3.5/dist-packages/prompt_toolkit/eventloop/posix.py", line 101, in ready
    return self._ready_for_reading(current_timeout[0] if wait else 0) != []
  File "/usr/local/lib/python3.5/dist-packages/prompt_toolkit/eventloop/posix.py", line 184, in _ready_for_reading
    r, _, _ =_select(read_fds, [], [], timeout)
  File "/usr/local/lib/python3.5/dist-packages/prompt_toolkit/eventloop/posix.py", line 278, in _select
    return select.select(*args, **kwargs)
ValueError: filedescriptor out of range in select()

[jonathanslenders]

Thanks for creating this issue! I really had no idea this would happen.

Probably, I can fall back to epoll or poll if ValueError is raised. Which one is better?

[takluyver]

AIUI, the main difference is in which platforms each function exists on. select is pretty universal, poll is widespread but not universal, epoll is Linux specific and kqueue is BSD specific. More info here:
https://docs.python.org/3/library/select.html

On Python 3, the selectors module provides a high-level interface which will automatically select the best available mechanism.

[jonathanslenders]

Hi @dsclose,

Normally this should be fixed by now. Can you try, using the latest commit from the master branch?

Thanks!

[QuiteClose]

Hi @jonathanslenders, well done on sorting that - I'll do my best to check that it is working over the next couple of days for you.

[QuiteClose]

@jonathanslenders looks like commit c3c994a has resolved this issue. Verified with the following steps:

1. Installed iPython 5.0 (which includes prompt-toolkit 1.0.3)
2. Opened over 1024 sockets to ramp up the file-descriptor count.
3. Launched an iPython console - crash occurred as expected.
4. Uninstalled prompt-toolkit and installed latest master branch from github.
5. Launched an iPython console, no crash occurred.

We'll continue to use iPython 4 until this commit gets into PyPi - thanks for your swift response on this!

[QuiteClose]

As of Release 1.0.4 I'm marking this issue as closed.