[Linux] psutil does not handle ENAMETOOLONG when accessing process file descriptors in procfs

[nradchenko]

Summary

• OS: Linux
• Architecture: 64bit
• Psutil version: 5.8.0
• Python version: 3.7.3
• Type: core

Description

Recently I've discovered that net_connections() fails sometimes on our storage servers. These hosts have a lot of files which are processed by rsync almost 24/7. The error was like this: OS error: [Errno 36] File name too long: '/proc/5259/fd/12.

After some digging it turned out that we have a number of files stored at deep filesystem directory levels, which are occasionally opened by rsync for reading or writing. Paths of such files may exceed PATH_MAX, which results in broken magic symlinks in procfs.

The bug can be reproduced with the following steps:

1. Create and open a file with a path longer than PATH_MAX (4096 usually):

$ d=$(python -c "print('d'*255)"); for i in {1..17}; do mkdir -p $d; cd $d; done; cat > test

Press Ctrl+Z or open a new terminal window at this point.

2. Call net_connections() and see what happens:

$ python3 -c 'import psutil; psutil.net_connections()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/nik/.local/lib/python3.7/site-packages/psutil/__init__.py", line 2153, in net_connections
    return _psplatform.net_connections(kind)
  File "/home/nik/.local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1027, in net_connections
    return _connections.retrieve(kind)
  File "/home/nik/.local/lib/python3.7/site-packages/psutil/_pslinux.py", line 1001, in retrieve
    inodes = self.get_all_inodes()
  File "/home/nik/.local/lib/python3.7/site-packages/psutil/_pslinux.py", line 849, in get_all_inodes
    inodes.update(self.get_proc_inodes(pid))
  File "/home/nik/.local/lib/python3.7/site-packages/psutil/_pslinux.py", line 827, in get_proc_inodes
    inode = readlink("%s/%s/fd/%s" % (self._procfs_path, pid, fd))
  File "/home/nik/.local/lib/python3.7/site-packages/psutil/_pslinux.py", line 211, in readlink
    path = os.readlink(path)
OSError: [Errno 36] File name too long: '/proc/12358/fd/1'
$ 

Let's inspect the directory and see what the problem with this symlink is:

$ ll /proc/12358/fd/
ls: cannot read symbolic link '/proc/12358/fd/1': File name too long
total 0
dr-x------ 2 nik nik  0 Apr 28 17:54 ./
dr-xr-xr-x 9 nik nik  0 Apr 28 17:54 ../
lrwx------ 1 nik nik 64 Apr 28 17:54 0 -> /dev/pts/18
l-wx------ 1 nik nik 64 Apr 28 17:54 1
lrwx------ 1 nik nik 64 Apr 28 17:54 2 -> /dev/pts/18
$ stat /proc/12358/fd/1
  File: /proc/12358/fd/1stat: cannot read symbolic link '/proc/12358/fd/1': File name too long

  Size: 64              Blocks: 0          IO Block: 1024   symbolic link
Device: 15h/21d Inode: 13911955    Links: 1
Access: (0300/l-wx------)  Uid: ( 1000/     nik)   Gid: ( 1000/     nik)
Access: 2021-04-28 17:58:18.920517996 +0300
Modify: 2021-04-28 17:54:23.553843692 +0300
Change: 2021-04-28 17:54:23.553843692 +0300
 Birth: -
$ readlink /proc/12358/fd/1
$ strace -e readlink readlink /proc/12358/fd/1
readlink("/proc/12358/fd/1", 0x56250042c400, 64) = -1 ENAMETOOLONG (File name too long)
+++ exited with 1 +++
$ 

My tests show that there seems to be no way to find out where this magic symlink points at all, even when calling readlink with large buffer:

readlink("/proc/12358/fd/1", 0x55d7cd489260, 40960) = -1 ENAMETOOLONG (File name too long)

How to fix

I suggest to ignore this kind of errors.

Skipping it in net_connections() is fine because these broken fds are not needed for preparing network connections list - netstat and ss do not report any errors like that.

And skipping it in psutil.Process.open_files() should be fine too, as there's nothing else we can actually do to fix this in userspace.

[nradchenko]

I'm going to prepare PR with a fix

[giampaolo]

Interesting. I agree we should ignore ENAMETOOLONG and continue both in open_files and net_connections.