psutil doesn't know about several connection types used by ZMQ

[giampaolo]

From tiran79 on June 15, 2012 13:55:44

What steps will reproduce the problem?  
See attached script. The output on my system is:

$ python psutil_fd.py 
startup
0 /dev/pts/1
1 /dev/pts/1
2 /dev/pts/1
3 None
9 /proc/2777/auxv
get_open_files [openfile(path='/proc/2777/auxv', fd=9)]
get_connections []

---

context
0 /dev/pts/1
1 /dev/pts/1
2 /dev/pts/1
3 socket:[24580675]
4 socket:[24580676]
5 socket:[24580677]
6 socket:[24580678]
7 anon_inode:[eventpoll]
8 socket:[24580679]
9 /proc/2777/auxv
10 socket:[24580680]
11 anon_inode:[eventpoll]
12 socket:[24580681]
13 socket:[24580682]
14 None
get_open_files [openfile(path='/proc/2777/auxv', fd=9)]
get_connections []

---

socket
0 /dev/pts/1
1 /dev/pts/1
2 /dev/pts/1
3 socket:[24580675]
4 socket:[24580676]
5 socket:[24580677]
6 socket:[24580678]
7 anon_inode:[eventpoll]
8 socket:[24580679]
9 /proc/2777/auxv
10 socket:[24580680]
11 anon_inode:[eventpoll]
12 socket:[24580681]
13 socket:[24580682]
14 socket:[24580700]
15 socket:[24580701]
16 None
get_open_files [openfile(path='/proc/2777/auxv', fd=9)]
get_connections []
--- 

What is the expected output?  
I expected that psutil has some means to list the connections used by ZeroMQ. 
For example 24601841 is listed in /proc/net/unix as 

Num       RefCount Protocol Flags    Type St Inode Path
0000000000000000: 00000003 00000000 00000000 0001 03 24601841

Attachment: psutil_fd.py

Original issue: http://code.google.com/p/psutil/issues/detail?id=285

[giampaolo]

From g.rodola on June 17, 2012 14:42:10

Is this report about get_connections() not reporting UNIX-socket connections?
In that case that's already tracked in issue 216 .

[giampaolo]

From tiran79 on June 27, 2012 15:06:04

I didn't know about #216. That tackles the unix domain socket issue.

It's a report about psutil not showing fds it doesn't understand. For example 
get_open_files() ignores all files that are not regular files: named pipes, 
block devices, char devices. The check os.path.isfile() in 
_pslinux.Process.get_open_files() skips the char device /dev/pts/1. 
os.path.exists() should be sufficient.

I'm not sure how to handle anon_inode:[eventpoll]. The FD is created by the 
epoll() syscall.

[giampaolo]

From g.rodola on June 28, 2012 04:58:46

get_open_files() is supposed to return *regular* files only, hence the 
os.path.isfile() check.
/dev/pts/1 is a terminal fd so it's ok to skip it.
If we want to return information about *all* fds that should be done as a 
separate get_open_fds() method mimicking lsof but problem is how to represent 
the data and how to conciliate the differences between UNIX platforms, which, 
IMO, it's likely to be impossible.

[giampaolo]

From tiran79 on July 01, 2012 08:27:25

IMHO the limitation is too strict and removes useful information from the 
output. All this files have a fs entry but are not listed

* open directories used by the newly introduced *at() functions like openat()
* char and block devices
* named fifos
* sockets

Deleted are also skipped although they are still opened. Deleted files aren't 
correctly handles by psutil, too. Consider the case fd = open(name), 
unlink(name), touch(name). psutil returns true for fd but the fd refers to a 
different file. You have to compare st_dev and st_ino of fstat(fd) with 
stat(name) to detect the case.

Proposal:
* modify signature of the function to get_open_files(kind='REG', deleted=False)
* modify returned named tuple to contain kind (REG, DIR, SOCK, BLK, CHR, FIFO) 
and deleted (bool) flag

lsof and kinfo_getfile return the file type information.

I can create a patch if you like my idea.

[giampaolo]

From g.rodola on July 01, 2012 17:54:33

Point is how to represent the different fd types, which brings to the question: 
what 'fields' the returning namedtuple should provide?
Judging from lsof output we should provide:

FD, TYPE, DEVICE, SIZE/OFF, NODE, NAME

...but if you look at hot lsof output is represented you'll notice how 
depending on the fd type it examines, certain fields are sometimes represented 
with a number, other times with a string.
That alone suggests that this is a feature which is hard to translate in a 
consistent and usable API, and I'm not even mentioning the differences between 
UNIX flavors.

> All this files have a fs entry but are not listed
> [...]
> * sockets

IMO, sockets are fine being treated as a different entity, as we're currently 
doing with get_connections(), mostly because they have a local and remote 
address field, which is not present in other fd types.

Feel free to attempt to write a patch for Linux if you want, but IMO pretty 
soon you'll realize how difficult it is to come up with a consistent API to 
express the data you pull out from the OS.

[giampaolo]

From g.rodola on December 13, 2012 06:38:09

Closing this one as won't fix.
Feel free to reopen it if you can come up with a consistent proposal addressing 
the problems outlined above.

Status: WontFix
Labels: -OpSys-Linux OpSys-UNIX