Leak of Mach ports on OS X

[giampaolo]

From rsesek@google.com on September 27, 2012 23:11:02

What steps will reproduce the problem?  
I noticed on a machine that uses psutil in a long-lived process, Mach ports 
seem to run low or get completely exhausted. Using DTrace, I was able to verify 
that the psutil-based process was in fact using a large number of Mach ports, 
and it appeared to be leaking them. I examined _psutil_osx.c and found that 
Mach port rights were not being properly managed. 

What is the expected output?  
Mach ports should not be leaked. 

What do you see instead?  
Mach ports getting leaked. 

What version of psutil are you using? What Python version?  
Trunk. 

On what operating system? Is it 32bit or 64bit version?  
Mac 32 and 64-bit. 

Please provide any additional information below.  
The justification for these changes are:

- mach_host_self() traps into the kernel via host_self_trap() in 
xnu-2050.7.9/osfmk/kern/ipc_host.c. The subroutine's comments state: "Give the 
caller send rights for his own host port.". This is verified with the 
documentation at xnu-2050.7.9/osfmk/man/mach_host_self.html. The kernel creates 
a copy of the send right, which means that right must be deallocated.

- task_for_pid() returns the Mach task port for a given process. This is leaked 
in two locations and also must be deallocated. The only time a Mach call that 
vends a send right that does not need to be manually deallocated is 
mach_task_self(), which is defined as a macro to a global variable for 
performance in /usr/include/mach/mach_init.h.

The attached patch modifies the OS X C implementation of psutil to properly 
manage port rights. I was not able to figure out a way to run the test suite. 
Please let me know how to do that.

Attachment: mach_port_leak_fix.patch

Original issue: http://code.google.com/p/psutil/issues/detail?id=333

[giampaolo]

From jlo...@gmail.com on September 27, 2012 14:21:11

Thanks for the report, we obviously were not aware that the ports had to be 
managed and deallocated after use. I've reviewed and merged the patch in r1543 
to roll this into a future release.

To run the test suite it's just python test/test_psutil.py and 
test/test_memory_leaks.py which just checks memory consumption following 
consecutive runs of psutil methods. It would be good for the future if we can 
develop a test to check for Mach port leaks as well to catch similar issues in 
the test suite.

Status: FixedInSVN

[giampaolo]

From rsesek@google.com on September 27, 2012 15:15:58

Thanks for the quick turnaround!

[giampaolo]

From g.rodola on September 27, 2012 15:19:39

Out of curiosity, is psutil used internally by Google and for what?

[giampaolo]

From rsesek@google.com on October 01, 2012 12:51:46

It is available internally at Google, but I cannot say for what it is used.

[giampaolo]

From g.rodola on November 18, 2012 10:41:14

Labels: Milestone-0.6.2

[giampaolo]

From g.rodola on March 02, 2013 04:13:24

Updated csets after the SVN -> Mercurial migration: r1543 == revision 
9d625f59b839

[giampaolo]

From g.rodola on April 07, 2013 18:21:49

Status: FixedInHG

[giampaolo]

From g.rodola on April 11, 2013 02:17:39

Labels: -Milestone-0.6.2 Milestone-0.7.0

[giampaolo]

From g.rodola on April 12, 2013 11:21:26

Status: Fixed

[holybin]

"Using DTrace, I was able to verify
that the psutil-based process was in fact using a large number of Mach ports,
and it appeared to be leaking them"

@giampaolo
Hello , how did you use DTrace to verify mach port leaks? Can you show me the usage or some details?
thanks!