Error when retrieving net connections

[sylvainduchesne]

This PR #880 causes an issue when connection count can vary during the probe.
I have tracked down the problem to the fact that the GIL is now being released when calling the windows api. For some reason, when the GIL is released, the windows api sometimes returns ERROR_NOT_ENOUGH_MEMORY. My only guess is that perhaps it doesn't like having socket changes during the call? It is hard to say for sure since the error is undocumented in the windows api.
In any case, calling
psutil.Process().connections(kind="tcp4")

can sometimes result in

  File "d:\open_source\psutil\psutil\__init__.py", line 1153, in connections
    return self._proc.connections(kind)
  File "d:\open_source\psutil\psutil\_pswindows.py", line 635, in wrapper
    return fun(self, *args, **kwargs)
  File "d:\open_source\psutil\psutil\_pswindows.py", line 917, in connections
    return net_connections(kind, _pid=self.pid)
  File "d:\open_source\psutil\psutil\_pswindows.py", line 325, in net_connections
    rawlist = cext.net_connections(_pid, families, types)
MemoryError

or

  File "d:\open_source\psutil\psutil\__init__.py", line 1153, in connections
    return self._proc.connections(kind)
  File "d:\open_source\psutil\psutil\_pswindows.py", line 635, in wrapper
    return fun(self, *args, **kwargs)
  File "d:\open_source\psutil\psutil\_pswindows.py", line 917, in connections
    return net_connections(kind, _pid=self.pid)
  File "d:\open_source\psutil\psutil\_pswindows.py", line 325, in net_connections
    rawlist = cext.net_connections(_pid, families, types)
WindowsError: [Error -1073741823] Windows Error 0xC0000001

I've attached the (somewhat messy) test to reproduce the issue.
test_psutil.py.txt

I understand the reasoning behind releasing the GIL in the loop (and you had mentioned it in the PR), so not sure how you wanted to handle this.

Thanks

[giampaolo]

Uhm... I cannot reproduce the issue by using your script.

[sylvainduchesne]

hrm, thats a shame.
took me a while to determine the GIL (or the absence of) was at fault.
If this helps:
OS Name: Microsoft Windows 10 Entreprise
OS Version: 10.0.14393 N/A Build 14393
although, I don't think the issue is related the windows version, as we have various production servers that have the same issue.
But without fault, I get one of the 2 errors within seconds of launching the script. If I remove the GIL release, no more issue.
What do you suggest?

[giampaolo]

What do you mean that the GIL is at fault?

As for the error you're seeing, it almost certainly originates from here:

psutil/psutil/_psutil_windows.c

Lines 1544 to 1548 in ac9dcca

	*data = malloc(*size);
	if (*data == NULL) {
	error = ERROR_NOT_ENOUGH_MEMORY;
	continue;
	}

And this is where it's translated into a MemoryError exception:

psutil/psutil/_psutil_windows.c

Lines 1707 to 1712 in ac9dcca

	error = __GetExtendedTcpTable(getExtendedTcpTable,
	AF_INET, &table, &tableSize);
	if (error == ERROR_NOT_ENOUGH_MEMORY) {
	PyErr_NoMemory();
	goto error;
	}

Since it's malloc() which fails I suppose that means you run out of physical memory. How much RAM does your system have and how many connections were reported by psutil.net_connections(kind='all') before it failed? You can try something like this:

try:
    print(len(psutil.net_connections(kind='all')))
except MemoryError:
    print(psutil.virtual_memory())

To conduct test I recommend psutil.net_connections(kind='all') instead of psutil.Process().connections(kind="tcp4") (both use the same C function anyway).

[sylvainduchesne]

Thanks for the reply.
Total Physical Memory: 65,434 MB
I tried with your suggestions, but the issue seems to be unrelated to actually memory usage. I had also suspected that prior to opening the issue, so I was monitoring the memory as the test was running, but memory usage remains quite low.
In any case, no MemoryError exception is raised (memory usage quite low). I decided to keep track of the max number of connections returned by the api before the error occurs, and it is also not very high, roughly 4000 connections.

As for the GIL, what I mean is that if I remove
https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_windows.c#L1539
https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_windows.c#L1556
the problem no longer occurs. So, keeping the GIL fixes the problem. I haven't tried, but I assume that even prior to this change, releasing the GIL would give similar results.

[giampaolo]

Do you have the possibility to put printf() statements in the C code and recompile with VS in order to understand where exactly the error originates from?
According to what you wrote we can incur into:

WindowsError: [Error -1073741823] Windows Error 0xC0000001
MemoryError

...both originating from here:

  File "d:\open_source\psutil\psutil\_pswindows.py", line 325, in net_connections
    rawlist = cext.net_connections(_pid, families, types)

[sylvainduchesne]

Hi,
Sorry, I didn't get an email for your last comment. Yes I can definitely add printf() statements, so let me know where.
I did some debugging prior to opening the issue to make sure it was something we couldn't solve without modifying psutil. Basically, it boils down to calling GetExtendedUdpTable (or TCP equivalent) can sometimes fail with either error codes.

WindowsError: [Error -1073741823] Windows Error 0xC0000001
ERROR_NOT_ENOUGH_MEMORY

I could probably reproduce the issue without psutil, simply calling the windows api in a c extension would most likely yield the same results.
Bottomline: the errors don't occur when GIL is kept.

[giampaolo]

Thank you.

[sylvainduchesne]

Thanks for merging :)
Any chance we can get a release? We are still on 4.x codebase

[giampaolo]

New release is out

[sylvainduchesne]

thanks alot!

[paatrofimov]

@sylvainduchesne @giampaolo
Have you found the reason, -1073741823 is happening? I actually catch same error code from GetExtendedTcpTable when trying to establish tcp connections and read tcp table in parallel.

[giampaolo]

Are you using latest 5.6.3? This issue was supposed to be fixed.

[paatrofimov]

Well, my question is more related to winapi. I actually do not use psping. The problem is that this issue is almost the only one that mentions error code -1073741823, so I wonder if you know what causes it to occur.

[paatrofimov]

I got reply here