high memory usage

[totaam]

Despite #1861 and #1838, the client and server still use too much memory, even when disabling most of the video and csc modules - or even nuking them from the filesystem.

[totaam]

According to how to get "real" process memory and environ in Python, we aren't really using as much memory as previously thought.
The amount of memory which would be freed if the process was terminated right now is only around 200MB for a server, much less for a client.
Caveats: this does not include shared libraries we share with other processes, GPU memory, etc.

With the debug code added in r19985, we use psutil (docs: https://psutil.readthedocs.io/en/latest/ psutil]) to dump memory info at time intervals:

XPRA_MEM_USAGE_LOGGER=100 xpra start-desktop --start=openbox --daemon=no
2018-08-01 16:13:11,847 init_html_proxy(..) html=None
2018-08-01 16:13:11,857 memory usage: pfullmem(rss=60104704, vms=552919040, shared=24821760, text=4096, lib=0, data=42868736, dirty=0, uss=46182400, pss=48150528, swap=0)
2018-08-01 16:13:11,944 serving html content from: /usr/share/xpra/www
2018-08-01 16:13:11,944 get_auth_modules(local-auth, [], {..})
2018-08-01 16:13:11,944 get_auth_modules(tcp, [], {..})
2018-08-01 16:13:11,944 get_auth_modules(ws, [], {..})
2018-08-01 16:13:11,944 get_auth_modules(wss, [], {..})
2018-08-01 16:13:11,944 get_auth_modules(ssl, [], {..})
2018-08-01 16:13:11,944 get_auth_modules(ssh, [], {..})
2018-08-01 16:13:11,945 get_auth_modules(rfb, [], {..})
2018-08-01 16:13:11,945 get_auth_modules(vsock, [], {..})
2018-08-01 16:13:11,945 get_auth_modules(udp, [], {..})
2018-08-01 16:13:11,945 init_auth(..) auth={'udp': None, 'rfb': None, 'vsock': None, 'tcp': None, 'ssl': None, 'wss': None, 'ws': None, 'ssh': None, 'unix-domain': None}
2018-08-01 16:13:11,945 file transfer: init_attributes('auto', 100, 'no', 'auto', 'auto', '/usr/bin/xdg-open', False)
2018-08-01 16:13:11,945 file transfer attributes={'open-files-ask': False, 'open-url': True, 'file-ask-timeout': 3600, 'open-files': True, 'file-size-limit': 100, 'file-transfer': True, 'file-chunks': 65536, 'file-transfer-ask': False, 'printing-ask': False, 'open-url-ask': False, 'printing': False}
2018-08-01 16:13:11,945 ChildReaper(None)
2018-08-01 16:13:11,958 memory usage: pfullmem(rss=68050944, vms=979922944, shared=29216768, text=4096, lib=0, data=173236224, dirty=0, uss=40009728, pss=50673664, swap=0)

The "data" memory usage goes up by ~128MB around the time we init websockify and the auth modules / file transfer.

[totaam]

Helped by r19986, I found that the memory usage is coming from importing websockify. More specifically: websockify imports numpy which is responsible for most of that 128MB memory usage. Follow up ticket: #1926.

r19987 changes some of the xpra codec loading code so we don't import numpy unless we really have to (ie: pycuda requires it).
With this change and turning off websockify with --html=no, the memory usage is now much lower:

2018-08-01 18:05:02,405 memory usage: pfullmem(rss=89583616, vms=755613696, shared=36974592, text=4096, lib=0, data=78819328, dirty=0, uss=74027008, pss=75572224, swap=0)

The next big chunks of memory come from the video and csc codecs (~40MB), which is to be expected. And also dbus (20MB!).

@maxmylyn: FYI.

[totaam]

2018-08-01 18:49:00: maxmylyn commented

---

Noted and closing.

(Dbus uses far more memory than I thought it would have - 30mb is quite a lot for something so simple)