server-side memory leak #535

totaam · 2014-03-17T03:46:53Z

Issue migrated from trac ticket # 535

component: server | priority: major | resolution: fixed

2014-03-17 03:46:53: onlyjob created the issue

0.11.4 leaks memory on server side. After using attached session for several days I've noticed that xpra process substantially grew in memory (RES > 2.5 GiB; VIRT > 3 GiB). I'm not sure how to troubleshoot this issue but I'm fairly certain that it is not related to audio which I had disabled. For now I have to run xpra upgrade periodically to release RAM.

The text was updated successfully, but these errors were encountered:

totaam · 2014-03-17T03:50:22Z

Could be related to #457.

Does it go away if you remove the webp codec from the system? (see #491)
(remove xpra/codecs/webm)

It may also be libav related...

totaam · 2014-03-17T04:28:00Z

2014-03-17 04:28:00: onlyjob commented

Thank you for quick reply.

Replying to [comment:1 totaam]:

Could be related to #457.

Does it go away if you remove the webp codec from the system? (see #491)
(remove xpra/codecs/webm)

I'll try without webp but it will take days to see the difference. Besides I have impression that those bugs are about leaks on the client side while mine is on server side.

I may also be libav related...

Everything is possible. :)
I'm still running libav-9.11 on client and server. It is the same libav that I was using for a while but I started to notice memory leak only recently...

totaam · 2014-03-17T04:54:52Z

I have impression that those bugs are about leaks on the client side while mine is on server side

True, but I suspect that the webp library may well be leaking when encoding as well as decoding.

I'm still running libav-9.11 on client and server

The reason why I suspect libav by default is twofold:

it has leaked before, and there are a number of memory related patches to make it build against libav instead of ffmpeg
the vast majority of leaks happen in areas of the code that handle large amounts of data: sound and video.

So, it would be interesting to see if the leak still occurs when using an encoding we know for sure does not leak: rgb or png have never shown signs of leaking - yet. (and to be certain video does not get used in any way, the best way is to not install the codec)

totaam · 2014-03-17T05:17:37Z

2014-03-17 05:17:37: onlyjob commented

For what's it worth this leak is with RGB encoding (set as default in config)...

totaam · 2014-03-19T09:01:13Z

It's still worth deleting all the other codecs to ensure that only RGB can be used. More and more the code is using the chosen encoding only as a hint and will switch to other encodings as appropriate (though in theory, RGB is probably the only one that should be honoured at all times...).

Can you post the output of: xpra info | grep memory then things get bad?
Does minimizing the windows help? Re-connecting the client?

totaam · 2014-03-28T16:26:36Z

2014-03-28 16:26:36: jamie commented

I seem to be experiencing the same issue. I can easily reproduce (the memory increases with every click of the web browser that I access via xrpa).

I'm running the client and server on the same machine (so I can run my web browser as an isolated user on my machine).

Question: how do I disable the other encodings? I don't seem to have a codecs/webm folder (maybe remove codecs/vpx?). I tried moving the entire codecs folder to codecs.off but it didn't seem to have any effect.

I've attached my /etc/xpra/xrpa.conf file and the server log file.

Here's my memory after starting xpra:

0 jamie@animal:cdtemp.5reEHd$ ps -e -o rss,cmd | egrep "[X]pra" | grep iceweasel
76784 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:cdtemp.5reEHd$

After launching iceweasel (Firefox)

jamie@animal:cdtemp.5reEHd$ ps -e -o rss,cmd | egrep "[X]pra" | grep iceweasel
110616 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:cdtemp.5reEHd$

Here's my command for starting xpra (run as iceweasel user):

/usr/bin/python /usr/bin/xpra --mmap-group --socket-dir=/home/iceweasel/xpra-socket --no-mdns start :201

Here's the attach command (run as jamie user):

/usr/bin/python /usr/bin/xpra --socket-dir=/home/iceweasel/xpra-socket --mmap-group --tray-icon=/usr/share/icons/hicolor/scalable/apps/iceweasel.svg --encoding=rgb24 attach :201

Here's the output of the info command:

0 jamie@animal:bin$ sudo -u iceweasel /usr/bin/python /usr/bin/xpra --mmap-group --socket-dir=/home/iceweasel/xpra-socket --no-mdns info :201 | grep memory
memory.children.idrss=0
memory.children.inblock=0
memory.children.isrss=0
memory.children.ixrss=0
memory.children.majflt=0
memory.children.maxrss=30048
memory.children.minflt=19940
memory.children.msgrcv=0
memory.children.msgsnd=0
memory.children.nivcsw=54
memory.children.nsignals=0
memory.children.nswap=0
memory.children.nvcsw=261
memory.children.oublock=32
memory.children.stime=0
memory.children.utime=0
memory.server.idrss=0
memory.server.inblock=0
memory.server.isrss=0
memory.server.ixrss=0
memory.server.majflt=4
memory.server.maxrss=64392
memory.server.minflt=27163
memory.server.msgrcv=0
memory.server.msgsnd=0
memory.server.nivcsw=234
memory.server.nsignals=0
memory.server.nswap=0
memory.server.nvcsw=3989
memory.server.oublock=12024
memory.server.stime=0
memory.server.utime=0
0 jamie@animal:bin$

totaam · 2014-03-28T16:34:24Z

2014-03-28 16:34:24: jamie uploaded file `xpra.conf` (3.4 KiB)

jamie-etc-xpra-xpra.conf

totaam · 2014-03-28T16:35:16Z

2014-03-28 16:35:16: jamie uploaded file `:201.log` (4.1 KiB)

jamie-server-log

totaam · 2014-03-28T16:46:41Z

2014-03-28 16:46:41: jamie commented

Also, minimizing the window causes the memory usage of the server to increase and maximizing the window causes the memory to increase.

Detaching has no impact on memory usage.

Attaching causes the memory usage to increase.

totaam · 2014-03-29T01:39:13Z

2014-03-29 01:39:13: totaam commented

@jamie: what is your distro? What version are you using? Do you get the same results with 0.12.x? Please include the full xpra info as an attachment, not just the memory bits.
[[BR]]

how do I disable the other encodings?
[[BR]]
You can disable most of them at build time.
At runtime, it is more difficult.
[[BR]]

I don't seem to have a codecs/webm folder (maybe remove codecs/vpx?).
[[BR]]
Strange, did you build without webp (aka webm) support?
[[BR]]

I tried moving the entire codecs folder to codecs.off but it didn't seem to have any effect.
[[BR]]
It will have an effect and will certainly break everything. If it didn't, maybe you moved the wrong one.
[[BR]]

minimizing the window causes the memory usage of the server to increase and maximizing the window causes the memory to increase
[[BR]]
Interesting, maximizing is understandable. But minimizing less so!
[[BR]]

Attaching causes the memory usage to increase
[[BR]]
This one is expected as we allocate resources for each client.

totaam · 2014-03-31T02:15:49Z

2014-03-31 02:15:49: jamie uploaded file `xpra-info.txt` (67.6 KiB)

Full output of xpra info

totaam · 2014-03-31T02:16:48Z

2014-03-31 02:16:48: jamie commented

Thanks for the response. I just added the full output of xpra info.

I'm running Debian Jessie and xpra 0.11.4+dfsg-1.

I will try to compile latest xpra and see if the problem goes away.

totaam · 2014-03-31T03:49:58Z

2014-03-31 03:49:58: jamie commented

Alas, no luck.

I tried installing using the Debian packaging with a 0.12.2 tar ball and slight modifications to the Debian-shipped patches:

diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/disable_html5.patch debian/patches/disable_html5.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/disable_html5.patch  2014-02-03 11:23:26.000000000 -0500
+++ debian/patches/disable_html5.patch  2014-03-30 22:12:09.411752548 -0400
@@ -7,7 +7,7 @@
  gtk3_ENABLED = client_ENABLED
  qt4_ENABLED = client_ENABLED
  opengl_ENABLED = client_ENABLED
--html5_ENABLED = True
+-html5_ENABLED = not WIN32 and not OSX
 +html5_ENABLED = False
  
  bencode_ENABLED = True
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/enable-x264-opencl.patch debian/patches/enable-x264-opencl.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/enable-x264-opencl.patch 2014-02-05 09:54:38.000000000 -0500
+++ debian/patches/enable-x264-opencl.patch 2014-03-30 22:12:09.411752548 -0400
@@ -6,18 +6,16 @@
 
 --- a/xpra/codecs/enc_x264/encoder.pyx
 +++ b/xpra/codecs/enc_x264/encoder.pyx
-@@ -69,8 +69,11 @@
-         int i_bframe_pyramid    #Keep some B-frames as references: 0=off, 1=strict hierarchical, 2=normal
+@@ -117,6 +117,9 @@
          int b_open_gop
          int b_bluray_compat
  
 +        #not all builds support this:
 +        int b_opencl            #use OpenCL when available
 +
+         rc  rc                  #rate control
+ 
      ctypedef struct x264_t:
-         pass
-     ctypedef struct x264_nal_t:
-         int i_ref_idc
 @@ -250,8 +253,9 @@
          #we don't want IDR frames either:
          param.i_keyint_min = 999999
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav10.patch debian/patches/libav10.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav10.patch  2014-03-03 07:33:40.000000000 -0500
+++ debian/patches/libav10.patch  2014-03-30 22:15:27.323962377 -0400
@@ -1,137 +1,24 @@
-Last-Update: 2014-03-03
+Description: libav10 compatibility
+Author: Antoine Martin <antoine@devloop.org.uk>
 Forwarded: not-needed
-Author: Dmitry Smirnov <onlyjob@member.fsf.org>
 Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=739459
 Bug-Xpra: https://www.xpra.org/trac/ticket/524
-Description: fix FTBFS with libav10
 
---- a/xpra/codecs/dec_avcodec/decoder.pyx
-+++ b/xpra/codecs/dec_avcodec/decoder.pyx
-@@ -86,12 +86,12 @@
-         int thread_type
-         int flags
-         int flags2
- 
--    AVPixelFormat PIX_FMT_NONE
--    AVCodecID CODEC_ID_H264
--    AVCodecID CODEC_ID_VP8
--    #AVCodecID CODEC_ID_VP9
-+    AVPixelFormat AV_PIX_FMT_NONE
-+    AVCodecID AV_CODEC_ID_H264
-+    AVCodecID AV_CODEC_ID_VP8
-+    #AVCodecID AV_CODEC_ID_VP9
- 
-     #init and free:
-     void avcodec_register_all()
-     AVCodec *avcodec_find_decoder(AVCodecID id)
-@@ -164,13 +164,13 @@
-     global CODECS
-     if CODECS is None:
-         avcodec_register_all()
-         CODECS = []
--        if avcodec_find_decoder(CODEC_ID_H264)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_H264)!=NULL:
-             CODECS.append("h264")
--        if avcodec_find_decoder(CODEC_ID_VP8)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_VP8)!=NULL:
-             CODECS.append("vp8")
--        #if avcodec_find_decoder(CODEC_ID_VP9)!=NULL:
-+        #if avcodec_find_decoder(AV_CODEC_ID_VP9)!=NULL:
-         #    CODECS.append("vp9")
-     return CODECS
- 
- 
-@@ -366,24 +366,24 @@
-                 break
-         if self.colorspace==NULL:
-             error("invalid pixel format: %s", colorspace)
-             return  False
--        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, PIX_FMT_NONE)
--        if self.pix_fmt==PIX_FMT_NONE:
-+        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, AV_PIX_FMT_NONE)
-+        if self.pix_fmt==AV_PIX_FMT_NONE:
-             error("invalid pixel format: %s", colorspace)
-             return  False
-         self.actual_pix_fmt = self.pix_fmt
- 
-         avcodec_register_all()
- 
-         if self.encoding=="h264":
--            self.codec = avcodec_find_decoder(CODEC_ID_H264)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_H264)
-             if self.codec==NULL:
-                 error("codec H264 not found!")
-                 return  False
-         else:
-             assert self.encoding=="vp8"
--            self.codec = avcodec_find_decoder(CODEC_ID_VP8)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_VP8)
-             if self.codec==NULL:
-                 error("codec VP8 not found!")
-                 return  False
- 
---- a/xpra/codecs/dec_avcodec2/decoder.pyx
-+++ b/xpra/codecs/dec_avcodec2/decoder.pyx
-@@ -83,12 +83,12 @@
-         int flags
-         int flags2
-         int refcounted_frames
- 
--    AVPixelFormat PIX_FMT_NONE
--    AVCodecID CODEC_ID_H264
--    AVCodecID CODEC_ID_VP8
--    #AVCodecID CODEC_ID_VP9
-+    AVPixelFormat AV_PIX_FMT_NONE
-+    AVCodecID AV_CODEC_ID_H264
-+    AVCodecID AV_CODEC_ID_VP8
-+    #AVCodecID AV_CODEC_ID_VP9
- 
-     #init and free:
-     void avcodec_register_all()
-     AVCodec *avcodec_find_decoder(AVCodecID id)
-@@ -155,13 +155,13 @@
-     global CODECS
-     if CODECS is None:
-         avcodec_register_all()
-         CODECS = []
--        if avcodec_find_decoder(CODEC_ID_H264)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_H264)!=NULL:
-             CODECS.append("h264")
--        if avcodec_find_decoder(CODEC_ID_VP8)!=NULL:
-+        if avcodec_find_decoder(AV_CODEC_ID_VP8)!=NULL:
-             CODECS.append("vp8")
--        #if avcodec_find_decoder(CODEC_ID_VP9)!=NULL:
-+        #if avcodec_find_decoder(AV_CODEC_ID_VP9)!=NULL:
-         #    CODECS.append("vp9")
-     return CODECS
- 
- 
-@@ -267,24 +267,24 @@
-                 break
-         if self.colorspace==NULL:
-             error("invalid pixel format: %s", colorspace)
-             return  False
--        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, PIX_FMT_NONE)
--        if self.pix_fmt==PIX_FMT_NONE:
-+        self.pix_fmt = FORMAT_TO_ENUM.get(colorspace, AV_PIX_FMT_NONE)
-+        if self.pix_fmt==AV_PIX_FMT_NONE:
-             error("invalid pixel format: %s", colorspace)
-             return  False
-         self.actual_pix_fmt = self.pix_fmt
- 
-         avcodec_register_all()
- 
-         if self.encoding=="h264":
--            self.codec = avcodec_find_decoder(CODEC_ID_H264)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_H264)
-             if self.codec==NULL:
-                 error("codec H264 not found!")
-                 return  False
-         else:
-             assert self.encoding=="vp8"
--            self.codec = avcodec_find_decoder(CODEC_ID_VP8)
-+            self.codec = avcodec_find_decoder(AV_CODEC_ID_VP8)
-             if self.codec==NULL:
-                 error("codec VP8 not found!")
-                 return  False
+Index: xpra/codecs/dec_avcodec/decoder.pyx
+===================================================================
+--- a/xpra/codecs/dec_avcodec/decoder.pyx  (revision 5906)
++++ b/xpra/codecs/dec_avcodec/decoder.pyx  (working copy)
+@@ -109,7 +109,13 @@
+     int avcodec_default_get_buffer(AVCodecContext *s, AVFrame *pic)
+     void avcodec_default_release_buffer(AVCodecContext *s, AVFrame *pic)
+ 
++cdef extern from "libavcodec/old_codec_ids.h":
++    AVPixelFormat PIX_FMT_NONE
++    AVCodecID CODEC_ID_H264
++    AVCodecID CODEC_ID_VP8
++    #AVCodecID CODEC_ID_VP9
+ 
++
+ def get_version():
+     return (LIBAVCODEC_VERSION_MAJOR, LIBAVCODEC_VERSION_MINOR, LIBAVCODEC_VERSION_MICRO)
  
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav-no0RGB.patch debian/patches/libav-no0RGB.patch
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/libav-no0RGB.patch 2014-02-02 22:17:00.000000000 -0500
+++ debian/patches/libav-no0RGB.patch 2014-03-30 22:12:09.411752548 -0400
@@ -36,12 +36,13 @@
  AV_PIX_FMT_GBRP
 --- a/xpra/codecs/dec_avcodec/decoder.pyx
 +++ b/xpra/codecs/dec_avcodec/decoder.pyx
-@@ -140,8 +140,6 @@
-             "YUV422P"   : "AV_PIX_FMT_YUV422P",
-             "YUV444P"   : "AV_PIX_FMT_YUV444P",
-             "RGB"       : "AV_PIX_FMT_RGB24",
--            "XRGB"      : "AV_PIX_FMT_0RGB",
--            "BGRX"      : "AV_PIX_FMT_BGR0",
-             "ARGB"      : "AV_PIX_FMT_ARGB",
-             "BGRA"      : "AV_PIX_FMT_BGRA",
-             "GBRP"      : "AV_PIX_FMT_GBRP",
+@@ -143,8 +143,6 @@
+         "YUV422P"   : "AV_PIX_FMT_YUV422P",
+         "YUV444P"   : "AV_PIX_FMT_YUV444P",
+         "RGB"       : "AV_PIX_FMT_RGB24",
+-        "XRGB"      : "AV_PIX_FMT_0RGB",
+-        "BGRX"      : "AV_PIX_FMT_BGR0",
+         "ARGB"      : "AV_PIX_FMT_ARGB",
+         "BGRA"      : "AV_PIX_FMT_BGRA",
+         "GBRP"      : "AV_PIX_FMT_GBRP",
+
diff -u xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/series debian/patches/series
--- xpra-debian-source/xpra-0.11.4+dfsg/debian/patches/series 2014-03-03 16:49:16.000000000 -0500
+++ debian/patches/series 2014-03-30 22:12:09.411752548 -0400
@@ -19,4 +19,4 @@
 libav10.patch
 
 ## Conditional patch, must be the last one.
-old-libav.patch
+#old-libav.patch

However, same result.

Next, I did an svn checkout of the 0.12.x branch with no debian patches. I first got a compile error:

x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python2.7 -c xpra/codecs/dec_avcodec/decoder.c -o build/temp.linux-x86_64-2.7/xpra/codecs/dec_avcodec/decoder.o -Wall -Werror=implicit-function-declaration -fPIC
xpra/codecs/dec_avcodec/decoder.c: In function ‘initdecoder’:
xpra/codecs/dec_avcodec/decoder.c:10104:45: error: ‘AV_PIX_FMT_0RGB’ undeclared (first use in this function)
   __pyx_t_1 # __Pyx_PyInt_From_unsigned_int(AV_PIX_FMT_0RGB); if (unlikely(!__pyx_t_1)) {__pyx_filename__pyx_f[1]; __pyx_lineno # 17; __pyx_clineno__LINE__; goto __pyx_L1_error;}
                                             ^
xpra/codecs/dec_avcodec/decoder.c:10104:45: note: each undeclared identifier is reported only once for each function it appears in
xpra/codecs/dec_avcodec/decoder.c:10116:45: error: ‘AV_PIX_FMT_BGR0’ undeclared (first use in this function)
   __pyx_t_1 # __Pyx_PyInt_From_unsigned_int(AV_PIX_FMT_BGR0); if (unlikely(!__pyx_t_1)) {__pyx_filename__pyx_f[1]; __pyx_lineno # 18; __pyx_clineno__LINE__; goto __pyx_L1_error;}
                                             ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
dh_auto_build: python setup.py build --force returned exit code 1
make: *** [build] Error 1
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1364:
dpkg-buildpackage -rfakeroot -D -us -uc -b failed
29 jamie@animal:src$

So, I just applied this patch:

--- a/xpra/codecs/csc_swscale/constants.txt
+++ b/xpra/codecs/csc_swscale/constants.txt
@@ -14,8 +14,8 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
+#AV_PIX_FMT_0RGB
+#AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/constants.txt
+++ b/xpra/codecs/dec_avcodec/constants.txt
@@ -14,8 +14,8 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
+#AV_PIX_FMT_0RGB
+#AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
--- a/xpra/codecs/dec_avcodec/decoder.pyx
+++ b/xpra/codecs/dec_avcodec/decoder.pyx
@@ -143,8 +143,6 @@
         "YUV422P"   : "AV_PIX_FMT_YUV422P",
         "YUV444P"   : "AV_PIX_FMT_YUV444P",
         "RGB"       : "AV_PIX_FMT_RGB24",
-        "XRGB"      : "AV_PIX_FMT_0RGB",
-        "BGRX"      : "AV_PIX_FMT_BGR0",
         "ARGB"      : "AV_PIX_FMT_ARGB",
         "BGRA"      : "AV_PIX_FMT_BGRA",
         "GBRP"      : "AV_PIX_FMT_GBRP",

And it successfully compiled.

However, I'm still seeing the server memory leak.

Let me know if there are any other tests you'd like me to perform or any other logs I can provide to help get to the bottom of this.

totaam · 2014-04-03T14:36:58Z

2014-04-03 14:36:58: totaam commented

Confirmed here at last, took a few hours, using glxspheres made it happen quicker (lots of frames per second).

Will take care of it as a priority. Wiki link: Debugging

totaam · 2014-04-03T15:22:54Z

2014-04-03 15:22:54: totaam commented

Output from top and running glxspheres with XPRA_XSHM=0:

20450 antoine   20   0 65.240g 101628  32916 R  97.2  0.8   0:06.47
20450 antoine   20   0 65.240g 111532  32976 S  94.5  0.9   2:39.92
20450 antoine   20   0 65.583g 110928  32984 S  92.6  0.9   7:59.86
20450 antoine   20   0 65.587g 126728  32984 S  94.7  1.0   9:54.56
20450 antoine   20   0 65.578g 113120  32984 S  95.0  0.9  10:20.34
20450 antoine   20   0 65.587g 133464  32984 S  93.2  1.1  12:33.52
20450 antoine   20   0 65.649g 156240  32984 S  92.2  1.3  17:20.23
20450 antoine   20   0 65.645g 177276  32984 R  88.7  1.4  18:57.47

Still leaking, so it isn't xshm refcount related.

totaam · 2014-04-04T06:08:30Z

2014-04-04 06:08:30: totaam commented

With XPRA_DETECT_LEAKS=1 support added in r6013, I see:

2014-04-04 12:01:46,738 print_leaks():
       55 : <type 'tuple'> : 2829 matches
       27 : <type 'cell'> : 762 matches
       22 : <type 'frame'> : 274 matches
        9 : <type 'list'> : 2883 matches
        7 : <type 'builtin_function_or_method'> : []
        4 : <type 'instancemethod'> : []
        2 : <type 'instance'> : 442 matches

Then a little while later:

2014-04-04 12:03:26,848 print_leaks():
       54 : <type 'tuple'> : 3383 matches
       34 : <type 'cell'> : 1173 matches
       17 : <type 'frame'> : 469 matches
       15 : <type 'builtin_function_or_method'> : []
        9 : <type 'list'> : 2982 matches
        6 : <type 'xpra.x11.bindings.ximage.XShmImageWrapper'> : 47 matches
        3 : <type 'instancemethod'> : []
        2 : <type 'dict'> : 3783 matches

2014-04-04 13:57:21,998 print_leaks():
       52 : <type 'tuple'> : 11378 matches
       45 : <type 'cell'> : 6503 matches
       35 : <type 'instancemethod'> : []
       23 : <type 'list'> : 6089 matches
       22 : <type 'frame'> : 9958 matches
       13 : <type 'dict'> : 3923 matches
       10 : <type 'collections.deque'> : 55 matches
        7 : <type 'builtin_function_or_method'> : []
        3 : <type 'xpra.x11.bindings.ximage.XShmImageWrapper'> : 129 matches

So, it looks like we're leaking:

tuples, dicts and lists (probably as part of other objects)
frames - no idea why we would leak those
cells (whatever they are...)
XShmImageWrapper is back in the picture (maybe more than one leak then...)

totaam · 2014-04-04T13:59:11Z

2014-04-04 13:59:11: totaam uploaded file `no-video-subregion.patch` (0.4 KiB)

disable subregion code

totaam · 2014-04-04T14:17:00Z

2014-04-04 14:17:00: totaam commented

After some more improvements to the leak debugging code in r6017, r6021, r6022

I have narrowed it down to the video subregions code, using the [/attachment/ticket/535/no-video-subregion.patch] seems to have stopped the leak.

Unless I can fix this reliably in the next 24 hours, I may disable video subregions (server side) for the next stable update.

@onlyjob and @jamie: can you please confirm that the patch does prevent the leak you were seeing?

totaam · 2014-04-04T14:35:54Z

2014-04-04 14:35:54: onlyjob commented

I had troubles reproducing the leak lately and I'm not sure why. It could be that it was leaking only while I was using some application (libreoffice?) that I don't use too often...

totaam · 2014-04-04T15:37:55Z

2014-04-04 15:37:55: totaam commented

I had troubles reproducing the leak lately
[[BR]]
Me too, and that makes me nervous... maybe the "fix" is only hiding the real cause, by making us follow a different codepath.

The debug output still points towards a very slow leak:

XShmImageWrapper: But the verbose debug code does not find anything referencing those objects!??!

<type 'frame'>: it could just be the debug code causing those.. heisenbug?

The other objects that cycle a lot are all fairly stable at around:
        3 : <type 'cell'>     : 459 matches
        1 : <type 'dict'>     : 3593 matches
        2 : <type 'list'>     : 2806 matches
        1 : <type 'instance'> : 427 matches
       12 : <type 'tuple'>    : 2365 matches
The two could be related. More investigation is needed.

I did find some other interesting leak issues along the way, like a fairly rare case when clients disconnect, fixed in r6023 - will need a backport.

totaam · 2014-04-04T15:52:01Z

2014-04-04 15:52:01: jamie commented

Unfortunately (or fortunately?) I'm still able to reproduce the memory leak, even with this patch. I tried applying the patch against the 12.x tag and against trunk itself. Here's the diff of my svn checkout that I most recently build a deb package from:

0 jamie@animal:xpra.svn.trunk$ svn diff
Index: src/xpra/codecs/csc_swscale/constants.txt
===================================================================
--- src/xpra/codecs/csc_swscale/constants.txt	(revision 6023)
+++ src/xpra/codecs/csc_swscale/constants.txt	(working copy)
@@ -14,8 +14,6 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
Index: src/xpra/codecs/dec_avcodec/constants.txt
===================================================================
--- src/xpra/codecs/dec_avcodec/constants.txt	(revision 6023)
+++ src/xpra/codecs/dec_avcodec/constants.txt	(working copy)
@@ -14,8 +14,6 @@
 AV_PIX_FMT_YUV422P
 AV_PIX_FMT_YUV444P
 AV_PIX_FMT_RGB24
-AV_PIX_FMT_0RGB
-AV_PIX_FMT_BGR0
 AV_PIX_FMT_ARGB
 AV_PIX_FMT_BGRA
 AV_PIX_FMT_GBRP
Index: src/xpra/server/window_video_source.py
===================================================================
--- src/xpra/server/window_video_source.py	(revision 6023)
+++ src/xpra/server/window_video_source.py	(working copy)
@@ -245,6 +245,7 @@
 
 
     def identify_video_subregion(self):
+	return
         if self.statistics.damage_events_count < self.video_subregion_set_at:
             #stats got reset
             self.video_subregion_set_at = 0
0 jamie@animal:xpra.svn.trunk$

I've just applied one debian patch (without which it fails to compile) and the patch you attached.

However, still getting substantial memory leak every time I click a link in my web browser.

Here's a example - I ran the command once, then clicked on a link which loaded a new web page and then I ran it again:


0 jamie@animal:~$ ps -e -o rss,cmd | egrep '[X]pra' | grep ice
148928 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$ ps -e -o rss,cmd | egrep '[X]pra' | grep ice
151040 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$

totaam · 2014-04-04T15:58:49Z

2014-04-04 15:58:49: jamie uploaded file `:201.2.log` (39.4 KiB)

Server log with XPRA_DETECT_LEAKS=1

totaam · 2014-04-04T15:59:41Z

2014-04-04 15:59:41: jamie commented

I just attached a server log with XPRA_DETECT_LEAKS=1 - let me know if there is any more debugging I can do that would be helpful.

totaam · 2014-04-05T02:19:54Z

2014-04-05 02:19:54: totaam commented

@jamie: I have only just noticed that the process whose memory you are claiming is going up is the X11 vfb process and not xpra itself. The X11 server will allocate memory whenever the client application (firefox or other, xpra too) requests any resources. xpra's allocations will go away if you exit the xpra server process (use "xpra exit") - problem with this is that we may also unmap some of the client windows on exit IIRC, so this will free some resources we should not really be accountable for..

Even if it was the xpra process, the memory going up by just 3MB when doing something is not in itself evidence of a leak, only if the increase is persistent.
Because of python's garbage collection, the memory will fluctuate, it is the long term trend that is more telling.
We also collect statistics, delta buffers, video encoders, etc... and those all take time to initialize and populate fully.

totaam · 2014-04-05T06:50:32Z

2014-04-05 06:50:32: totaam commented

Running for over 30 minutes with the 0.12.x branch as of 6036, and I saw no real change with mmap:

11:44 xpra server rss: 246220, Xorg rss: 116036

12:19 xpra server rss: 226732, Xorg rss: 116036

Or with plain rgb:

12:21 xpra server rss: 98968, Xorg rss: 131432

12:48 xpra server rss: 96932, Xorg rss: 131432

So... I just cannot reproduce any leaks. What am I missing?

totaam · 2014-04-05T10:40:05Z

2014-04-05 10:40:05: totaam uploaded file `debux-xshmimage-leak.patch` (1.3 KiB)

patch I've used to verify that we are not leaking xshm image wrappers

totaam · 2014-04-07T06:10:04Z

2014-04-07 06:10:04: totaam commented

Another idea, please disable (or even remove) OpenCL support, either:

build xpra with --without-opencl (and ensure you don't have an old copy installed)

use --csc-modules=cython,swscale at runtime (should work)

temporarily move /etc/OpenCL out of the way

Does that make any difference? Some of the OpenCL ICDs (IIRC, AMD's in particular) cause problems with threads and signals when loaded.

totaam · 2014-04-07T14:21:04Z

2014-04-07 14:21:04: jamie commented

Replying to [comment:19 totaam]:

@jamie: I have only just noticed that the process whose memory you are claiming is going up is the X11 vfb process and not xpra itself. The X11 server will allocate memory whenever the client application (firefox or other, xpra too) requests any resources. xpra's allocations will go away if you exit the xpra server process (use "xpra exit") - problem with this is that we may also unmap some of the client windows on exit IIRC, so this will free some resources we should not really be accountable for..

Yes! That is definitely true. Sorry - I didn't realize it was not xpra itself. Should I file this against xvfb? Are there other steps you might suggest to help keep this memory usage under control or any other ways of running xpra to avoid it?

Even if it was the xpra process, the memory going up by just 3MB when doing something is not in itself evidence of a leak, only if the increase is persistent.

I'm so sorry to waste your time on this - there is minimal increase in memory in the actual xpra server that I am noticing (the xpra client does go up, but nothing like the X11 vfb process) but nothing worth reporting and I imagine it is fluctuating like you say.

I think we should close this bug.

totaam · 2014-04-07T14:37:38Z

2014-04-07 14:37:38: onlyjob commented

I agree, let's close this bug for now. I can't reproduce the leak any more. First thing I tried to do is to uninstall "python-webp" on server side. When leak did not manifest for a while I installed patched version "0.2.2-3" of "python-webp" (on Debian) which was also built against "libwebp5". Leak was there with "python-webp-0.2.2-2/libwebp4". If leak was related to libwebp4 or to python-webp prior to 0.2.2-3 then perhaps it was fixed somewhere...

totaam · 2014-04-07T14:42:28Z

2014-04-07 14:42:28: totaam commented

Should I file this against xvfb?
[[BR]]
Not necessarily, it could still be xpra consuming those X11 server resources.

Does stopping the browser whilst xpra is running restore all the memory?
Does turning off xshm help (it will reduce the memory used, but does it affect the rate significantly?):
XPRA_XSHM=0 xpra start ...

totaam · 2014-04-07T15:18:07Z

2014-04-07 15:18:07: jamie commented

When closing the browser, it goes down a little (from 146064 kb to 145936) but doesn't restore all of it.

Adding XPRA_XSHM=0 doesn't seem to have any effect on the memory.

totaam · 2014-04-08T04:42:15Z

2014-04-08 04:42:15: totaam commented

When closing the browser, it goes down a little (from 146064 kb to 145936) but doesn't restore all of it.
[[BR]]
I would have expected it to go down a lot more.
How much does it go down by if you exit the xpra server? (using xpra exit)

totaam · 2014-04-08T15:15:27Z

2014-04-08 15:15:27: jamie commented

Here's memory usage after three or four clicks:

0 jamie@animal:~$ ps -e -o rss,cmd | grep Xpra | grep iceweasel
182752 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$

Here's memory usage after xpra exit:

0 jamie@animal:~$ ps -e -o rss,cmd | grep Xpra | grep iceweasel
136308 /usr/bin/Xorg-for-Xpra-:201 -dpi 96 -noreset -nolisten tcp +extension GLX +extension RANDR +extension RENDER -logfile /home/iceweasel/.xpra/Xorg.:201.log -config /etc/xpra/xorg.conf :201
0 jamie@animal:~$

totaam · 2014-07-10T17:28:00Z

Sorry for taking so long to get onto this bug, should be fixed in r6874. (easily spotted with xrestop)
Further improvements in r6875 and r6876. (will backport at least the first one)

@onlyjob / @jamie: can I close this ticket?

totaam · 2014-07-20T03:14:22Z

2014-07-20 03:14:22: onlyjob commented

Yes let's close please. I haven't seen this issue for a while.

totaam closed this as completed Jul 20, 2014

totaam added the v0.11.x label Jan 22, 2021

This was referenced Jan 22, 2021

crash running glxgears and gtkperf in a loop #569

Closed

Xorg memory leak while resizing #610

Closed

totaam mentioned this issue May 11, 2021

client side memory leak #3123

Closed

totaam mentioned this issue Jan 3, 2025

client memory leak #457

Closed

server-side memory leak #535

server-side memory leak #535

Comments

totaam commented Mar 17, 2014

2014-03-17 03:46:53: onlyjob created the issue

totaam commented Mar 17, 2014 • edited Loading

totaam commented Mar 17, 2014

2014-03-17 04:28:00: onlyjob commented

totaam commented Mar 17, 2014 • edited Loading

totaam commented Mar 17, 2014

2014-03-17 05:17:37: onlyjob commented

totaam commented Mar 19, 2014 • edited Loading

totaam commented Mar 28, 2014

2014-03-28 16:26:36: jamie commented

totaam commented Mar 28, 2014

2014-03-28 16:34:24: jamie uploaded file xpra.conf (3.4 KiB)

totaam commented Mar 28, 2014

2014-03-28 16:35:16: jamie uploaded file :201.log (4.1 KiB)

totaam commented Mar 28, 2014

2014-03-28 16:46:41: jamie commented

totaam commented Mar 29, 2014

2014-03-29 01:39:13: totaam commented

totaam commented Mar 31, 2014

2014-03-31 02:15:49: jamie uploaded file xpra-info.txt (67.6 KiB)

totaam commented Mar 31, 2014

2014-03-31 02:16:48: jamie commented

totaam commented Mar 31, 2014

2014-03-31 03:49:58: jamie commented

totaam commented Apr 3, 2014

2014-04-03 14:36:58: totaam commented

totaam commented Apr 3, 2014

2014-04-03 15:22:54: totaam commented

totaam commented Apr 4, 2014

2014-04-04 06:08:30: totaam commented

totaam commented Apr 4, 2014

2014-04-04 13:59:11: totaam uploaded file no-video-subregion.patch (0.4 KiB)

totaam commented Apr 4, 2014

2014-04-04 14:17:00: totaam commented

totaam commented Apr 4, 2014

2014-04-04 14:35:54: onlyjob commented

totaam commented Apr 4, 2014

2014-04-04 15:37:55: totaam commented

totaam commented Apr 4, 2014

2014-04-04 15:52:01: jamie commented

totaam commented Apr 4, 2014

2014-04-04 15:58:49: jamie uploaded file :201.2.log (39.4 KiB)

totaam commented Apr 4, 2014

2014-04-04 15:59:41: jamie commented

totaam commented Apr 5, 2014

2014-04-05 02:19:54: totaam commented

totaam commented Apr 5, 2014

2014-04-05 06:50:32: totaam commented

totaam commented Apr 5, 2014

2014-04-05 10:40:05: totaam uploaded file debux-xshmimage-leak.patch (1.3 KiB)

totaam commented Apr 7, 2014

2014-04-07 06:10:04: totaam commented

totaam commented Apr 7, 2014

2014-04-07 14:21:04: jamie commented

totaam commented Apr 7, 2014

2014-04-07 14:37:38: onlyjob commented

totaam commented Apr 7, 2014

2014-04-07 14:42:28: totaam commented

totaam commented Apr 7, 2014

2014-04-07 15:18:07: jamie commented

totaam commented Apr 8, 2014

2014-04-08 04:42:15: totaam commented

totaam commented Apr 8, 2014

2014-04-08 15:15:27: jamie commented

totaam commented Jul 10, 2014 • edited Loading

totaam commented Jul 20, 2014

2014-07-20 03:14:22: onlyjob commented

totaam commented Mar 17, 2014 •

edited

Loading

totaam commented Mar 17, 2014 •

edited

Loading

totaam commented Mar 19, 2014 •

edited

Loading

2014-03-28 16:34:24: jamie uploaded file `xpra.conf` (3.4 KiB)

2014-03-28 16:35:16: jamie uploaded file `:201.log` (4.1 KiB)

2014-03-31 02:15:49: jamie uploaded file `xpra-info.txt` (67.6 KiB)

2014-04-04 13:59:11: totaam uploaded file `no-video-subregion.patch` (0.4 KiB)

2014-04-04 15:58:49: jamie uploaded file `:201.2.log` (39.4 KiB)

2014-04-05 10:40:05: totaam uploaded file `debux-xshmimage-leak.patch` (1.3 KiB)

totaam commented Jul 10, 2014 •

edited

Loading