Memory leak in page.getPixmap()

[thefloe1]

I'm using pymupdf to loop though all pages in a PDF and extracting pages from it:

doc = fitz.open(filename)
for num,page in enumerate(doc):
	pix = page.getPixmap(m)
	data = pix.getPNGData()	
	# doing some magic here

Using memory profiler I see that every loop getPixmap allocates 20MB of data which are not freed. I tried different things like del pix or gc.collect but the memory usage increases every run. As I'm handling quiet some pages, my scripts runs out of memory...

getPixmap also shops the following warning/error:

error: Unable to read ICC workflow
warning: Attempt to read Output Intent failed

[JorjMcKie]

I am on a journey abroad, so I am limited to dig into this very much. I'll be back Thirsday next week.

Anyway, this sounds like a problem that I thought was fixed. Please provide me with your system parameters: OS, bitness, Python version and PyMuPDF version.
I will at least be able to look into the source.

Are you sure that the memory accumulation is caused by pixmap creation - and not by the getPNGdata method?

[thefloe1]

Hi,
It's definitely on the getPixmap.

OS is Win 10, 64bit,
Phyton 2.7.12 (32bit),
PyMuPDF 1.12.2: Python bindings for the MuPDF 1.12.0 library

I see the same memory behavior when running the demo pdf viewer (PDFdisplay.py).

[JorjMcKie]

ok, thanks for the info. That's pixmap creation then.
As I said, give me some time / opportunity to look into that ...

[JorjMcKie]

I have found the cause for the issue, and I am testing the fix.
I will upload it next week, probably Thursday.

[thefloe1]

Thanks a lot

[JorjMcKie]

Just uploaded a fix into branch 1.12.2.
Please check that your issue is being resolved.

[thefloe1]

Memory consumption stays now below 400MB and does not gradually increase. I guess still some stuff stays in memory as when the process finishes still 312MB are used by process.

[JorjMcKie]

What is your OS and PyMuPDF version time stamp?
I am on Win 10 x64 and used the following session (Python 3.6.4 x64).

• I created pixmaps for all 1310 pages of the Adobe manual.
• after open I had 40 MB memory usage:
  
• at the end, this had increased to 94 MB:
  
• deleting the leftover page and pixmap objects from the loop, this dropped to 92.5 MB:
  
• after closing the document, memory usage went down to 26.4 MB (below my start value 40 MB after open):
  

So from my end, I find your 400 MB confusing. I will now go ahead and also create PNG images from each pixmap to see what that impact is ...

[JorjMcKie]

Hmm, no big change either:

• I am going up to 97 MB now
  

[JorjMcKie]

Maybe I should be using an example with larger pages than this one? Let me try more documents ...

[JorjMcKie]

With larger page sizes (100 A4 pages, complex graphics) I am also staying around 100 MB.
It still is hard to explain, why I should have an increase at all after the first pixmap ...

[JorjMcKie]

I did some MuPDF store memory analysis: Obviously, their strategy is to maintain stuff in memory as much as possible for performance reasons.
For example, any font or image once encountered is held in memory. This explains why memory usage goes up when more pages are processed - even though a pixmap is properly deleted.

There is an upper limit for the so-called global context (256 MB), which is specified in fitz_wrap.c. The storable memory is part of this global context.

There do exist low-level functions to exert some control, too. Like deleting some storable memory by percentage value and so on. Todate not implemented in PyMuPDF.