Matplotlib 3.3.0 Breaks _png

[ax3l]

Bug report

Bug summary

Matplotlib 3.3.0 removed the internal _png module, which breaks
https://github.com/yt-project/yt/blob/yt-3.6.0/yt/utilities/png_writer.py#L13

See the last mention in https://matplotlib.org/3.3.0/api/api_changes.html#matplotlib-now-uses-pillow-to-save-and-read-pngs

Code for reproduction

Just saw this on our CI:
https://travis-ci.com/github/ECP-WarpX/WarpX/jobs/361956903

MPL 3.3.0 was release about 5hrs ago.
https://github.com/matplotlib/matplotlib/releases/tag/v3.3.0

Actual outcome

File "/home/travis/.local/lib/python3.6/site-packages/yt/utilities/png_writer.py", line 13, in <module>
    import matplotlib._png as _png
ModuleNotFoundError: No module named 'matplotlib._png'

Expected outcome

:-)

Version Information

• Operating System: Ubuntu 18.04
• Python Version: 3.6
• yt version: 3.6.0

Installed via python -m pip install --upgrade cmake matplotlib mpi4py numpy scipy yt.

Work-Around

Downgrade matplotlib via python -m pip install --upgrade matplotlib==3.2.2.

Exact details:

• https://travis-ci.com/github/ECP-WarpX/WarpX/jobs/361956903
• https://github.com/ECP-WarpX/WarpX/blob/384c6ab9a864d430868a39a065f4a1d4426231af/.travis.yml#L30-L31

[welcome]

Hi, and welcome to yt! Thanks for opening your first issue. We have an issue template that helps us to gather relevant information to help diagnosing and fixing the issue.

[neutrinoceros]

Thank you for reporting this !
If I'm not mistaken, the only function that needs to be updated is call_png_write_png since it calls matplotlib._png.write_png.
breadcrumbs: here are some of the iterations matplotlib did itself during their transition to pillow

• Allow saving PNGs through Pillow instead of the builtin _png module. matplotlib/matplotlib#13207
• Allow passing arguments to PIL.Image.save(). matplotlib/matplotlib#13094

Furthermore it looks like they started using pillow as an optional dep since their 3.1.0 release (see release notes).

[neutrinoceros]

and this is the crucial commit of this transition: matplotlib/matplotlib@370e9a2

[matthewturk]

I thought for sure we had addressed this previously. Hmmmm. Thanks for looking at this, will test it out over here...

[neutrinoceros]

FWIW I tested it locally with the mpl 3.3 and it didn't crash, but changing the dpi seems to have no effect (I tried hardcoding a different value since it's not currently exposed).
edit: I though you were answering my pull request instead of the issue itself. My bad.

[ax3l]

@neutrinoceros I have to admit that I just quickly saw this broke our CI, downgraded to mpl 3.2.2 and went on. The full call stack in the linked CI error above is:

  File "outscript.py", line X, in <module>
    import yt
  File "/home/travis/.local/lib/python3.6/site-packages/yt/__init__.py", line 63, in <module>
    from yt.fields.api import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/fields/api.py", line 19, in <module>
    from . import angular_momentum
  File "/home/travis/.local/lib/python3.6/site-packages/yt/fields/angular_momentum.py", line 26, in <module>
    from .vector_operations import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/fields/vector_operations.py", line 32, in <module>
    from yt.geometry.geometry_handler import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/geometry/geometry_handler.py", line 29, in <module>
    from yt.utilities.parallel_tools.parallel_analysis_interface import \
  File "/home/travis/.local/lib/python3.6/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 30, in <module>
    from yt.data_objects.image_array import ImageArray
  File "/home/travis/.local/lib/python3.6/site-packages/yt/data_objects/image_array.py", line 18, in <module>
    from yt.visualization.image_writer import write_bitmap, write_image
  File "/home/travis/.local/lib/python3.6/site-packages/yt/visualization/image_writer.py", line 29, in <module>
    import yt.utilities.png_writer as pw
  File "/home/travis/.local/lib/python3.6/site-packages/yt/utilities/png_writer.py", line 13, in <module>
    import matplotlib._png as _png
ModuleNotFoundError: No module named 'matplotlib._png'

[neutrinoceros]

@ax3l I'm sorry if my last message caused confusion, I was talking about #2754 . You are absolutely right to state that yt is currently not compatible with matplotlib 3.3. ! My aforementioned pull request should resolve the issue, so our next release will contain the fix.
Currently, the next release we're aiming for is 4.0, and will contain a lot of new stuff but also some -minor- backward incompatible changes. We may consider doing a 3.6.1 bugfix release but I'm not in a position to make such a promise.
Anyway thanks again for your early report !

[ax3l]

@neutrinoceros ah, got it now - thanks for the clarification and quick fix!

[munkm]

This should be fixed now on the master branch thanks to #2754

[ax3l]

Thank you for the quick fix!
Are you anticipating a potential 3.6.1 with this? Or maybe just an update pip release that markes mpl<3.3.0 in its meta-data?

[2sn]

I had the same issue. (Matplotlib 3.3.0)

As getting current version of yt from master is also broken

----> 3 from yt.utilities.lib.misc_utilities import (
      4     obtain_position_vector,
      5     obtain_relative_velocity_vector,

ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities'

I just copied the current version of of yt/utilities/png_writer.py from the current master branch and now the import of yt is no longer a roadblock...thanks. (below the file contents in case anyone else needs it)

from io import BytesIO

try:
    # matplotlib switched from an internal submodule _png to using pillow (PIL)
    # between v3.1.0 and v3.3.0
    # So PIL should be available on any system where matplotlib._png doesn't exist
    import matplotlib._png as _png
except ImportError:
    from PIL import Image


def call_png_write_png(buffer, fileobj, dpi):
    try:
        _png.write_png(buffer, fileobj, dpi)
    except NameError:
        Image.fromarray(buffer).save(fileobj, dpi=(dpi, dpi), format="png")


def write_png(buffer, filename, dpi=100):
    with open(filename, "wb") as fileobj:
        call_png_write_png(buffer, fileobj, dpi)


def write_png_to_string(buffer, dpi=100):
    fileobj = BytesIO()
    call_png_write_png(buffer, fileobj, dpi)
    png_str = fileobj.getvalue()
    fileobj.close()
    return png_str

[neutrinoceros]

As getting current version of yt from master is also broken

Hi @2sn , this error probably means you need to recompile yt (this is an expected side effect of how much the master branch evolved lately), so you should be able to fix this with pip install -e . from the top level of the repo !

[2sn]

I did install from scratch using pip3 install -U git+https://github.com/yt-project/yt.git after pip3 uninstall yt. No worries, I know it is an active development version and may have retrieved the snapshot at a bad time.

[awsteiner]

Hello. pip3 install -U git+https://github.com/yt-project/yt.git doesn't seem to work for me, as it just gives the error from above regarding misc_utilities. Is there a different branch with the right fix somewhere?

Type "help", "copyright", "credits" or "license" for more information.

import yt
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/site-packages/yt/init.py", line 63, in
from yt.fields.api import (
File "/usr/local/lib/python3.8/site-packages/yt/fields/api.py", line 2, in
from . import (
File "/usr/local/lib/python3.8/site-packages/yt/fields/angular_momentum.py", line 3, in
from yt.utilities.lib.misc_utilities import (
ModuleNotFoundError: No module named 'yt.utilities.lib.misc_utilities'

[awsteiner]

Oh nvm, the following works...

pip uninstall yt
git clone http://github.com/yt-project/yt.git
cd yt
pip install -e .