Import Issue, ModuleNotFoundError: No module named 'ESMF' / Problem on Jupyter

[mchoblet]

I know this is very similar to an old issue that was previously reported (#246), sorry for reopening it, but I am not able to fix it despite several hours of trying and googling.
For the installation of xesmf, I followed the instructions and created a new environment.

When I try to import xesmf (in a jupyter notebook), I get the following error.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/esmpy/interface/loadESMF.py:26
     25 try:
---> 26     esmfmk = os.environ["ESMFMKFILE"]
     27 except:

File <frozen os>:679, in __getitem__(self, key)

KeyError: 'ESMFMKFILE'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/xesmf/backend.py:22
     21 try:
---> 22     import esmpy as ESMF
     23 except ImportError:

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/esmpy/__init__.py:112
    110 #### IMPORT LIBRARIES #########################################################
--> 112 from esmpy.api.esmpymanager import *
    113 from esmpy.api.grid import *

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/esmpy/api/esmpymanager.py:9
      7 #### IMPORT LIBRARIES #########################################################
----> 9 from esmpy.interface.cbindings import *
     11 from esmpy.api.constants import *

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/esmpy/interface/cbindings.py:13
     12 from esmpy.util.decorators import *
---> 13 from esmpy.interface.loadESMF import _ESMF
     16 def copy_struct(src):

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/esmpy/interface/loadESMF.py:28
     27 except:
---> 28     raise ImportError('The ESMFMKFILE environment variable is not available.')
     30 #### INVESTIGATE esmf.mk ######################################################
     31 
     32 # TODO: look for various dependecies in the ESMF build log
   (...)
     37 #       use this information to set variables that can be checked at beginning
     38 #       of the routines that require an ESMF build with these dependencies

ImportError: The ESMFMKFILE environment variable is not available.

During handling of the above exception, another exception occurred:

ModuleNotFoundError                       Traceback (most recent call last)
Cell In[5], line 1
----> 1 import xesmf

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/xesmf/__init__.py:4
      1 # flake8: noqa
      3 from . import data, util
----> 4 from .frontend import Regridder, SpatialAverager
      6 try:
      7     from ._version import __version__

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/xesmf/frontend.py:12
      9 import xarray as xr
     10 from xarray import DataArray, Dataset
---> 12 from .backend import Grid, LocStream, Mesh, add_corner, esmf_regrid_build, esmf_regrid_finalize
     13 from .smm import (
     14     _combine_weight_multipoly,
     15     _parse_coords_and_values,
   (...)
     19     read_weights,
     20 )
     21 from .util import LAT_CF_ATTRS, LON_CF_ATTRS, split_polygons_and_holes

File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/xesmf/backend.py:24
     22     import esmpy as ESMF
     23 except ImportError:
---> 24     import ESMF
     25 import numpy as np
     26 import numpy.lib.recfunctions as nprec

ModuleNotFoundError: No module named 'ESMF'

Downgrading esmf does not work with my newly created environment as mentioned in the issue, and the problem is also not solved by deactivating/reactivating the environment.

Curiously, I do not get the error when I try to import xesmf via the python console in the terminal.
My Python version is 3.11, but I've also tried with other versions and this also didn't work.

I know this could be something with my python installation on the cluster, but I've checked the usual hacks for using the same python version in jupyter notebook and python console, and they didn't help.

Any help is appreciated. xESMF is such a useful package, I just don't want a miss it in my jupyter notebooks. Thanks!

[aulemahal]

Hi @mchoblet, sorry to hear that... This bug truly is painful.

First of all, in the traceback, the very first error is :

Traceback (most recent call last)
File ~/miniconda3/envs/xesmf_new_2/lib/python3.11/site-packages/esmpy/interface/loadESMF.py:26
     25 try:
---> 26     esmfmk = os.environ["ESMFMKFILE"]
     27 except:

This line of code just doesn't exist in ESMpy 8.3.1 . If you had a similar issue when downgrading, then it was a different issue. If you try downgrading again, I suggest you post the traceback in this discussion so we can see what is this other bug.

You say:

do not get the error when I try to import xesmf via the python console in the terminal

my question is then, what was the other way of importing xesmf ? Is there a way to activate/deactivate an environment that doesn't concern the terminal ?

Because that is what the bug is : activating the environment adds a shell variable that points to some configuration file. Thus, the package (ESMpy) becomes unusable outside a shell with this variable.

One other trick, is to set this environment variable in the code:

import os
from pathlib import Path

if 'ESMFMKFILE' not in os.environ:
    os.environ['ESMFMKFILE'] = str(Path(os.__file__).parent.parent / 'esmf.mk')
 
import xesmf

Where we have to make assumptions about the file structure, but this would usually work in a conda env. This snippet goes at the top of your notebook.

[mchoblet]

Hi @aulemahal, thanks a lot for the help! The trick of setting the environment variable in the jupyter notebook worked with Python 3.11! 🎉
Just for reference: It also requires the pathlib library (from pathlib import Path) to work.

It was really weird that the environment used in the terminal and in the jupyter notebook did something else, no idea, I'm too illiterate concerning environments on the cluster I'm working on unfortunately 😞

Actually, I alternatively also got it to work via the downgrading esmpy solution, I probably did something stupid before. Downgrading esmpy with Python 3.11 won't work despite several conda solving environment intents. With Python 3.10 it took conda also several solving environment intents for almost an hour, but finally it found a way to downgrade esmpy.