MdbClasses using radis.api

[HajimeKawahara]

I made api.py that uses radis.api (common api prototype, see #256 #257 ) as ExoMol, HITEMP, HITRAN database I/O.
This capability currently needs to install radis common-api branch.

@erwanp
After resolving this PR, should we merge [common-api branch](https://github.com/radis/radis/tree/add/common-api) to develop branch in radis? or any other solution?

[HajimeKawahara]

included #273

[chonma0ctopus]

In my environment, MdbExomol seems to cause the following AttributeError. Could you please check?

mdb = MdbExomol(".database/CO/12C-16O/Li2015", nurange=[4200.0, 4300.0], crit=1.e-30)

↓

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tako/miniconda3/lib/python3.9/site-packages/exojax-1.1.3-py3.9.egg/exojax/spec/api.py", line 133, in __init__
    load_mask = self.compute_load_mask(df)
  File "/home/tako/miniconda3/lib/python3.9/site-packages/exojax-1.1.3-py3.9.egg/exojax/spec/api.py", line 145, in compute_load_mask
    self.Ttyp, df.Sij0, df.nu_lines, df.elower,
  File "/home/tako/miniconda3/lib/python3.9/site-packages/vaex/dataframe.py", line 284, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrameLocal' object has no attribute 'elower'

Note that I did the following in the radis directory cloned from github.

$ git pull origin add/common-api
$ python setup.py install

[erwanp]

Hello, it may be because you require a different Radis branch than the develop version. Can you checkout the add/common-api branch ? https://github.com/radis/radis/tree/add/common-api

[chonma0ctopus]

Thank you very much for the suggestion Erwan, but I already checked out the common-api branch.

[chonma0ctopus]

The problem was the existence of hdf5 files created in the past (by previous version of mdb modules). The error was solved by deleting the relevant file from the .database.
Thank you.

[chonma0ctopus]

MdbHitemp and MdbHitran cause the ValueError below.
Once I delete the ~/radis.json, it runs without the ValueError only once, but the next time the same code is run, the ValueError occurs again.

For example,

from exojax.spec.api import MdbHitemp
mdb = MdbHitemp(".database/CO/05_HITEMP2019/05_HITEMP2019.par.bz2", nurange=[4200.0, 4300.0], crit=1.e-30)

↓

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_4122958/3592366008.py in <module>
      1 from exojax.spec.api import MdbHitemp
----> 2 mdb = MdbHitemp(".database/CO/05_HITEMP2019/05_HITEMP2019.par.bz2",
      3                  nurange=[4200.0, 4300.0],
      4                  crit=1.e-30)

~/miniconda3/lib/python3.9/site-packages/exojax-1.1.3-py3.9.egg/exojax/spec/api.py in __init__(self, path, nurange, margin, crit, Ttyp, isotope, gpu_transfer)
    249         load_wavenum_max = self.nurange[1] + self.margin
    250 
--> 251         super().__init__(
    252             molecule=self.simple_molecule_name,
    253             name="HITEMP-{molecule}",

~/miniconda3/lib/python3.9/site-packages/radis-0.13.0-py3.9-linux-x86_64.egg/radis/api/hitempapi.py in __init__(self, name, molecule, local_databases, engine, verbose, chunksize, parallel)
    118         parallel=True,
    119     ):
--> 120         super().__init__(
    121             name,
    122             molecule,

~/miniconda3/lib/python3.9/site-packages/radis-0.13.0-py3.9-linux-x86_64.egg/radis/api/dbmanager.py in __init__(self, name, molecule, local_databases, engine, verbose, parallel, nJobs, batch_size)
    135                     .startswith(abspath(expanduser(local_databases).lower()))
    136                 ):  # TODO: replace with pathlib
--> 137                     raise ValueError(
    138                         f"Databank `{self.name}` is already registered in radis.json but the declared path ({registered_path}) is not in the expected local databases folder ({local_databases}). Please fix/delete the radis.json entry, change the `databank_name`, or change the default local databases path entry 'DEFAULT_DOWNLOAD_PATH' in `radis.config` or ~/radis.json"
    139                     )

ValueError: Databank `HITEMP-{molecule}` is already registered in radis.json but the declared path (/home/tako/ghR/exojax/tests/.database/CO/05_HITEMP2019/CO-05_HITEMP2019.hdf5) is not in the expected local databases folder (.database/CO/05_HITEMP2019). 
Please fix/delete the radis.json entry, change the `databank_name`, or change the default local databases path entry 'DEFAULT_DOWNLOAD_PATH' in `radis.config` or ~/radis.json

Please note that "/home/tako/ghR/exojax/tests/.database/CO/05_HITEMP2019/CO-05_HITEMP2019.hdf5" does indeed exist.

[chonma0ctopus]

Rewriting this line with

local_databases="/",

looks resolve the above error for MdbHitemp.
Is it ok?

[chonma0ctopus]

In contrast, rewriting this line to local_databases="/", still left MdbHitran with a PermissionError as below.

from exojax.spec.api import MdbHitran
mdb = MdbHitran(".database/CO/05_hit12.par", nurange=[4200.0, 4300.0], crit=1.e-30) 

↓

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
/tmp/ipykernel_4125346/607486000.py in <module>
      1 from exojax.spec.api import MdbHitran
----> 2 mdb = MdbHitran(".database/CO/05_hit12.par",
      3                  nurange=[4200.0, 4300.0],
      4                  crit=1.e-30) 

~/miniconda3/lib/python3.9/site-packages/exojax-1.1.3-py3.9.egg/exojax/spec/api.py in __init__(self, path, nurange, margin, crit, Ttyp, isotope, gpu_transfer)
    502         download_files = self.get_missing_files(local_file)
    503         if download_files:
--> 504             self.download_and_parse(download_files,
    505                                     cache=True,
    506                                     parse_quanta=True)

~/miniconda3/lib/python3.9/site-packages/radis-0.13.0-py3.9-linux-x86_64.egg/radis/api/hitranapi.py in download_and_parse(self, local_file, cache, parse_quanta)
   1183         # Use HAPI only to download the files, then we'll parse them with RADIS's
   1184         # parsers, and convert to RADIS's fast HDF5 file formats.
-> 1185         isotope_list, data_file_list, header_file_list = download_all_hitran_isotopes(
   1186             molecule, tempdir
   1187         )

~/miniconda3/lib/python3.9/site-packages/radis-0.13.0-py3.9-linux-x86_64.egg/radis/api/hitranapi.py in download_all_hitran_isotopes(molecule, directory)
   1141             from radis.misc.basics import make_folders
   1142 
-> 1143             make_folders(*split(abspath(directory)))
   1144 
   1145             db_begin(directory)

~/miniconda3/lib/python3.9/site-packages/radis-0.13.0-py3.9-linux-x86_64.egg/radis/misc/basics.py in make_folders(path, folders)
     37         folders = [folders]
     38     for folder in folders:
---> 39         os.makedirs(
     40             join(path, folder), exist_ok=True
     41         )  # makedirs create folders recursively

~/miniconda3/lib/python3.9/os.py in makedirs(name, mode, exist_ok)
    213     if head and tail and not path.exists(head):
    214         try:
--> 215             makedirs(head, exist_ok=exist_ok)
    216         except FileExistsError:
    217             # Defeats race condition when another thread created the path

~/miniconda3/lib/python3.9/os.py in makedirs(name, mode, exist_ok)
    223             return
    224     try:
--> 225         mkdir(name, mode)
    226     except OSError:
    227         # Cannot rely on checking for EEXIST, since the operating system

PermissionError: [Errno 13] Permission denied: '/downloads__can_be_deleted'

[erwanp]

@chonma0ctopus your error messages refer to Radis 0.13 in the python site-packages. I think that maybe, you have two Radis on your computer. The one cloned and another one installed by pip.
Can you go back to your Radis local folder (with the common-api branch) and use

pip install -e .

(dont forget the dot)
This is the same as python setup.py develop except it also cleans previously installed versions

Edit : actually the public version doesn't have hitranapi.py. So you're using the right version and the error is somewhere else

[erwanp]

@chonma0ctopus I've been looking into your problem. It requires some decisions. We'll need @HajimeKawahara input too.

Basically the question is how should you identify a database.

In ExoJax, the user specifies the path of the database file. The file is downloaded if it doesn't exist. The name of the molecule is infered from the database file.
In Radis, the database file is not given by the user, but the molecule name is. Then Radis chooses the corresponding filename.

Do you want to keep ExoJax as it currently is ? For instance, if computing a new molecule; let's say acetylene C2H2, what would be the user script (and how does the user find it ) ?
It's possible to keep both options, but will require some tiny changes in the current API.
For instance, RADIS uses some get_filenames() functions ; which should be overriden in the Exojax variant by just returning the path initially given by the user.

On top of that, Radis currently systematically registers all files into the ~/radis.json config file. Although I find it convenient, I think it's a Radis-specific feature that shouldn't be forced to other codes, and so it should not be included in the common-API. Some of your current errors are due to the common API trying to read from this file. We'll remove this. The easiest option is probably a "register_database" boolean which would be False by default, and True only in Radis.

I also pushed some changes to radis/common-api related to relative/absolute path, it might help. Please pull .

[HajimeKawahara]

@erwanp @chonma0ctopus
Thank you for identifying the problem.

In our field, we frequently examine the differences in results between databases. This is because some molecules are detected in the new database while they were not detected in the old database. That's why ExoJAX specifies the database file directly.
But, yes the molecular name is more user-friendly for general purposes. So ideally, both options in the common API would be great, I think. But a bit difficult to implement?

[chonma0ctopus]

@erwanp Thank you for looking into the problem and fixing it. I no longer get the ValueError and PermissionError reported earlier.

One thing that still bothers me a little is that the code

from exojax.spec.api import MdbHitemp
mdb = MdbHitemp(".database/CO/05_HITEMP2019/05_HITEMP2019.par.bz2", nurange=[4200.0, 4300.0], crit=1.e-30)
mdb = MdbHitemp(".database/CO/05_HITEMP2019/05_HITEMP2019.par.bz2", nurange=[4200.0, 4300.0], crit=1.e-30)

still raises a ValueError below:

ValueError: Database HITEMP-{molecule} is inconsistent : it should be stored in 
/home/tako/work/.database/co/05_hitemp2019 but files registered in ~/radis.json contains 
/home/tako/work/.database/CO/05_HITEMP2019/CO-05_HITEMP2019.hdf5. Please fix or delete the ~/radis.json entry.

After deleting the ~/radis.json, using the lowercase for the directory names like

from exojax.spec.api import MdbHitemp
mdb = MdbHitemp(".database/co/05_hitemp2019/05_HITEMP2019.par.bz2", nurange=[4200.0, 4300.0], crit=1.e-30)
mdb = MdbHitemp(".database/co/05_hitemp2019/05_HITEMP2019.par.bz2", nurange=[4200.0, 4300.0], crit=1.e-30)

circumvents the error. There seems to be some trouble with case sensitivity?

Regarding the database identification, I would also prefer to retain both option of the database designation and the molecular name designation, if possible.

[erwanp]

Hello thanks for the explanation. Then we'll make sure that the common-api does not require (it will be optional) the molecule name, and the registration, and therefore no radis.json.
Local Hdf5 fast-access file would be generated alongside the user given path, or in a local_databases folder if given by the user.

I can have a look next weekend

[HajimeKawahara]

Confirmed to work with radis / develop branch.

[erwanp]

Great! I can publish RADIS 0.14 which will include the changes.

[HajimeKawahara]

That's nice! I'm gonna set up CI using github action and radis.api in RADIS 0.14 makes it easy.