Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load_earth_relief() failed to find earth relief files #972

Closed
core-man opened this issue Feb 26, 2021 · 15 comments
Closed

load_earth_relief() failed to find earth relief files #972

core-man opened this issue Feb 26, 2021 · 15 comments
Labels
bug Something isn't working

Comments

@core-man
Copy link
Member

Description of the problem

As I am working on #966, I first ran the examples on pygmt.datasets.load_earth_relief. However, it seems that load_earth_relief() may have a bug to find low-resolution earth relief files, which seems to be related to which.

Full code that generated the error

>>> from pygmt.datasets import load_earth_relief
>>> grid = load_earth_relief()

Full error message

gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]
gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 425, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/datasets/earth_relief.py", line 111, in load_earth_relief
    fname = which(f"@earth_relief_{resolution}{reg}", download="a")
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 281, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/src/which.py", line 55, in which
    raise FileNotFoundError("File '{}' not found.".format(fname))
FileNotFoundError: File '@earth_relief_01d' not found.

I also tested high-resolution earth relief. load_earth_relief() uses grdcut to handle it instead of which.

>>> from pygmt.datasets import load_earth_relief
>>> grid = load_earth_relief("05m", region=[120, 160, 30, 60], registration="gridline")
grdblend [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]
grdblend [NOTICE]: Earth Relief at 5x5 arc minutes from Gaussian Cartesian filtering (9 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
grdblend [NOTICE]:   -> Download 180x180 degree grid tile (earth_relief_05m_g): S90E000

Both high- and low-resolution earth relief files have been downloaded.

# Check GMT Earth data directory
$ ls ~/.gmt/server/earth/earth_relief
earth_relief_01d_p.grd  earth_relief_05m_g
$ ls ~/.gmt/server/earth/earth_relief/earth_relief_05m_g 
S90E000.earth_relief_05m_g.nc

Then I used gmt which command in the terminal to check those files:

# High resolution @earth_relief_05m_g
$ gmt which @earth_relief_05m_g                 
gmtwhich [ERROR]: Tile @S90W180.earth_relief_05m_g.nc not found!
/home/core-man/.gmt/server/earth/earth_relief/earth_relief_05m_g/S90E000.earth_relief_05m_g.nc
$ gmt which @S90E000.earth_relief_05m_g.nc
/home/core-man/.gmt/server/earth/earth_relief/earth_relief_05m_g/S90E000.earth_relief_05m_g.nc

# Low resolution @earth_relief_01d
$ gmt which @earth_relief_01d_p.grd                            
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ gmt which @earth_relief_01d_p    
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ gmt which @earth_relief_01d  
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

System information

Please paste the output of python -c "import pygmt; pygmt.show_versions()":

PyGMT information:
  version: v0.3.1.dev34+g4db28ef
System information:
  python: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46)  [GCC 9.3.0]
  executable: /home/core-man/.anaconda3/envs/pygmt/bin/python
  machine: Linux-3.10.0-862.11.6.el7.x86_64-x86_64-with-glibc2.17
Dependency information:
  numpy: 1.20.1
  pandas: 1.2.2
  xarray: 0.16.2
  netCDF4: 1.5.6
  packaging: 20.9
  ghostscript: 9.53.3
  gmt: 6.1.1
GMT library information:
  binary dir: /home/core-man/.anaconda3/envs/pygmt/bin
  cores: 8
  grid layout: rows
  library path: /home/core-man/.anaconda3/envs/pygmt/lib/libgmt.so
  padding: 2
  plugin dir: /home/core-man/.anaconda3/envs/pygmt/lib/gmt/plugins
  share dir: /home/core-man/.anaconda3/envs/pygmt/share/gmt
  version: 6.1.1
@core-man core-man added the bug Something isn't working label Feb 26, 2021
@welcome
Copy link

welcome bot commented Feb 26, 2021

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. You might also want to take a look at our contributing guidelines and code of conduct.

@seisman
Copy link
Member

seisman commented Feb 26, 2021

Could you please post the output of

gmt --version
gmt which @earth_relief_01d_p.grd -Vd

@core-man
Copy link
Member Author

@seisman

$ gmt --version
6.1.1
$ gmt which @earth_relief_01d_p.grd -Vd
gmt [DEBUG]: Obtained the ppid from parent: 105618
gmt [DEBUG]: Enter: gmtinit_new_GMT_ctrl
gmt [DEBUG]: GMT->session.SHAREDIR = /home/core-man/.anaconda3/envs/pygmt/share/gmt
gmt [DEBUG]: GMT->session.HOMEDIR = /home/core-man
gmt [DEBUG]: GMT->session.USERDIR = /home/core-man/.gmt [created]
gmt [DEBUG]: GMT->session.CACHEDIR = /home/core-man/.gmt/cache [created]
gmt [DEBUG]: GMT: 0. Will try to find subdir=postscriptlight stem = PSL_custom_fonts suffix=.txt
gmt [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
gmt [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /home/core-man/.gmt
gmt [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /home/core-man/.gmt/postscriptlight
gmt [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /home/core-man/.anaconda3/envs/pygmt/share/gmt/postscriptlight
gmt [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /home/core-man/.anaconda3/envs/pygmt/share/gmt
gmt [DEBUG]: GMT: 6. gmt_getsharepath failed
gmt [DEBUG]: Map distance calculation will be Cartesian
gmt [DEBUG]: Exit:  gmtinit_new_GMT_ctrl
gmt [DEBUG]: Enter: New_PSL_Ctrl
gmt [DEBUG]: Exit:  New_PSL_Ctrl
gmt [DEBUG]: Enter: gmt_manage_workflow
gmt [DEBUG]: Exit : gmt_manage_workflow
gmt [DEBUG]: Enter: PSL_beginsession
gmt [DEBUG]: Exit : PSL_beginsession
gmt [DEBUG]: Enter: PSL_setdefaults
gmt [DEBUG]: Exit : PSL_setdefaults
gmt [DEBUG]: Enter: gmtlib_io_init
gmt [DEBUG]: Exit : gmtlib_io_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit:  gmt_hash_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit:  gmt_hash_init
gmt [DEBUG]: Enter: gmt_reload_settings
gmt [DEBUG]: The PROJ_GEODESIC set to Vincenty
gmt [DEBUG]: Look for file /home/core-man/gmt.conf
gmt [DEBUG]: Look for file /home/core-man/.gmt/gmt.conf
gmt [DEBUG]: Look for file /home/core-man/.gmt/server/gmt.conf
gmt [DEBUG]: Look for file /home/core-man/.gmt/cache/gmt.conf
gmt [DEBUG]: Could not find file gmt.conf
gmt [DEBUG]: Exit:  gmt_reload_settings
gmt [DEBUG]: Enter: gmtlib_plot_C_format
gmt [DEBUG]: Exit:  gmtlib_plot_C_format
gmt [DEBUG]: Enter: gmtinit_get_history
gmt [DEBUG]: Initialize FFTW with 8 threads.
gmt [DEBUG]: GMT_Create_Session initialized GMT structure
gmt [DEBUG]: Loading core GMT shared library: libgmt.so
gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.so
gmt [DEBUG]: Loading GMT plugins from: /home/core-man/.anaconda3/envs/pygmt/lib/gmt/plugins
gmt [DEBUG]: Shared Library # 1 (supplements). Path = /home/core-man/.anaconda3/envs/pygmt/lib/gmt/plugins/supplements.so
gmt [DEBUG]: Local file /home/core-man/.gmt/server/gmt_data_server.txt found
gmt [DEBUG]: File /home/core-man/.gmt/server/gmt_data_server.txt less than 24 hours old, refresh is premature.
gmt [DEBUG]: Load contents from /home/core-man/.gmt/server/gmt_data_server.txt
gmt [DEBUG]: Local file /home/core-man/.gmt/server/gmt_hash_server.txt found
gmt [DEBUG]: File /home/core-man/.gmt/server/gmt_hash_server.txt less than 24 hours old, refresh is premature.
gmt [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
gmt [DEBUG]: Revised options: @earth_relief_01d_p -Vd
gmtwhich [DEBUG]: gmtapi_init_export: Passed family = Data Table and geometry = Text
gmtwhich [DEBUG]: Object ID 0 : Registered Data Table Stream 7fae37791400 as an Output resource with geometry Text [n_objects = 1]
gmtwhich [DEBUG]: gmtapi_init_export: Added stdout to registered destinations
gmtwhich [DEBUG]: GMT_Init_IO: Returned first Output object ID = 0
gmtwhich [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Output
gmtwhich [DEBUG]: gmtapi_next_io_source: Selected object 0
gmtwhich [INFORMATION]: Writing Data Table to Standard Output stream
gmtwhich [DEBUG]: GMT_Begin_IO: Output resource access is now enabled [record-by-record]
gmtwhich [DEBUG]: Look for file earth_relief_01d_p.grd in /home/core-man/.gmt
gmtwhich [DEBUG]: Look for file earth_relief_01d_p.grd in /home/core-man/.gmt/cache
gmtwhich [DEBUG]: Look for file earth_relief_01d_p.grd in /home/core-man/.gmt/server
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
gmtwhich [DEBUG]: GMT_End_IO: Output resource access is now disabled
gmtwhich [DEBUG]: gmtlib_unregister_io: Unregistering object no 0 [n_objects = 0]
gmtwhich (gmtlib_free_tmp_arrays): tried to free unallocated memory
gmt [DEBUG]: Entering GMT_Destroy_Session

@seisman
Copy link
Member

seisman commented Feb 26, 2021

It's weird. I don't see anything wrong in the output.

@seisman
Copy link
Member

seisman commented Feb 26, 2021

@PaulWessel Do you have any idea why gmt which can't find the remote file even though it's already downloaded?

@PaulWessel
Copy link
Member

On the face of it, no. But if I delete my copy and run that command I get the same error. I think you need -G if you want to download a file via gmt which though.

@seisman
Copy link
Member

seisman commented Feb 27, 2021

@core-man Is it possible that the earth_relief_01d_p.grd file is not complete due to unstable internet connections? Could you check the file size and md5sum?

@core-man
Copy link
Member Author

core-man commented Mar 1, 2021

@core-man Is it possible that the earth_relief_01d_p.grd file is not complete due to unstable internet connections? Could you check the file size and md5sum?

@seisman

Download earth relief grid via load_earth_relief():

$ python
>>> from pygmt.datasets import load_earth_relief
>>> grid = load_earth_relief()
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]
gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 425, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/datasets/earth_relief.py", line 111, in load_earth_relief
    fname = which(f"@earth_relief_{resolution}{reg}", download="a")
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 281, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/src/which.py", line 55, in which
    raise FileNotFoundError("File '{}' not found.".format(fname))
FileNotFoundError: File '@earth_relief_01d' not found.

Rename the grid to be earth_relief_01d_p_load.grd:

$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd
$ mv ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p_load.grd

Then I can use the grdimage to download and plot @earth_relief_01d.

import pygmt

fig = pygmt.Figure()
fig.grdimage(grid="@earth_relief_01d")
fig.show()
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd  earth_relief_01d_p_load.grd

The two grids are the same.

$ du -h ~/.gmt/server/earth/earth_relief/*
116K	/home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
116K	/home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd

$ diff ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd

$ md5sum ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd
74a884c902015dda516d17605f317efe  /home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
74a884c902015dda516d17605f317efe  /home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd

@core-man
Copy link
Member Author

core-man commented Mar 1, 2021

On the face of it, no. But if I delete my copy and run that command I get the same error. I think you need -G if you want to download a file via gmt which though.

@seisman @PaulWessel
Yes, I can use -G to download and find the grid.

# work in ~/workspace folder
$ cd ~/workspace

# use `-Gl` option
$ ls
$ gmt which -Gl @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls
earth_relief_01d_p.grd

# Use `-Ga` option
$ ls ~/.gmt/server/earth/earth_relief/
$ gmt which -Ga @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

# those two grids are the same
$ diff ./earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd

However, there could still exist some bugs. A strange thing occurred. If the grid is the current folder is deleted and the grid in ~/.gmt/server/earth/earth_relief/ is kept, neither gmt which -Gl nor gmt which -Ga can work.

# I am still in ~/workspace
$ ls
earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

# remove the grid in the current folder
$ rm ./earth_relief_01d_p.grd

# gmt which cannot work
$ gmt which -Gl @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ gmt which -Ga @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

I removed ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd. I then ran commands in the above first window again and they work.

# I am still in ~/workspace which is empty now
$ ls
$ rm ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/

# use `-Gl` option
$ ls
$ gmt which -Gl @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls
earth_relief_01d_p.grd

# Use `-Ga` option
$ ls ~/.gmt/server/earth/earth_relief/
$ gmt which -Ga @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

Finally, I removed the grids in the current folder and ~/.gmt/server/earth/earth_relief/. gmt which -Ga can download the grid but cannot find the grid.

# I am still in ~/workspace
$ rm earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd

$ gmt which @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ ls
$ ls ~/.gmt/server/earth/earth_relief/

$ gmt which -Ga @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

$ ls
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

$ gmt which @earth_relief_01d_p 
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

$ gmt which -Gl @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ ls

In summary:

  1. If there is the same grid in ~/.gmt/server/earth/earth_relief/, gmt which -Gl cannot work.
  2. If there is no the same grid in the current folder, gmt which -Ga can download the grid to ~/.gmt/server/earth/earth_relief but cannot find it.
  3. If there is the same grid in the current folder, gmt which -Ga can download the grid to ~/.gmt/server/earth/earth_relief and can find a grid with the same name.
  4. It seems that the file gmt which -Ga tries to find is the grid in the current folder instead of ~/.gmt/server/earth/earth_relief.

@seisman
Copy link
Member

seisman commented Mar 2, 2021

Thanks for the detailed reports. However, I followed your steps but still can't reproduce your issue. I'm using GMT 6.1.1 (installed using conda) on macOS.

@core-man
Copy link
Member Author

core-man commented Mar 2, 2021

Thanks for the detailed reports. However, I followed your steps but still can't reproduce your issue. I'm using GMT 6.1.1 (installed using conda) on macOS.

I use GMT 6.1.1 (installed using conda) CentOS7. I don't have this issue on my macOS either.

@seisman
Copy link
Member

seisman commented Mar 2, 2021

I just tried GMT 6.1.1 on CentOS 7.8. Still can't reproduce your issue.

@core-man
Copy link
Member Author

core-man commented Mar 2, 2021

I just tried GMT 6.1.1 on CentOS 7.8. Still can't reproduce your issue.

I use CentOS 7.5. I also use the GMT built from the source code (6.2.0_416e03f_2020.08.18) instead of conda and it also has the same bug.

If only I have the bug, it should be okay cause I also work on macOS. Maybe it is caused by an unknown/strange thing in my CentOS. Shall we close this issue or open it until someone also have this issue?

@seisman
Copy link
Member

seisman commented Mar 2, 2021

I will close this one. Feel free to reopen it if someone else can reproduce the issue or have a clue why it doesn't work.

@seisman seisman closed this as completed Mar 2, 2021
@maxrjones
Copy link
Member

I don't have a way to reproduce the issue, but I wanted to point out that this seems similar in some ways to an issue that I had earlier which could not be reproduced by other users: GenericMappingTools/gmt#4522. So, it could be related to the specific user environment rather than the OS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants