Clarifying the 'under the hood' data download (and other GIS stuff) #1325
Comments
|
I second this. We plan to use cartopy for a flight planning tool used in remote locations with bad internet connection and want to automatically install it and the necessary data fully before going abroad. |
|
@jypeter I just encountered the same issue working on a supercomputer. Did you happen to figure out a workaround for this? |
|
+1 for this. I recently tried to guide a supercomputer user through this issue by debugging and picking apart the various functions. I missed some steps, confusing the situation So a thorough guide would be very helpful |
|
I had completely forgotten about that question until you guys asked about it yesterday, and I also had to plot a map on my laptop during a conference and it triggered the download on my laptop (WiFi was fortunately working). I have just tried to find out what was downloaded:
So the workaround would probably be to copy those files from a computer with network access to the I'd rather have a solution where cartopy first checks if the data files are available in a 'conda installation' centralized location, and then checks the ~/.local directory @bjlittle any ideas here? |
|
@jypeter I have all of the files in ~/.local/share/cartopy/shapefiles/natural_earth/physical and still no luck unfortunately. I've verified that my cartopy.config['data_dir'] is pointing to that directory as well. |
|
#1072 is relevant here too. Looks like they had some success with the above method, but it's not working for me... |
|
@cpatrizio88 maybe you can try the download tool mentioned in Location of stored offline data for cartopy We probably have to play with this, and possibly use However, when installing cartopy with conda, I'm not too sure how to initialize cleanly I still think there should be a documentation page on the cartopy site connecting all the dots for this. Or maybe it is there and I have not found it yet |
|
@jypeter thanks for your suggestion. I got this working using the following steps:
And that's all! This will allow m.add_feature(cart.feature.LAND) and m.add_feature(cart.feature.COASTLINE) to work offline. I'm sure the steps are similar for other cartopy features. Just a note that setting the cartopy.config['data_dir'] to exactly where the shape files are being stored did not work for me. This made the problem more difficult than necessary I think. Your point about having multiple users point to the same data directory is well taken though. I was just happy to get it working for myself for now. |
|
Just here to say that setting # Download some NaturalEarth data for cartopy
ENV CARTOPY_DIR=/usr/local/cartopy-data
ENV NE_PHYSICAL=${CARTOPY_DIR}/shapefiles/natural_earth/physical
RUN mkdir -p ${NE_PHYSICAL}
RUN wget https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_coastline.zip -P ${CARTOPY_DIR}
RUN apt-get -yq install unzip
RUN unzip ${CARTOPY_DIR}/ne_10m_coastline.zip -d ${NE_PHYSICAL}
RUN rm ${CARTOPY_DIR}/*.zipAnd then in Python: import os
import cartopy
cartopy.config['data_dir'] = os.getenv('CARTOPY_DIR', cartopy.config.get('data_dir'))And then the repeated download is avoided (except for the initial one when the image is built). |
|
The above instructions haven't worked for me and this is a problem. I have several big customers who absolutely can't connect to the internet and need mapping features. They have the maps in their .local/share/cartopy/... directory, I've verified that. I've tried setting the pre_existing_data_dir and that didn't seem to do anything. And I second jypeter's suggestion that the documentation on this is lacking. Here's their stack trace, there might be formatting strangeness, I had convert a pdf of a scan to text... `/opt/anaconda/5.3.0/lib/python3.6/site-packages/cartopy/io/init.py:260: DownloadWarning: Downloading: http://naciscdn.org/naturalearth/110m/physical/ne_110m_ocean.zip File "/opt/anaconda/5.3.0/lib/python3.6/urllib/request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) During handling of the above exception, another exception occurred: Traceback (most recent call last): |
|
Note that if the 'pre_existing_data_dir' config key value is set in ../cartopy/init.py, then that will be the location cartopy looks for first. if you set that key value for a non-networked machine, then it should prevent attempting to download, as long as the shapefiles cartopy is looking for are copied to that directory. You can of course in-line set it on each script run with cartopy.config['pre_existing_data_dir'] = path. I personally had issues getting this to work with the conda-forge build on windows, documented in #1435. Using Christoph Golhke's build for windows, it works fine. I don't have access to a non-windows machine to test, but given my recent difficulties with this same issue but on Windows, something tells me there's a conda recipe issue somewhere along the way. |
|
Hello guys, today go through the same problem with you in an application deployment that contained Cartopy on a server. I tried to download the .zip file and unzip it in the respective directory (as I had seen in slacks), however I was not successful. As another attempt I accessed the files present in the same directory on my machine (which is a Windows), and the files present in that folder I copied to the directory, and oddly enough it worked for me. "ne_110m_coastline.dbf, ne_110m_coastline, ne_110m_coastline.shx, ne_110m_land.dbf, ne_110m_land, ne_110m_land.shx, ne_110m_ocean.dbf, ne_110m_ocean e ne_110m_ocean.shx." So, if you want to test, open the directory in Windows and copy the files to the respective application folder you want, to gain access to the files offline. |
|
Hello, I had the same issue in a cluster with no outside network connection. The best solution for me is to download data using Then all users of the Cartopy package can plot their maps without problems. |
|
We have developed a workaround for supplying the map files for systems that cannot download the files, however it looks like webpage that supplies these images has been down all day: https://naciscdn.org Update: I learned of the script that is provided with the cartopy source code to obtain the maps. I also found the new location of the files that are automatically downloaded by cartopy: Will a patch be issued to cartopy to update these URLs? |
|
@georgemccabe Not sure whether this S3 bucket is meant to replace the existing CDN - I think it's just an alternate source. That being said, my bet is always that S3 will be more reliable than just about any other source, so I'm updating my build systems to drop in this custom cartopy config: _SOURCE_TEMPLATE = 'https://naturalearth.s3.amazonaws.com/{resolution}_{category}/ne_{resolution}_{name}.zip'
def update_config(config):
"""Configures cartopy to download NaturalEarth shapefiles from S3 instead
of naciscdn."""
from cartopy.io.shapereader import NEShpDownloader
target_path_template = NEShpDownloader.default_downloader().target_path_template
downloader = NEShpDownloader(url_template=_SOURCE_TEMPLATE,
target_path_template=target_path_template)
config['downloaders'][('shapefiles', 'natural_earth')] = downloaderMy deployment method: usersitedir=$(python -c 'from __future__ import print_function; import site; print(site.getusersitepackages())')
mkdir -p "$usersitedir"/cartopy_userconfig
cp THE_PYTHON_SCRIPT_ABOVE.py "$usersitedir"/cartopy_userconfig/__init__.py |
|
@acarapetis, I believe it is meant to replace it reading through this issue: nvkelso/natural-earth-vector#445 |
|
Thanks for an excellent library! I had a question w.r.t. downloading the ne_shaded vhigh raster. When I run this command |
|
Related to this issue, the feature download script help is wrong about the default location: cartopy/tools/cartopy_feature_download.py Lines 115 to 117 in c184ead It's the user data dir, not the user cache dir. And even then, it's only the user data dir according to XDG, not (necessarily) according to the OS. That is ok, but an alternative could be to use platformdirs. In any case, as the OP suggests, it could be useful to document somewhere what |
|
I have a general question (hopefully related to this thread). Is it possible to use the |
I have just used
ax_plot.add_feature(cfeature.LAND)for the first time, which triggered an external data downloadThis went quite well, but it reminded me that such a download recently failed when using cartopy on a supercomputer with no outside world access. I was in a hurry and did not write down the error message, so I can't say if it was friendly and informative or not (I figured out what it was about, and a workaround, but you want to make this easy for all users)
Anyway, I think it would be nice to add a page somewhere about:
cartopy.config['data_dir']'/shapefiles/natural_earth/physical', more precisely in a hidden directory~/.local/share/cartopy/shapefiles/natural_earth/physical)And you could then add a link to this new page from The cartopy Feature interface page and other relevant places
The text was updated successfully, but these errors were encountered: