access failure when trying to download opendap from an specific node.

[karimnorouzi]

Hi,
I am trying to download cordex data sets. I have created an account on esg-dn1.nsc.liu.se data node.
My openID is:
'https://esg-dn1.nsc.liu.se/esgf-idp/openid/XXXXX'

I use pyclient to download series of simulations for a location (lat, lon). My search is sucessful, how ever, why I try to get the download, I get a Access Failure message.
could you please let me what part of my scripts I am doing wrong ?
I successfully logon using my openid and password.

here is the script:

from pyesgf.search import SearchConnection
conn = SearchConnection('https://esg-dn1.nsc.liu.se/esg-search', distrib=True)
ctx = conn.new_context(
project='CORDEX',
variable = ['pr'],
time_frequency = '3hr',
domain = 'MNA-44',
data_node = 'esg-dn1.nsc.liu.se'
)
ctx.hit_count
rslts = ctx.search()

urls = [] # get the urls here.
for r in rslts:
files = r.file_context().search()
for file in files:
if file.opendap_url is not None:
urls.append(file.opendap_url)

for url in urls:
path, filename = os.path.split(url)
print('downloading {}'.format(filename))
lat_v = 29.639659
lon_v = 52.569935
ds = xr.open_dataset(url)
**_

data = ds['pr']

_** ,-------------------------gives the Access Failure errror
da = data.sel(rlat=lat_v, rlon=lon_v, method = 'nearest')

        da.to_netcdf(filename)
        print('saved file {}'.format(filename))

ERROR MESSAGE:

OSError: [Errno -77] NetCDF: Access failure: b'http://esg-dn1.nsc.liu.se/thredds/dodsC/esg_dataroot3/cordexdata/cordex/output/MNA-44/SMHI/CNRM-CERFACS-CNRM-CM5/rcp85/r1i1p1/SMHI-RCA4/v1/3hr/pr/v20180109/pr_MNA-44_CNRM-CERFACS-CNRM-CM5_rcp85_r1i1p1_SMHI-RCA4_v1_3hr_200601010130-200612312230.nc'

[karimnorouzi]

I like to add that this works with other nodes. Only when I try to download cordex data set on this server, the script fails.
Many thanks.

[alaniwi]

This looks like not an issue with esgf-pyclient, which is only being used here to find the URL from the search index.

Regarding access to the URL itself, I am not succeeding with xarray but that is probably just me doing something wrong as I'm insufficient familiarity with xarray, but using netCDF4 I can open the dataset after creating an appropriate .dodsrc file.

>>> import netCDF4
>>> url
'http://esg-dn1.nsc.liu.se/thredds/dodsC/esg_dataroot3/cordexdata/cordex/output/MNA-44/SMHI/CNRM-CERFACS-CNRM-CM5/rcp85/r1i1p1/SMHI-RCA4/v1/3hr/pr/v20180109/pr_MNA-44_CNRM-CERFACS-CNRM-CM5_rcp85_r1i1p1_SMHI-RCA4_v1_3hr_200601010130-200612312230.nc'
>>> ds = netCDF4.Dataset(url)
>>> ds.variables.keys()
odict_keys(['rlat', 'rlon', 'rotated_pole', 'time', 'time_bnds', 'lat', 'lon', 'pr'])

where .dodsrc references a short-lived user credential (I'm registered for CORDEX access under that identity)

$ cat ~/.dodsrc
HTTP.COOKIEJAR=/home/vagrant/dods_cookies
HTTP.SSL.CERTIFICATE=/home/vagrant/credentials.pem
HTTP.SSL.KEY=/home/vagrant/credentials.pem

I'm guessing that either you weren't presenting a valid certificate, or there was a temporary problem with the server.

Note that the cookie jar file doesn't have to exist already - you point at the location and it gets created for you the first time.