Updates to handle virtual files #143
Conversation
tools/ARIAtools/ARIAProduct.py
Outdated
| if any("https://" in i for i in self.files): | ||
| gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','cookies.txt') | ||
| #check if reader is being captured as netcdf | ||
| [s for s in self.files if 'https://' in s][0] |
There was a problem hiding this comment.
L75 might have been used for debugging?
There was a problem hiding this comment.
Yup, will have to remove. Will do now.
| if 'https://' in fname: | ||
| version=str(gdal.Open(fname).GetMetadata_Dict()['version']) | ||
| else: |
There was a problem hiding this comment.
Reading of 'NC_GLOBAL#version' should not be different if in S3.
L122-124 likely to be removed.
To be tested on opensarlabs as it needs the netcdf to be recognized!
There was a problem hiding this comment.
Yup, will have to confirm on opensarlabs, once the technical issue on my end gets resolved. From my end locally though, I could successfully access the version from the URLs this way.
tools/ARIAtools/ARIAProduct.py
Outdated
| if any("https://" in i for i in self.files): | ||
| gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','cookies.txt') | ||
| #check if reader is being captured as netcdf | ||
| [s for s in self.files if 'https://' in s][0] | ||
| fmt=gdal.Open([s for s in self.files if 'https://' in s][0]).GetDriver().GetDescription() | ||
| if fmt != 'netCDF': raise Exception ('Your GDAL install is not configured to handle virtual files') |
There was a problem hiding this comment.
We should also add here a test for a single file which does not have https.
e.g. verify for a local file that it also gets captured with the netcdf reader.
If not netcdf error out and say version >2.3 is needed
There was a problem hiding this comment.
Yes, I will simplify and add in the proposed check for locally downloaded products.
tools/ARIAtools/ARIAProduct.py
Outdated
| if any("https://" in i for i in self.files): | ||
| gdal.SetConfigOption('GDAL_HTTP_COOKIEFILE','cookies.txt') | ||
| #check if reader is being captured as netcdf | ||
| [s for s in self.files if 'https://' in s][0] | ||
| fmt=gdal.Open([s for s in self.files if 'https://' in s][0]).GetDriver().GetDescription() | ||
| if fmt != 'netCDF': raise Exception ('Your GDAL install is not configured to handle virtual files') |
There was a problem hiding this comment.
L77 the error for S3 being h5, could be more specific indicating: Linux kernel >=4.3 and libnetcdf >=4.5
|
@sssangha added a review with comments. |
Sanity check to confirm local files are captured with the netcdf error. Exception statement also updated to be more transparent.
|
@dbekaert I have made some of the suggested changes. Did not change the way the version is read yet though, as I cannot login to opensarlab yet to confirm. |
Added updates as proposed in issue #138
Specifically, updates to appropriately load virtual products and also test whether or not the gdal install is properly configured to handle these virtual product files.
Additionally, files no longer opened with the gdal driver to parse product layer keys. These layer names are now hardcoded and parsed under the "mappingVersion" subfunction, similar to how radar metdata keys were handled in the previous commit #141