Science Base URL Error

[MattReimer]

Science base has the wrong URLS so we're getting missions data in our patchwork DEMs that we build for RSContext

Sometimes this fails in Cybercastor and sometimes it just produces blank data

1. Implement a fix where we use a lookup table after scraping the S3 bucket.
2. Make sure we fail if the DEM Raster coverage is less than X% (I'd say 85% but use whatever value you think makes sense).
   • Make sure this still works with our HUCs that cross the Canadian border.

[KellyMWhitehead]

Summary/context:

1. DEM's are stored in s3 folders that are indexed spatially by lat/long.

2. USGS periodically updates the dems and adds them to the folder. They appear in the S3 buckets with the date appended in the filename. Several versions of the dem can appear in the folder.

3. When we spatial query Science base for the intersecting huc dems, we are getting every DEM, which includes the all versions for a particular lat/long. Fortunately GDAL doesn't fuss about this and mosaics them together in the vrt file.

4. However, occasionally science base has returned non-existent data /bad urls.

5. a separate issue of missing data (referenced in the image) was related to producing a processing extent that allowed us to generate riverscapes projects for Canadian-bounding hucs. This is fixed in 6adf397

Fixes:

• Lookup table for cleaning bad urls (#4) implemented in 5b2f6f5
• Added a check to make sure dem data covers at least 85% of the huc boundary. This may cause some Canadian-Bounding hucs to fail, but we can keep an eye out for these.

[joewheaton]

Thanks @KellyMWhitehead this is helpful. Can you confirm whether or not fixes have been made in the impacted Mississippi HUCs so that we have the DEMs now, or do you still need @lauren-herbine and I to flag those for you?