Downloading ERA5 hourly timeouts

[jonschwenk]

Hi Freddy, nice work on Caravan!

We're using our CAMELS-factory "Veins of the Earth" to create a CAMELS-Arctic, and I had a couple of questions about downloading the ERA5-Land hourly datasets. Even for single, small basins I am timing out or getting Memory Errors on GEE for fetching any given ERA5 bands. I see a few tricks you're using to help with this but wanted to doublecheck that I understand the implications of these:

• polygon simplification: it appears that you used a value of 1.0 (polygons defined in degrees, presumably), which definitely simplifies them but might be too much? Top image is 0.01 degree simplification, bottom is 1.0.
  
  
  I just wanted to be sure that I am interpreting the 1.0 correctly and that you're ok with that level of simplification (not throwing shade, just checking).

• area thresholding: I didn't do a deep dive into your code, but it appears you ignore basins with drainage areas between 200 and 2000 km^2--is my reading of this correct? If so, what's your procedure for basins larger than the upper bound? The distribution of drainage areas for arctic gages has maybe 1/3 to 1/2 of them above 2000 km^2, so we don't want to just throw these away.

We ingested HydroATLAS and do those static characteristic samplings in PostGIS + Python, so I'm really only interested in how these choices affect the ERA-Land fetching if that helps.

I was trying to avoid batching but I don't think I'll be able to. We have an initial set of ~1,500 gages/basins across the Arctic and can't even get one to complete one ERA5 band for the full 1981-2022 period.

[kratzert]

Hi @jonschwenk

Thanks for reaching out. Regarding your points:

• The simplification that I applied to CAMELS US and CAMELS CL polygons did not use a 1.0 degree threshold but a 1.0 meter threshold. This is the default for me in QGIS and I assumed that this (meters instead of degree) is the default everywhere. Seems like not, so I should
  probably clarify. Here is e.g. the result of the simplification for one CAMELS US basins. Purple is after the simplification and blue is the original CAMLES US shape. As you see, there is barely any difference. I definitely do not recommend to use a 1.0 degree threshold for this process.

• Currently, our code only processes polygons with an area between 100km2 and 2000km2 (see paper for justification) but this upper threshold of 2000km2 will be removed. I do not recommend to use ERA5-Land and, especially, HydroATLAS with basins smaller than 100km2, as, especially for HydroATLAS, this results in large uncertainties (due to resolution mismatch). You can already remove the upper threshold from the notebook (or increase the integer value) and it should work just fine. E.g. the Caravan - Denmark extension includes basins > 2000km2. I was also able to process larger basins with the existing code. There is only a problem if the area gets too large, as then the number of HydroATLAS polygons that intersect with the polygon surpass the user limit. But this should only happen for very large basins (in the area of upper hundred thousand sqkm). We have a solution for this as well, but it will take some more time before this is publicly available.

So to summarize, not really sure what is causing your problem. I processed over 10k basins already with the code (usually in batches of 500-700) and never had problems. Do you mind sending me the shapefile (or just a subset of the polygons) so that I can take a look at it?

[jonschwenk]

Ah, 1 meter is quite a bit different than 1 degree 😅 . We weren't planning on going smaller than 100 km^2, but a few of our basins are over 100,000 km^2 (probably we will exclude these, but they meet our initial "arctic" criteria). I've already processed all the HydroAtlas characteristics locally, so no GEE issues there. I will run some of them with your code too and compare against my local results to verify that we're doing things the same way. Will let you know if I see any issues.

To be clear, I was having time out problems when trying to run without batching, i.e. not using your script. I ran yours a couple days ago and was able to fetch the full time series for a few test basins, so there's no issue with your method. I was just looking for clarity on the simplifications to understand if a non-batching approach will work (it won't). Thanks for the clarifications!