Orbit issues RuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token
#634
Comments
|
I will tag @scottstanie and @asjohnston-asf in case they have additional input. Thank you. |
|
Some discussion of the rate-limiting and orbit fetching in #610 may be relevant. |
Is our use of ASF first for the topsApp step avoiding this issue? |
|
The downloads from copernicus for these granules seem to work for me with my account/credentials $ touch S1A_IW_SLC__1SDV_20230204T141536_20230204T141604_047087_05A618_C23D
$ touch S1A_IW_SLC__1SDV_20230204T141602_20230204T141629_047087_05A618_CCEB
$ touch S1B_IW_SLC__1SDV_20211217T141443_20211217T141510_030066_039705_D4FC
$ touch S1B_IW_SLC__1SDV_20211217T141508_20211217T141535_030066_039705_A371
$ eof
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1A on 2023-02-04
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1A on 2023-02-04
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1B on 2021-12-17
[04/02 16:26:01] [INFO download.py] Downloading precise orbits for S1B on 2021-12-17
[04/02 16:26:01] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:02] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:04] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:04] [INFO dataspace_client.py] Querying for AUX_POEORB orbit files from endpoint https://catalogue.dataspace.copernicus.eu/odata/v1/Products
[04/02 16:26:06] [INFO download.py] Attempting download from SciHub
[04/02 16:26:08] [INFO dataspace_client.py] Orbit file downloaded to S1A_OPER_AUX_POEORB_OPOD_20230224T080742_V20230203T225942_20230205T005942.EOF
[04/02 16:26:08] [INFO dataspace_client.py] Orbit file downloaded to S1B_OPER_AUX_POEORB_OPOD_20220106T111538_V20211216T225942_20211218T005942.EOF
[04/02 16:26:09] [INFO dataspace_client.py] Orbit file downloaded to S1A_OPER_AUX_POEORB_OPOD_20230224T080742_V20230203T225942_20230205T005942.EOF
[04/02 16:26:10] [INFO dataspace_client.py] Orbit file downloaded to S1B_OPER_AUX_POEORB_OPOD_20220106T111538_V20211216T225942_20211218T005942.EOF
|
|
I suspect (but haven't tested) that we may be hitting the "100 active sessions" limit documented at https://documentation.dataspace.copernicus.eu/Quotas.html I notice You can see a list of active tokens by logging into CDSE and navigating through My Account -> Security -> Device Activity. hyp3-lib built a class to request CDSE tokens and automatically release them when they go out of scope. If we can confirm my hypothesis then I'd recommend implementing something similar in sentineleof: https://github.com/ASFHyP3/hyp3-lib/blob/develop/src/hyp3lib/get_orb.py#L24 |
|
We had throughput of 1000 jobs so this is a very likely the culprit. I can't see any active sessions when I log into the email we are using for our access account, but it was at 9 this morning when the tokens were generated. |
|
Confirmed. Ran a few copies of this script in parallel: import datetime
import eof
eof.log._set_logger_handler()
date = datetime.datetime(2024, 1, 1, 0, 0, 0)
for ii in range(101):
print(ii)
eof.download.download_eofs([date], ['S1A'], cdse_user='***', cdse_password='***')Initially the downloads work fine, but they all fail with an unauthorized error after the total number of requests get to around 100: |
|
It's too bad they don't return HTTP 429 -- "too many requests" here, or similar, like they do in rate-limiting situations: because it'd be a lot more clear what's happening. |
jlmaurer
changed the title
RuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/tokenRuntimeError: Access token creation failed. Reason: 401 Client Error: Unauthorized for url: https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token
|
Hey all, sounds like this might be better handled in |
|
I agree that sounds like a good thing to add to |
|
@scottstanie if ASF is much more stable and not rate limited (@jhkennedy and @cmarshak might have experience), perhaps an easy fix that decreases amount of calls would be to swap ESA and ASF in order. Then only restituted orbits would rely on ESA. |
|
As of now, you can skip CDSE by using the $ time eof -d 2024-04-08 -m S1A --force-asf
[04/10 13:54:24] [INFO asf_client.py] Downloading all filenames from ASF (may take awhile)
...I'd been using the Copernicus API as default for two reasons
I now see that the RESORB list on ASF has been truncated to just 2024? So it only took a little longer to run for recent orbits (10 seconds for ASF, 4 seconds for CDSE), but I'd be interested in seeing what support the ASF mirror wants to have, or if they think it's a good idea to switch to ASF by default. |
|
@scottstanie on the HyP3-side, we also prefer ESA and fall back to ASF by default -- ASF's orbit "catalog" leaves a lot to be desired, and the stability isn't as high as I'd like due to it being an on-prem dataset. If you like, I can open a PR into In the future, and hopefully, near future, ASF may entirely redo our orbit offerings and move everything to S3 with a simple search and discovery API in front. So hopefully, we can just fix these issues for the community. I can't promise a timeline right now, though. |
|
@bbuzz31 @dbekaert - I submitted 198 jobs yesterday afternoon (after the ASF outage in the morning) using the fix from #635. Unfortunately, 153/198 of the jobs failed with a similar error as before. It is unclear if ASF orbit availability was cut off exposing the same ESA issue. Before I resubmit these jobs, we should discuss briefly here - maybe @jhkennedy or @asjohnston-asf have some insights regarding ASF availability. Here are a sample of 3 of such jobs: All such jobs can be obtained: |
|
For clarity, here is a log from one the jobs: |
|
I ran a small test locally and was able to get orbits without error. Can share a script if useful. |
|
Yes - please share in this thread. I am assuming you are doing something similar to what @asjohnston-asf did above with 100 simultaneous requests. |
|
You may meed to use multiprocessing to better hit your rate limit as RAiDER takes 20 minutes to complete so the scaling issue might not be observed locally as easily as in a serial loop. Or use a really small test dataset? |
|
Resolved by #637. I was able to successfully run 100+ jobs through the dev branch via hyp3 after the above PR was merged. |
Describe the bug
It appears a new exception is being raised in the orbit download of raider. I am not sure if this is transient issue or something that is a error with
EOFin how exceptions are being raised. In <1 hour, I saw several thousand jobs fail so am disabling processing.To Reproduce
Here are 10 job parameter dicts supplied to hyp3 all of which failed with this error:
The text was updated successfully, but these errors were encountered: