Add the ability to specify a different ESGF node #20
Comments
|
Did you use parse_instance_ids or The fact that even though we are making a 'distributed request' the results differ across search nodes is sketchy though... |
|
@jbusecke I'm confused. I'm doing: from pangeo_forge_esgf.parsing import parse_instance_ids
parse_iids = [
'CMIP6.HighResMIP.*.HadGEM3-GC31-HH.*.*.*.so.gn.*',
'CMIP6.HighResMIP.*.HadGEM3-GC31-HH.*.*.*.thetao.gn.*'
]
iids = []
for piid in parse_iids:
iids.extend(parse_instance_ids(piid))
iidswhich results in only monthly data being shown: but that's because it's using the LLNL node: pangeo-forge-esgf/pangeo_forge_esgf/parsing.py Lines 42 to 45 in 8036e36 How would I make the same request with get_urls_from_esgf? |
|
ping @jbusecke in case this slid off the radar screen... 🐱 |
|
Oh I got confused. So two things: First this is how you would use get_urls_from_esgf import asyncio
from pangeo_forge_esgf.recipe_inputs import get_urls_from_esgf
iids = ['']
# when running from a script
url_dict = asyncio.run(
get_urls_from_esgf(
iids,search_nodes=["http://esgf-node.llnl.gov/esg-search/search"]
)
)
# for a jupyter notebook
url_dict = await get_urls_from_esgf(
iids,search_nodes=["http://esgf-node.llnl.gov/esg-search/search"]
)This will not expand wildcards like Either way, give #21 a try with the node of your choice? |
|
I have seen some of this behavior in several issues, I am not getting the same results from the API as from that search interface, which is upsetting. Will have to investigate more at some point, but am short for time today. |
Merged
|
Ok apparently the LLNL node is totally down? Not even their examples work... |
|
Interesting. the dkrz node API seems to be working. |
|
Just running this from a dev branch locally (will push in a sec) and it seems that the node does not make a difference: parse_iids = [
...: 'CMIP6.HighResMIP.*.HadGEM3-GC31-HH.*.*.*.thetao.gn.*',
...: ]
...: iids_dict = {}
...: search_nodes = [
...: "https://esgf-node.ipsl.upmc.fr/esg-search/search",
...: "https://esgf-index1.ceda.ac.uk/esg-search/search",
...: "https://esgf-data.dkrz.de/esg-search/search",
...: "https://esg-dn1.nsc.liu.se/esg-search/search",
...: "https://esgf-node.llnl.gov/esg-search/search",
...: "https://esgf.nci.org.au/esg-search/search",
...: "https://esgf-node.ornl.gov/esg-search/search",
...: ]
...: for node in search_nodes:
...:
...: iids = []
...: for piid in parse_iids:
...: iids.extend(parse_instance_ids(piid, search_node="https://esgf-node.ipsl.upmc.fr/esg-search/search"))
...: iids_dict[node] = iids
...: iids_dictgives: So the good news is the returns from the API are consistent. The bad news is: WHY ON EARTH DOES THE WEB INTERFACE SHOW DIFFERENT DATASETS THAN THE API? |
|
Actually! Looking at the screenshot again I do not see |
|
@rsignell are you sure that daily files exist for the ocean variables? from pangeo_forge_esgf.parsing import parse_instance_ids
...:
...: parse_iids = [
...: 'CMIP6.HighResMIP.*.HadGEM3-GC31-HH.*.*.*.pr.gn.*',
...: ]
...: iids_dict = {}
...: search_nodes = [
...: "https://esgf-node.ipsl.upmc.fr/esg-search/search",
...: "https://esgf-index1.ceda.ac.uk/esg-search/search",
...: "https://esgf-data.dkrz.de/esg-search/search",
...: "https://esg-dn1.nsc.liu.se/esg-search/search",
...: "https://esgf-node.llnl.gov/esg-search/search",
...: "https://esgf.nci.org.au/esg-search/search",
...: "https://esgf-node.ornl.gov/esg-search/search",
...: ]
...: for node in search_nodes:
...:
...: iids = []
...: for piid in parse_iids:
...: iids.extend(parse_instance_ids(piid, search_node="https://esgf-node.ipsl.upmc.fr/
...: esg-search/search"))
...: iids_dict[node] = iids
...: iids_dictgives this (which is what I expect to see from your screenshot above) |
|
So my current conclusion is that What exists as daily output are surface temperature and surface salinity (and a bunch of atmos variables that you showed above): |
|
@jbusecke , OMG. I'm so sorry for this wild goose chase. You are correct -- If I search the CEDA node for Why isn't there an egg-on-face emoticon? |
|
🥚🙈 |
|
FWIW the goose chase shamed me into adding some more CI and actually fixing the |
|
Ill close this now. |
|
Thanks for being understanding and kind here @jbusecke !! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I searched using
pangeo-forge-esgfand told a colleague that daily files for HADGEM did not exist, and he showed me they do exist, but he used the UK ESGF node:The text was updated successfully, but these errors were encountered: