Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Update parallel=False for open_mfdataset() as potential workaround to NetCDF: Not a Valid ID issue #246

Closed
tomvothecoder opened this issue Feb 28, 2024 · 0 comments · Fixed by #247
Labels
bug Something isn't working

Comments

@tomvothecoder
Copy link
Collaborator

What happened?

From: E3SM-Project/zppy#548 (comment)

Searching for the error NetCDF: Not a valid ID. I landed with a similar issue and a potential solution. To summarise in this thread, it looks like a work-around in netcdf4-python to deal with netcdf-c not being thread safe was removed in 1.6.1. The solution (for now) is to [make sure your cluster only uses 1 thread per worker](https://forum.access-hive.org.au/t/netcdf-not-a-valid-id-errors/389/14).

@tomvothecoder In this case, it might be worth to try by using parallel=False instead in xarray.open_mfdataset to try by passing this issue.

What did you expect to happen? Are there are possible answers you came across?

ds = xc.open_mfdataset(
all_filepaths,
add_bounds=["X", "Y"],
decode_times=False,
combine="nested",
data_vars="minimal",
coords="minimal",
parallel=True,
)

Minimal Complete Verifiable Example (MVCE)

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

Latest stable version of e3sm_to_cmip and master branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant