Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hide value related GRIB metadata in NumpyFieldList #215

Closed
sandorkertesz opened this issue Oct 4, 2023 · 0 comments · Fixed by #216
Closed

Hide value related GRIB metadata in NumpyFieldList #215

sandorkertesz opened this issue Oct 4, 2023 · 0 comments · Fixed by #216
Assignees
Labels
enhancement New feature or request

Comments

@sandorkertesz
Copy link
Collaborator

sandorkertesz commented Oct 4, 2023

Current situation

If we create a new NumpyFieldList with the following code:

# f is a GribField
vals_new = f.values + 1
md_new = f.metadata().override(shortName="pt")
ds_new = FieldList.from_numpy(vals_new, md_new)

then the each new NumpyField consists of a numpy array (vals_new) and a metadata object, which stores a GRIB handle cloned from the handle in the original GRIB field but the values in it are not updated with vals_new. See #203 for an explanation.

As a result, there is an inconsistency between the various Field methods related to the values:

# these use the numpy array
ds_new[0].values
ds_new[0].to_numpy()
ds_new[0].data()

# these use the metadata object
ds_new[0].metadata("max")
ds_new[0].metadata("bitmapPresent")
ds_new[0].metadata("numberOfMissing")
ds_new[0].metadata(namespace="statistics")
ds_new[0].dump()
ds_new[0].dump(namespace="statistics")
>>> ds_new[0].values.max()
234.12
>>> ds_new[0].metadata("max")
233.12

The proposed solution

These GRIB keys are in fact not metadata keys but rather related to the actual values in the field. Moreover we can get the correct value related stats by inspecting the numpy array. So the right decision is to hide them in a NumpyFieldLists.

The expected behaviour:

# this throws a KeyError
ds_new[0].metadata("max")
ds_new[0].metadata()["max"]

# these return None
ds_new[0].metadata("max", None)
ds_new[0].metadata().get("max")

The "statistics" namespace will be hidden, whether we use the namspace kwarg in metadata() or we use the dump(). E.g. this is how the dump() output would look like:

image
@sandorkertesz sandorkertesz added the bug Something isn't working label Oct 4, 2023
@sandorkertesz sandorkertesz self-assigned this Oct 4, 2023
@sandorkertesz sandorkertesz added enhancement New feature or request and removed bug Something isn't working labels Oct 5, 2023
@sandorkertesz sandorkertesz changed the title Fix value related metadata access in NumpyFieldList Hide value related GRIB metadata in NumpyFieldList Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant