Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-generate zero data for missing data payload #362

Merged
merged 3 commits into from
Nov 16, 2023

Conversation

bjlittle
Copy link
Member

@bjlittle bjlittle commented Nov 16, 2023

This pull-request extends the capability of the iris_grib.message._DataProxy to detect and auto-generate missing data payload for a GRIB2 message.

The scenario is that the message data payload is all zero, and rather than serialize this within the message, thus increasing the footprint, an application can generate the data at runtime whenever the Data Representation Section [5] has bitsPerValue=0 and an empty Data Section [7].

Failure to support this extension results in no encoded Data Section [7] within a iris_grib.message._RawGribMessage causing a KeyError traceback within the user script. The impact of this is that an N-D Iris cube can be loaded but whenever at least one GRIB2 message has such missing data, touching the cube data will result in this failure.

Reference: SciTools/iris-test-data#89

@bjlittle bjlittle requested a review from pp-mo November 16, 2023 15:27
@bjlittle
Copy link
Member Author

@pp-mo As a result of this fix, we now have "unexpected successes" in the tests regarding pickling.

We could now simply not mark these tests are expected failures.

How would you like to proceed?

@bjlittle bjlittle force-pushed the fix-data-proxy-bitspervalue branch from 1d9e159 to 0537bfe Compare November 16, 2023 15:36
@bjlittle
Copy link
Member Author

Out of curiosity, I remove the @unittest.expectedFailure decorators ...

@pp-mo
Copy link
Member

pp-mo commented Nov 16, 2023

We could now simply not mark these tests are expected failures.

After some chat, I think this is OK.
I'll do some back-fill to check what the error USED to be, but I suspect we just found that latest-eccodes has fixed it (= made the pickling possible)

Copy link
Member

@pp-mo pp-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless you want to follow a tiny suggestion..
good to go I think

@pp-mo pp-mo merged commit 70f3020 into SciTools:main Nov 16, 2023
5 checks passed
@bjlittle
Copy link
Member Author

@pp-mo Sweet, cheers 💯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants