Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor I/O #31

Merged
merged 33 commits into from
Oct 16, 2019
Merged

Refactor I/O #31

merged 33 commits into from
Oct 16, 2019

Conversation

sfinkens
Copy link
Member

@sfinkens sfinkens commented Oct 2, 2019

Move masking/correction logic from gac_io to the readers so that the satpy reader (which is using pygac as a library) returns the same data as pygac-run writes to hdf5.

The refactoring doesn't change the output expect for one case:

  • The current version doesn't catch NaN lat/lon so that they end up as -2147483648 (the minimum integer) in the hdf5 file. That will be fixed by this PR.

Other features added:

  • Make coordinate interpolation optional
  • Make clock drift adjustment optional
  • Add TLE dir & name attributes to the reader (-> no PYGAC_CONFIG_FILE required for use in satpy)

The changes have been validated with 130 "best of corruption" orbits. Summary:

  • Out of 129 orbits 7 differ in the refactored version. The other 122 are binary identical.
  • If two files differ, it's only the mask, not the data. If there are two valid data points at a given pixel, they are identical.
  • The differences are all caused by very few NaNs in the lat/lon arrays. Here the masking of the new version is clearly better.
  • Broken-scan-motor orbits are identical, too. The differences I reported before were due to a unit/channel mismatch. This is now fixed.

You can find some plots here: https://public.cmsaf.dwd.de/data/sfinkens/pygac_refactoring/ . Display them in 100%. You might have to clean your screen in order to see the mask difference in some cases ;)

TODO:

  • Thorough validation of the altered results
  • Fix tests
  • Add more tests

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comprehensive refactor and adding all the tests! Just a few small points to fix.

pygac/gac_io.py Show resolved Hide resolved
pygac/gac_io.py Show resolved Hide resolved
pygac/gac_pod.py Outdated Show resolved Hide resolved
pygac/gac_reader.py Outdated Show resolved Hide resolved
pygac/gac_reader.py Show resolved Hide resolved
@mraspaud mraspaud merged commit 2d2a162 into pytroll:master Oct 16, 2019
@sfinkens sfinkens deleted the refactor-io branch October 16, 2019 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants