Adapt reader mviri_l1b_fiduceo_nc #2802

bkremmli · 2024-05-17T09:07:13Z

This PR fixes the mviri_l1b_fiduceo_nc reader when being used with a new xarray version (2024.3.0). When using the original reader, a ValueError about not being able to decode the times is thrown. The file is now opened without decoding. The decoding is now done in DatasetWrapper()._decode_cf(). The time is decoded separatly from the other data values. FillValues for the time are recognized to replace time values with NaT and time is decoded using the offset values included within the attributes.

Also, opening the dataset using chunks is deactivated because the input files contains dimensions of the same name which cannot be processed by xarray at the moment. The chunking as well as renaming the dimensions is also performed in DatsetWrapper().

[x ] Tests added
Fully documented --> this change should not be visible to users
[x ] Add your name to AUTHORS.md if not there already

…mes after import

…ead_error

mraspaud

Thanks for adding this feature! I'm no expert here, so problably @sfinkens needs to have a look, but I wrote some comments already :)

satpy/readers/mviri_l1b_fiduceo_nc.py

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py

satpy/readers/mviri_l1b_fiduceo_nc.py

Co-authored-by: Martin Raspaud <martin.raspaud@smhi.se>

sfinkens

Nice work @bkremmli! Thanks for updating the tests, so that the problem is triggered.

satpy/readers/mviri_l1b_fiduceo_nc.py

sfinkens · 2024-05-17T15:49:35Z

Also I noticed that space pixels now have some finite values (instead of NaN), because decode_cf=False. You can use decode_cf=True together with time.encoding["add_offset/_FillValue"], see sfinkens@7045a87

codecov · 2024-05-24T06:10:42Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.07%. Comparing base (6a4f2af) to head (484b723).
Report is 367 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2802   +/-   ##
=======================================
  Coverage   96.06%   96.07%           
=======================================
  Files         373      373           
  Lines       54406    54467   +61     
=======================================
+ Hits        52267    52328   +61     
  Misses       2139     2139

Flag	Coverage Δ
behaviourtests	`3.98% <0.00%> (-0.01%)`	⬇️
unittests	`96.16% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ead_error Conflicts: satpy/readers/mviri_l1b_fiduceo_nc.py

sfinkens

Nice work, almost there!

I think in the tests you need to write the dataset to disk and read it back in. Otherwise the dask error with duplicate dimensions is not triggered.

satpy/readers/mviri_l1b_fiduceo_nc.py

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py

bkremmli · 2024-06-04T13:33:17Z

The test for the duplicate dimensions still needs to be added.

…rf_size"

small fix to DataWrapper._decode_cf()

adds support for filenames of MVIRI FCDR L1.5 release 2

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py

satpy/readers/mviri_l1b_fiduceo_nc.py

- include u_independent_toa_bidirectional_reflectance in tests - patch decode_cf/fix_duplicate_dimensions in test_reassign_coords()

sfinkens

Looks very good, thanks! I'm just nitpicking with minor things now 🙂

satpy/readers/mviri_l1b_fiduceo_nc.py

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py

satpy/readers/mviri_l1b_fiduceo_nc.py

sfinkens

LGTM!

satpy/readers/mviri_l1b_fiduceo_nc.py

sfinkens · 2024-09-23T13:53:01Z

Let's see if we can make CodeScene happy

sfinkens

Good job!

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py

Merge main into read_error

Merge from satpy/main

coveralls · 2024-10-01T08:10:18Z

Pull Request Test Coverage Report for Build 11122895363

Details

75 of 75 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.005%) to 96.169%

Totals
Change from base Build 11109706783:	0.005%
Covered Lines:	52562
Relevant Lines:	54656

💛 - Coveralls

test original fill_val

sfinkens · 2024-10-01T10:08:14Z

@mraspaud Do you want to have a final look? In the next PR we'll do a bit of refactoring and split "DatasetWrapper" into preprocessor and accessor

mraspaud · 2024-10-17T08:13:09Z

I just had a quick look and this looks good to merge. In a future PR, you might want to ensure the sensor data is in float32 to increase performance.

sfinkens · 2024-10-18T16:33:40Z

you might want to ensure the sensor data is in float32 to increase performance.

That's checked in the tests

https://github.com/pytroll/satpy/blob/main/satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py#L433
https://github.com/pytroll/satpy/blob/main/satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py#L77

bkremmli added 4 commits May 17, 2024 07:02

fix for file reading; includes removing chunk reading and decoding ti…

f27a91d

…mes after import

Merge branch 'read_error' of https://github.com/bkremmli/satpy into r…

a08d9bc

…ead_error

remove import datetime

2dbae1a

add bkremmli to AUTHORS.md

eb0382a

bkremmli requested review from djhoese and mraspaud as code owners May 17, 2024 09:07

correct for failures from hook id ruff

7d6608a

mraspaud requested changes May 17, 2024

View reviewed changes

mraspaud requested a review from sfinkens May 17, 2024 10:09

bkremmli and others added 2 commits May 17, 2024 10:18

minor adaptations from PR comments

ec22136

Update satpy/readers/mviri_l1b_fiduceo_nc.py

5beedea

Co-authored-by: Martin Raspaud <martin.raspaud@smhi.se>

sfinkens reviewed May 17, 2024

View reviewed changes

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

bkremmli added 2 commits May 24, 2024 08:02

perform chunking after open_dataset and use decode_cf = False

21679c6

Merge branch 'read_error' of https://github.com/bkremmli/satpy into r…

a7cb10d

…ead_error Conflicts: satpy/readers/mviri_l1b_fiduceo_nc.py

sfinkens reviewed May 24, 2024

View reviewed changes

decode times separatly from other variables, adds TestInterpolator

59880ce

sfinkens reviewed May 29, 2024

View reviewed changes

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

fixes _decode_cf() and tests

fb93f00

bkremmli and others added 4 commits June 5, 2024 13:37

adds test_fix_duplicate_dimensions and removes leftover dimensions "s…

951c9b0

…rf_size"

Update mviri_l1b_fiduceo_nc.py

a379a08

small fix to DataWrapper._decode_cf()

adds support for filenames of MVIRI FCDR L1.5 release 2

5ec0cf9

Merge pull request #1 from bkremmli/mviri_release2

73acfb7

adds support for filenames of MVIRI FCDR L1.5 release 2

sfinkens reviewed Jun 12, 2024

View reviewed changes

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Show resolved Hide resolved

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

sfinkens reviewed Jun 12, 2024

View reviewed changes

satpy/readers/mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

bkremmli and others added 2 commits September 3, 2024 08:41

Merge branch 'pytroll:main' into read_error

01856fd

sync/merge with fork diffs

c1cd334

bkremmli added 4 commits September 4, 2024 08:45

moved code _get_projection_longitude()

f41e068

fix test_fix_duplicate_dimensions

6b079b1

- suppress warnings for renaming duplicate dimensions

fd099cd

- include u_independent_toa_bidirectional_reflectance in tests - patch decode_cf/fix_duplicate_dimensions in test_reassign_coords()

define fill_val in test

7c02d2d

sfinkens reviewed Sep 19, 2024

View reviewed changes

ancillary changes like comments, etc.

e90d7d0

sfinkens approved these changes Sep 23, 2024

View reviewed changes

satpy/readers/mviri_l1b_fiduceo_nc.py Show resolved Hide resolved

change fill_val definition in test, changes for Code Scene status

51d8e77

bkremmli requested a review from mraspaud September 24, 2024 06:53

sfinkens reviewed Sep 24, 2024

View reviewed changes

satpy/tests/reader_tests/test_mviri_l1b_fiduceo_nc.py Outdated Show resolved Hide resolved

bkremmli and others added 8 commits September 24, 2024 08:04

change fill_val to int64

89d9931

include comments, create fixture_time_fake_dataset()

7d0ecef

adapt fill_val definition

11cf080

adapt fill_val definition

03ec1fe

lower fill_val for tests for windows compatibility

bcae3f7

Merge remote-tracking branch 'origin/main' into read_error

89bb691

Merge pull request #2 from pytroll/main

672cd9c

Merge main into read_error

Merge pull request #3 from pytroll/main

c742238

Merge from satpy/main

bkremmli and others added 4 commits October 1, 2024 08:52

add test for get_projection_longitude,

38c8e9b

test original fill_val

Merge remote-tracking branch 'origin/read_error' into read_error

2042b1a

Fix test miss

d88d3d3

revert to smaller fill_val

484b723

sfinkens approved these changes Oct 1, 2024

View reviewed changes

mraspaud merged commit 07d2c00 into pytroll:main Oct 17, 2024
20 checks passed

mraspaud assigned bkremmli Oct 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapt reader mviri_l1b_fiduceo_nc #2802

Adapt reader mviri_l1b_fiduceo_nc #2802

bkremmli commented May 17, 2024 •

edited

Loading

mraspaud left a comment

sfinkens left a comment

sfinkens commented May 17, 2024

codecov bot commented May 24, 2024 •

edited

Loading

sfinkens left a comment

bkremmli commented Jun 4, 2024

sfinkens left a comment

sfinkens left a comment

sfinkens commented Sep 23, 2024

sfinkens left a comment

coveralls commented Oct 1, 2024 •

edited

Loading

sfinkens commented Oct 1, 2024 •

edited

Loading

mraspaud commented Oct 17, 2024

sfinkens commented Oct 18, 2024

Adapt reader mviri_l1b_fiduceo_nc #2802

Adapt reader mviri_l1b_fiduceo_nc #2802

Conversation

bkremmli commented May 17, 2024 • edited Loading

mraspaud left a comment

Choose a reason for hiding this comment

sfinkens left a comment

Choose a reason for hiding this comment

sfinkens commented May 17, 2024

codecov bot commented May 24, 2024 • edited Loading

Codecov Report

sfinkens left a comment

Choose a reason for hiding this comment

bkremmli commented Jun 4, 2024

sfinkens left a comment

Choose a reason for hiding this comment

sfinkens left a comment

Choose a reason for hiding this comment

sfinkens commented Sep 23, 2024

sfinkens left a comment

Choose a reason for hiding this comment

coveralls commented Oct 1, 2024 • edited Loading

Pull Request Test Coverage Report for Build 11122895363

Details

💛 - Coveralls

sfinkens commented Oct 1, 2024 • edited Loading

mraspaud commented Oct 17, 2024

sfinkens commented Oct 18, 2024

bkremmli commented May 17, 2024 •

edited

Loading

codecov bot commented May 24, 2024 •

edited

Loading

coveralls commented Oct 1, 2024 •

edited

Loading

sfinkens commented Oct 1, 2024 •

edited

Loading