Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Himawari-8 AHI L2 reader #1250

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add Himawari-8 AHI L2 reader #1250

wants to merge 1 commit into from

Conversation

zxdawn
Copy link
Member

@zxdawn zxdawn commented Jul 6, 2020

This PR adds a reader for the Himawari-8 AHI L2 data which are NC files.

  • Tests added
  • Tests passed
  • Passes flake8 satpy
  • Fully documented

@zxdawn
Copy link
Member Author

zxdawn commented Jul 6, 2020

Current stage of this reader:

1. No end_time property

As only start_time is shown in the filename and many files miss end_time, I don't assign values to it.

2. csv not supported

There's a special L2 data called Wildfire. Here's the filename pattern:

 H08_YYYYMMDD_hhmm_L2WLFVER_FLDK.xxxxx_yyyyy.csv

 where YYYY: 4-digit year of observation start time (timeline);
       MM: 2-digit month of timeline;
       DD: 2-digit day of timeline;
       hh: 2-digit hour of timeline;
       mm: 2-digit minutes of timeline;
       VER: version;
       xxxxx: pixel number; and
       yyyyy: line number.

 Example: H08_20150727_0800_L2WLFbet_FLDK.06001_06001.csv

Maybe we can add another reader called ahi_l2_csv. Any suggestions for this problem?

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.1%) to 89.882% when pulling b65d533 on zxdawn:h8_l2 into b909413 on pytroll:master.

@simonrp84
Copy link
Member

Where is this Himawari L2 data from? I guess p-tree?
The file name format is different to the data on the AWS cloud (e.g: https://noaa-himawari8.s3.amazonaws.com/index.html#AHI-L2-FLDK-Clouds/2020/06/04/0030/) so I wonder if the internal data in the netCDF is also different. Hopefully not!

This is mainly just a note to remind me to look into this when I have some free time later in the week, it'd be good to support both source of L2 data.

@codecov
Copy link

codecov bot commented Jul 6, 2020

Codecov Report

Merging #1250 into master will decrease coverage by 0.14%.
The diff coverage is 24.28%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1250      +/-   ##
==========================================
- Coverage   90.02%   89.88%   -0.15%     
==========================================
  Files         218      219       +1     
  Lines       31269    31339      +70     
==========================================
+ Hits        28151    28168      +17     
- Misses       3118     3171      +53     
Impacted Files Coverage Δ
satpy/readers/ahi_l2_nc.py 24.28% <24.28%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b909413...b65d533. Read the comment docs.

@zxdawn
Copy link
Member Author

zxdawn commented Jul 6, 2020

@simonrp84 Yes, they're from p-tree. The attached README txt illustrates all information. BTW, it seems the L2 data on AWS just has three cloud products, right? Then, that's different from these data on p-tree ...

Not sure that we need two readers or just one for dealing with both kinds of data.

README_HimawariGeo_en.txt

@simonrp84
Copy link
Member

@zxdawn Hmm, that looks quite different. I'll investigate more later in the week!

@songhan89
Copy link

There's also another set of L2 product from JMA in .grib. I've uploaded some samples here.

Not sure if you would like to incorporate them.

Sample files for JMA HCAI

@djhoese
Copy link
Member

djhoese commented Jul 16, 2020

Any GRIB files should be handled by a separate reader. Either the generic grib reader or a new ahi_l2_grib reader.

@mraspaud
Copy link
Member

Great work so far @zxdawn ! don't hesitate to put some image previews here to, it's always nice to see what it will look like.

@zxdawn
Copy link
Member Author

zxdawn commented Jul 20, 2020

@mraspaud Thanks! I'm writing a parallax_correction composite based on Cloud Top Height as suggested by @simonrp84. I would post some CTH images here and create a PR for the composite to check the effect of correction ;)

(I'm preparing the proposal for my PhD research, so that would be a little later)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants