Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance MET to handle CF-compliant time strings with an offset defined in months or years. #2155

Closed
7 of 20 tasks
JohnHalleyGotway opened this issue May 6, 2022 · 8 comments · Fixed by #2300
Closed
7 of 20 tasks
Assignees
Labels
MET: Configuration priority: high High Priority requestor: NOAA/EMC NOAA Environmental Modeling Center type: enhancement Improve something that it is currently doing
Milestone

Comments

@JohnHalleyGotway
Copy link
Collaborator

JohnHalleyGotway commented May 6, 2022

Describe the Enhancement

This issue arose via GitHub discussions dtcenter/METplus#1604 and an enhancement to the MET's support for CF-compliant NetCDF files is needed to read WOA climatology files. This enhancement supports development of the NOAA/EMC EVS verification system.

As of MET version 10.1.0, processing CF-compliant data with times defined with a months or years offset results in this warning message.

WARNING: parse_cf_time_string() -> Unsupported time step in the CF convention time unit "months since 1955-01-01 00:00:00"

And MET uses sets the valid time to unixtime 0 which is Jan 1, 1970.

The "months" offset is commonly used for climatology data and climate simulations, and this short-coming has come up a few times in the past.

The use of months is discouraged since they vary. According to this section of the CF-convention documentation, 1 year = 365.242198781 days and 1 month = 1 year / 12. So 1 month = 30.436 days.

However, I recommend against implementing this logic. Instead, let's implement the logic that users actually intend and expect. For example, when time is defined as a value of 377.5 with the string "months since 1955-01-01 00:00:00"...

  • Use integer division to convert 377.5 to 31 years + 5 months + 15 days.
  • Increment the year from 1955 to 1986.
  • Increment the month from 01 to 06.
  • Increment the day from 01 to 15.

The last step is confusing. Modelers use 0.5 month to mean the 15th of the months rather than an offset of 15 days (01 + 15). Need to think about that one.

@hsoh-u also ask how to handle these cases, where the compute time does not actually exist?
Another issue is what if the start day is 29, 30 or 31.
"month since 2020-01-29 00:00:00"
"month since 2020-01-31 00:00:00"
30, Feb
31, Apr

We'd need to define logic for handling these edge cases.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
No sub-issues needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2773542 (was 2793541)

Define the Metadata

Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Linked issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@JohnHalleyGotway JohnHalleyGotway added type: enhancement Improve something that it is currently doing requestor: NOAA/EMC NOAA Environmental Modeling Center priority: high alert: NEED MORE DEFINITION Not yet actionable, additional definition required alert: NEED ACCOUNT KEY Need to assign an account key to this issue MET: Configuration labels May 6, 2022
@JohnHalleyGotway JohnHalleyGotway added this to the MET 11.0.0 milestone May 6, 2022
@JohnHalleyGotway JohnHalleyGotway added priority: high High Priority and removed priority: high labels May 9, 2022
@JohnHalleyGotway JohnHalleyGotway added priority: blocker Blocker priority: high High Priority required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project and removed priority: high High Priority priority: blocker Blocker alert: NEED MORE DEFINITION Not yet actionable, additional definition required labels Aug 2, 2022
@JohnHalleyGotway JohnHalleyGotway added priority: medium Medium Priority and removed priority: blocker Blocker required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project labels Aug 22, 2022
@TaraJensen TaraJensen removed the alert: NEED ACCOUNT KEY Need to assign an account key to this issue label Sep 14, 2022
@hsoh-u
Copy link
Collaborator

hsoh-u commented Sep 14, 2022

  • For month
    1. get integer part and add it to target month (adjust year if necessary)
    2. get fractional part and get the nearest day after multiplying 30
    3. add it to target day (do not adjust month, change day to the last day of the month if necessary)
  • For year
    1. get integer part and add it to target year
    2. get fractional part and get the nearest month after multiplying 12
    3. add it to target month (and adjust year and day if necessary)

Examples:

units = "months since 1955-01-01 00:00:00"
1.5 month ==> "1955-02-16 00:00:00"
  + Add 1 months to the base time: "1955-02-01 00:00:00"
  + Add 15 days (0.5 * 30 = 15): "1955-02-16 00:00:00"
3.33 month == > "1955-04-11 00:00:00"
  + Add 3 months to the base time: "1955-04-01 00:00:00"
  + Add 10 days (0.33 * 30 = 9.9 days to 10 days): "1955-04-11 00:00:00"

units = "years since 1955-01-01 00:00:00"
3.5 year ==> "1958-07-01 00:00:00"
  + Add 3 years to the base time: "1958-01-01 00:00:00"
  + Add 6 months (0.5 * 12 = 6): "1958-07-01 00:00:00"
3.3 year ==> "1958-05-01 00:00:00"
  + Add 3 years to the base time: "1958-01-01 00:00:00"
  + Add 4 months (0.3 * 12 = 3.6 months to 4 months): "1958-05-01 00:00:00"

@j-opatz
Copy link
Contributor

j-opatz commented Sep 29, 2022

I think the approach stated above,

The other approach for non-integer value will be

    For month
        get integer part and add it to target month (adjust year if necessary)
        get fractional part and get the nearest day after multiplying 30
        add it to target day (do not adjust month, change day to the last day of the month if necessary)
    For year
        get integer part and add it to target year
        get fractional part and get the nearest month after multiplying 12
        add it to target month (and adjust year and day if necessary)

Seems like good logic. However, there's two issues I want to raise.

In the last example, 3.3 years is evaluated to 1958-07-01. However, that seems like it's slightly further in time than might be desired.

After adjusting the base time by 3 years, the remainder (0.3) is multiplied by 12 with a resulting 3.6 months. To me, that means the months should be incremented by 3 rather than 4, and there needs to be an increment of days as well. Using the logic of months_since, would it be possible to take that remainder and multiply it by 30 to increment the days by 9? Any further down the time stamp than that (e.g. taking the 2nd example's remainder of 0.33 and needing to change the hours in addition to changing the times) seems too picky.

John HG mentioned that

Modelers use 0.5 month to mean the 15th of the months rather than an offset of 15 days (01 + 15). Need to think about that one.

I noticed that in the logic provided, 1.5 months goes to the 16th of the month. Do we need a special logic case where a remainder of 0.5 always takes the days to the 15th of whatever month is evaluated to?

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 3, 2022

Need a decision for the algorithm: Day for 0.5 month
What's the scientific preference: "2020-03-15" or "2020-03-16"?

  • date=0.0, 0.5, 1.0. 1.5, 2,0. 2.5, 2.0 with units = "month since 2020-03-01 00:00:00"
  • which one?
    • 2020-03-01, 2020-03-15, 2020-04-01, 2020-04-15, 2020-05-01, 2020-05-15
    • 2020-03-01, 2020-03-16, 2020-04-01, 2020-04-16, 2020-05-01, 2020-05-16

Assuming for month and year units:

  • 1 month = 30 days (1 year = 360 days)
  • DO NOT use nearest day/month
    • 3.5 days is 3 days, NOT 3 days plus 0.5 * 24 hours
    • 3.5 months is 3 months plus (0.5 * 30) days
  • Only month and day units are updated for month unit
  • Only year, month, and day units are updated for year unit
  • When day is updated, the month is not adjusted. Set the last day of the month instead of updating month

Algorithm.

  • For month
    • get an integer part and add it to the target month (adjust year if necessary)
    • get a fractional part and multiply 30 days
    • add an integer part of additional days to the target day (do not adjust month, change day to the last day of the month if necessary)
  • For year
    • get an integer part and add it to the target year
    • get fractional part and multiply 12 months
    • get an integer part and add it to the month (adjust year in necessary)
    • get a fractional part and multiply 30 days
    • get an integer part and add it to the day (do not adjust month, change day to the last day of the month if necessary)

Examples:

- 3.33 months = 3 months + 9 days 
  + 3.33 months = (3 + 0.33) months
  + 0.33 months = (0.33 * 30) days = 9.9 days = int(9.9) days = 9 days
- 3.66 months = 3 months + 19 days 
  + 3.66 months = (3 + 0.66) months
  + 0.66 months = (0.66 * 30) days = 19.8) days = int(19.8) days = 19 days

units = "months since 1955-01-01 00:00:00"
3.33 month == > "1955-04-10 00:00:00"
  + Add 3 months to the base time: "1955-04-01 00:00:00"
  + Add 9 days (0.33 * 30 = 9.9 days to 9 days): "1955-04-10 00:00:00"
3.66 month == > "1955-04-20 00:00:00"
  + Add 3 months to the base time: "1955-04-01 00:00:00"
  + Add 19 days (0.66 * 30 = 19.8 days to 19 days): "1955-04-20 00:00:00"

units = "years since 1955-01-01 00:00:00"
3.5 year ==> "1958-07-01 00:00:00"
  + Add 3 years to the base time: "1958-01-01 00:00:00"
  + Add 6 months (0.5 * 12 = 6): "1958-07-01 00:00:00"
3.33 year ==> "1958-04-29 00:00:00"
  + Add 3 years to the base time: "1958-01-01 00:00:00"
  + Add 3 months (0.33 * 12 = 3.96 months to 3 months): "1958-04-01 00:00:00"
  + Add 4 months (0.96 * 30 = 28.8 days to 28 days): "1958-04-29 00:00:00"

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 7, 2022

Decision for 0.5 month: set the day to 15:

  • 0.5 month since 2022-03-01 00:00:00" ==> "2022-03-15 00:00:00"

  • 0.5 month since 2022-02-01 00:00:00" ==> "2022-02-15 00:00:00"

    • Note: not "2022-02-14 00:00:00"
  • 0.5 month since 2022-03-02 00:00:00" ==> "2022-03-16 00:00:00"

  • 0.5 month since 2022-02-02 00:00:00" == >"2022-02-16 00:00:00"

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 7, 2022

Is it OK not to support negative offsets, like -4.5 month or -10 years?
No technical issues on supporting the negative offsets. Just checking before implementing it.
If this is acceptable, MET will give an error message and quit.

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 7, 2022

Two options to support negative offsets

  • Option 1 (-4.3 month):
    • move 4 months back
    • move (0.3*30) days back (set to 1 if less than 1)
  • Option 2 (-4.3 month):
    • move 5 months back
    • move ((1-0.3)*30 = 0.7*30) days forward (set to last day of month if necessary)

hsoh-u pushed a commit that referenced this issue Oct 7, 2022
hsoh-u pushed a commit that referenced this issue Oct 7, 2022
hsoh-u pushed a commit that referenced this issue Oct 7, 2022
hsoh-u pushed a commit that referenced this issue Oct 7, 2022
hsoh-u pushed a commit that referenced this issue Oct 7, 2022
@j-opatz
Copy link
Contributor

j-opatz commented Oct 10, 2022

Is it OK not to support negative offsets, like -4.5 month or -10 years? No technical issues on supporting the negative offsets. Just checking before implementing it. If this is acceptable, MET will give an error message and quit.

I think it's OK to ignore negative offsets for time. I've never seen it used before and can't imagine a situation where it would be appropriate.

@hsoh-u
Copy link
Collaborator

hsoh-u commented Oct 12, 2022

Modified to handle the case when the start day (the reference day) is not the first day of the month.
Note: use nearest day, no exceptional calculation for February

  1. months unit
  • when the start day is the first of the month (most common cases)
    • add months from the integer part
    • case 1: 0.5 month is just changing the day to 15
    • case 2: otherwise adds nearest days from the fractional part (times 30 days), but do not update month and adjust to the end of the month if necessary
  • when the start day is not the first of the month (few cases)
    • no special meaning for 0.5 month
    • add months from the integer part
    • add the nearest days to the target day from the fractional part (times 30 days)
      • case 1: the target day is equal or less than day 30 (for example, reference time is "months since 2020-02-05")
        • keep the target day (do not update month & adjust day if necessary)
          • for example: day 5 + (0.6 month * 30 days) = day 5 + 18 days = day 23
            • calculated day: 2020-02-23
      • case 2: the target day is greater than day 30 (for example, reference time is "months since 2020-02-05")
        • add one month
        • the target day is subtracted by 30 days
          • for example, day 5 + 0.9 month = day 5 + (0.9 month * 30 days) = day 5 + 27 days = day 32
            • day 32 - 30 days = day 2
            • calculated day: 2020-03-02
  1. years unit
  • add years from the integer part
  • months comes from the fractional part times 12 months
  • Repeat above algorithm for months

hsoh-u pushed a commit that referenced this issue Oct 12, 2022
@JohnHalleyGotway JohnHalleyGotway linked a pull request Oct 13, 2022 that will close this issue
15 tasks
Repository owner moved this from In Progress to Done in MET-11.0.0-beta4 (11/02/22) Oct 18, 2022
@JohnHalleyGotway JohnHalleyGotway changed the title Enhance the MET library code to handle CF-compliant time strings with an offset defined in months or years. Enhance MET to handle CF-compliant time strings with an offset defined in months or years. Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MET: Configuration priority: high High Priority requestor: NOAA/EMC NOAA Environmental Modeling Center type: enhancement Improve something that it is currently doing
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

4 participants