Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: external forcings converter #647

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
559ddac
Initial setup of ext_old_to_new tool
arthurvd Jun 6, 2024
1bb5532
converter can now convert e02_dflowfm\f006_external_forcing\c011_extr…
FlorisBuwaldaDeltares Jul 4, 2024
2ab564f
modified tests to no longer use local directories
FlorisBuwaldaDeltares Jul 4, 2024
1fb7583
added extrapolate_slr case
FlorisBuwaldaDeltares Jul 4, 2024
851975d
Added some exception handling, now validate Meteo quantity before mak…
FlorisBuwaldaDeltares Jul 4, 2024
f8dfbce
added basinsquares test, fixed rainschematic.ext as an uncommented li…
FlorisBuwaldaDeltares Jul 4, 2024
34704fa
restore rainschematic.ext as otherwise unit tests get broken
FlorisBuwaldaDeltares Jul 8, 2024
c1033df
autoformat: isort & black
FlorisBuwaldaDeltares Jul 8, 2024
fdb2981
chore!: Refactored ext old converter to prepare for more other quanti…
arthurvd Jul 24, 2024
096d194
Merge branch 'main' into feature/621_extforce_converter
arthurvd Jul 29, 2024
9266e7b
Remove non-existing structure file reference in a test MDU
arthurvd Jul 29, 2024
314fbc8
Remove obsolete keywords from example windcase.mdu
arthurvd Jul 29, 2024
9dd1d70
New feature `exclude_unset` in ModelSaveSettings
arthurvd Nov 11, 2024
e2720ad
Combine power of LegacyFMModel and ResearchFMModel
arthurvd Nov 11, 2024
26e6d9b
Work-in-progress making converter tool more flexible for repeated runs
arthurvd Nov 11, 2024
2fb8c8b
Merge branch 'main' into feature/621_extforce_converter
arthurvd Nov 11, 2024
037cbc3
feat: initial condition converter from old external forcing file
MAfarrag Dec 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions hydrolib/core/dflowfm/bc/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
AstronomicCorrection, Constant, T3D.

"""

import logging
import re
from pathlib import Path
Expand Down
24 changes: 21 additions & 3 deletions hydrolib/core/dflowfm/ext/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
validator_set_default_disk_only_file_model_when_none,
)
from hydrolib.core.dflowfm.bc.models import ForcingBase, ForcingData, ForcingModel
from hydrolib.core.dflowfm.common.models import Operand
from hydrolib.core.dflowfm.ini.models import (
INIBasedModel,
INIGeneral,
Expand Down Expand Up @@ -194,12 +195,24 @@ class MeteoForcingFileType(StrEnum):
bcascii = "bcAscii"
"""str: Space-uniform time series in <*.bc> file."""

netcdf = "netcdf"
"""str: NetCDF, either with gridded data, or multiple station time series."""

uniform = "uniform"
"""str: Space-uniform time series in <*.tim> file."""

unimagdir = "uniMagDir"
"""str: Space-uniform wind magnitude+direction in <*.tim> file."""

meteogridequi = "meteoGridEqui"
"""str: Space- and time-varying wind and pressure on an equidistant grid in <*.amu/v/p> files."""

spiderweb = "spiderweb"
"""str: Space- and time-varying cyclone wind and pressure in <*.spw> files."""

meteogridcurvi = "meteoGridCurvi"
"""str: Space- and time-varying wind and pressure on a curvilinear grid in <*.grd+*.amu/v/p> files."""

netcdf = "netcdf"
"""str: NetCDF, either with gridded data, or multiple station time series."""

allowedvaluestext = "Possible values: bcAscii, netcdf, uniform."


Expand Down Expand Up @@ -246,6 +259,10 @@ class Comments(INIBasedModel.Comments):
interpolationmethod: Optional[str] = Field(
"Type of (spatial) interpolation.", alias="interpolationMethod"
)
operand: Optional[str] = Field(
"How this data is combined with previous data for the same quantity (if any).",
alias="operand",
)

comments: Comments = Comments()

Expand All @@ -264,6 +281,7 @@ class Comments(INIBasedModel.Comments):
interpolationmethod: Optional[MeteoInterpolationMethod] = Field(
alias="interpolationMethod"
)
operand: Optional[Operand] = Field(Operand.override.value, alias="operand")

def is_intermediate_link(self) -> bool:
return True
Expand Down
30 changes: 30 additions & 0 deletions hydrolib/core/dflowfm/mdu/legacy.py
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@priscavdsluis / @tim-vd-aardweg : what do you guys think about this extra "legacy.py" inside the MDU directory?
As long as we don't have versioned support for several old/newer formats of some of our filetypes, this was the solution I came up with.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arthurvd / @priscavdsluis
I am not entirely sure to be honest. My first question is: Do we want tools in HYDROLIB-core, or do we create a separate package for tools that use HYDROLIB-core?

But let's assume we do want to add tools to the HYDROLIB-core package. I think my next question is: Should we implement this 'legacy' model into hydrolib, or should it be defined in the tool that uses it? Also, the change you request touches on a broader topic that should be discussed: How do we handle old/deprecated files/sections/keywords. Your requested change is due to the fact that we cannot handle older/unexpected file formats. And your change fixes things for your specific use case.

Having said all that, my thoughts are as follows:

  • We will need to generically support older/unexpected formats. That way, we don't need your specific legacy model.
  • Since that probably won't happen in the short term, we need a different solution for your use case. My preference would be to add the LegacyModel to the specific tool, so we do not pollute the main HYDROLIB-core package with quick fixes for specific tools.

Copy link
Contributor

@MRVermeulenDeltares MRVermeulenDeltares Jun 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @arthurvd / @tim-vd-aardweg / @priscavdsluis , I know you didn't ask my opinion but I still want to ask some questions regarding this.
Previously I remember a presentation was given with a picture showing hydrolib-core and hydrolib.
And how it should be used by tools.
See https://www.deltares.nl/expertise/projecten/hydrolib-gezamenlijke-automatisering-op-watersysteemmodellen

image

Should the converter not be a tool build on top of hydrolib-core?
Especially if the support for the old file format is dropped and no longer backwards compatible supported by the kernels.
Should hydrolib-core still support the old format, or only the new format from version x?

Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
from typing import Literal

from pydantic import Field

from .models import FMModel, General


class LegacyGeneralAsModel(General):
"""
The `[Model]` section in a legacy MDU file.

This a small helper class to support legacy versions of .mdu files, where
the `[General]` section with metadata did not exist yet, and was called
`[Model]` instead.
"""

_header: Literal["Model"] = "Model"
mduformatversion: str = Field("1.06", alias="MDUFormatVersion")


class LegacyFMModel(FMModel):
"""
Legacy version of an MDU file, specifically for: MDUFormatVersion <= 1.06.

This a small wrapper class to support legacy versions of .mdu files, where
the `[General]` section with metadata did not exist yet, and was called
`[Model]` instead.
"""

model: LegacyGeneralAsModel = Field(default_factory=LegacyGeneralAsModel)
1 change: 1 addition & 0 deletions hydrolib/tools/__init__ .py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

Loading
Loading