0002-toml-config-support

[savente93]

• Feature Name: toml-config-support
• Start Date: 2023-07-18

Summary

Currently we only support yaml for configuring either the data catalogs, or the models. Users have expressed desire for the option to configure HydroMT models using the toml format. This, however raises the question: Should we only support it for the model configuration or everywhere? This RFC argues in favor of supporting toml throughout all of HydroMT

Motivation

Users have expressed the desire to be able to use toml for their model configuration, because this is more compatible with other systems. Therefore it is desired to at least allow the configuration of models in toml. For teams that use toml for their other configurations, it avoids having to mix and match somewhat incompatible language formats. (yaml supports null whereas toml has no such functionality)

Toml is also gaining more popularity as a configuration format, (e.g. pyproject.toml). This means that having support for TOML would also open up the possibility for tooling made for other purposes such as linters and schema checkers to be used for HydroMT as well.

Finally a recommendation has been made by DSC to use toml as a first option for new projects within Deltares. To strive for a uniformity with future projects a strong case would be made to at least support toml for those projects, if not outright move towards toml (though the latter is outside the scope of this RFC). Even if our users experience some discomfort from it, it might make the Deltares software suite more consistent to external users which can then be used to only use toml.

Guide-level explanation

• Accepting this RFC would mean that users gain the option to configure both their data catalogs and their models using files written in TOML format. That means that for example they might use a data catalog like the following:

root = "d:/hydromt_testdata"

[era5]
path = "ERA5/daily/era5_{year}_daily.nc"
data_type = "RasterDataset"
driver = "netcdf"
crs = 4_326

[era5.kwargs]
concat_dim = "time"
decode_times = true
combine = "by_coords"
parallel = true
chunks = {
	time = 100,
	longitude = 120,
	latitude = 125,
}

[era5.rename]
tp = "precip"
t2m = "temp"
tmin = "temp_min"
tmax = "temp_max"
msl = "press_msl"
ssrd = "kin"
tisr = "kout"

[era5.unit_mult]
precip = 1_000
press_msl = 0.01
kin = 0.000277778
kout = 0.000277778

[era5.unit_add]
time = 86_400
temp = -273.15
temp_min = -273.15
temp_max = -273.15

[era5.meta]
category = "meteo"
history = "Extracted from Copernicus Climate Data Store; resampled by Deltares to daily frequency"
source_version = "ERA5 daily data on pressure levels"
source_url = "https://doi.org/10.24381/cds.bd0915c6"
paper_ref = "Hersbach et al. (2019)"
paper_doi = "10.1002/qj.3803"
source_license = "https://cds.climate.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products"

[hydro_lakes]
path = "waterbodies/lake-db.gpkg"
data_type = "GeoDataFrame"
driver = "vector"
crs = 4_326

[hydro_lakes.rename]
Hylak_id = "waterbody_id"
Lake_area = "Area_avg"
Vol_total = "Vol_avg"
Depth_avg = "Depth_avg"
Dis_avg = "Dis_avg"
Pour_long = "xout"
Pour_lat = "yout"

[hydro_lakes.unit_mult]
Area_avg = 1_000_000

[hydro_lakes.meta]
category = "surface water"
source_version = 1
source_author = "Arjen Haag"
source_info = "HydroLAKES.v10_extract"

[gtsmv3_eu_era5]
path = "water_level\\reanalysis-waterlevel-{year}-m{month:02d}.nc"
data_type = "GeoDataset"
driver = "netcdf"
crs = 4_326

[gtsmv3_eu_era5.kwargs]
concat_dim = "time"
decode_times = true
combine = "by_coords"
parallel = true

[gtsmv3_eu_era5.kwargs.chunks]
stations = 100
time = 1_500

[gtsmv3_eu_era5.rename]
station_x_coordinate = "lon"
station_y_coordinate = "lat"

[gtsmv3_eu_era5.meta]
category = "ocean"
source_version = "GTSM v3.0"
paper_doi = "10.24381/cds.8c59054f"
paper_ref = "Copernicus Climate Change Service 2019"
source_url = "https://cds.climate.copernicus.eu/cdsapp#!/dataset/10.24381/cds.8c59054f?tab=overview"
source_license = "https://cds.climate.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products"

• For the purposes of this RFC HydroMT's internal configuration will remain in YAML format. TOML is not meant to replace YAML, simply complement it. It is an optional feature for teams that already have significant configuration in TOML. As HydroMT, and in all of our documents we will continue to use a single language: YAML.
• The rule of thumb is that as HydroMT we accept Toml, but never produce it unless explicitly asked.
• The CLI interface should remain the same. So instead of

hydromt build sfincs /path/to/model_root -r "{'bbox': [4.6891,52.9750,4.9576,53.1994]}" -i /path/to/sfincs_config.yaml -d /path/to/data_catalog.yml -v

the user should be able to type

hydromt build sfincs /path/to/model_root -r "{'bbox': [4.6891,52.9750,4.9576,53.1994]}" -i /path/to/sfincs_config.toml -d /path/to/data_catalog.toml -v

(only changed the extentions on the config files)

• We could either opt to have HydroMT try to infer the format through either extensions, or through attempting to parse (though this last one can get rather expensive for large files). Also note that these options are not mutually exclusive. An option would be to enable a format flag on the CLI like so:

hydromt build sfincs /path/to/model_root -r "{'bbox': [4.6891,52.9750,4.9576,53.1994]}" -f toml -i /path/to/sfincs_config.toml  -d /path/to/data_catalog.toml -v

• Whether this would allow of mixing and matching (e.g. have the catalog in yaml and the model config in toml) is as of yet undecided. If desirable we could introduce seperate flags for them

hydromt build sfincs /path/to/model_root -r "{'bbox': [4.6891,52.9750,4.9576,53.1994]}" --config-format toml -i /path/to/sfincs_config.toml --catalog-format toml -d /path/to/data_catalog.toml -v

• Since this RFC introduces toml purely as something HydroMT would consume it means that for users that have no toml as of yet, nothing will change, they can continue as normal. For users that have or wish to move to toml, but still have yaml, this can be updated quite easily with the following python snippet that depends on only two packages pyyaml, tomli-w

from yaml import load, FullLoader
from tomli_w import dump

def clean_dict(d):
    ret = {}
    if isinstance(d, dict):
        for k, v in d.items():
            if v is not None:
                ret[k] = "NONE"
            else:
                ret[k] = _process_config_out(v)
    else:
        ret = d

    return ret

	
path_to_yaml = 'test_sources.yml'
path_to_toml = 'test_sources.toml'
with open(path_to_yaml, "rb") as f_in:
     yaml_dict = load(f_in, Loader=FullLoader)
with open(path_to_toml, "wb") as f_out:
     dump(clean_dict(yaml_dict),f_out)

Since TOML does not support null values like YAML does, the extra cleaning function is needed. This function can be trivially adjusted to simply discard None values if these are not necessary.

• Since the semantics do not change any use case that is valid under TOML will be valid under YAML, with corresponding error messages, if there are unclarities, they should be dealt with generally.
• During trainings this could provide more flexibility as well given on what the attendees are more familiar with.
• Again since the semantics do not change, we use dictionaries everywhere internally, and both Yaml and Toml are very well understood formats, there is very little risk in terms of maintainability for accepting this RFC.

Reference-level explanation

For this more detail on this see #444

Drawbacks

One potential drawback is that, as some core team members have expressed, mixing and matching languages could be confusing for users. This is mitigated, in the authors view, by the fact that we would only accept toml, meaning it would only be used by those that are already familiar with it. Everything else, such as HydroMT configs, the main catalog, and examples should remain in yaml, and should not be duplicated. We can add a simple section on how to move from yaml to toml akin to the section above, for those that are interested but otherwise only use yaml in our documentation.

One other drawback is that due to it's flexible nature, there are multiple ways of representing the same data structure in toml. This could provide some issues with consistency, although this should be fairly easily remedied by introducing a lintier.

Rationale and alternatives

• HydroMT is a facilitation tool at it's core. This means that we should strive to speak the languages that our users do. Accepting this RFC would be in line with that philosophy
• Not accepting this RFC would mean upholding the current status quo (a remark the author makes without judgement). Currently users who have toml configuration such as Rebasim, must also use yaml for the other HydroMT parts. This makes their workflow more complicated, but is not a blocker.
• We could also choose to only support toml for the model configurations and keep only yaml support for the HydroMT specific configuration. This is possible, but actually increases the mixing of languages required for users. This, as others have commented upon, can be confusing for users and therefore, in the author's view, should not be done. If we support a configuration format somewhere, we should strive to support it everywhere, as much as is reasonable.
• We have in the past also supported ini configuration for HydroMT but this has since been deprecated. It is technically still supported but it's use is discouraged. This means even though our aim was to move to a single supported format, there is precedent for supporting multiple formats concurrently.
• It is somewhat unknown what issues the conversion from ini to toml could cause, but given that the transition to yaml was deemed sufficient, and the translation from yaml to toml was described above, the author does not expect this to create a significant barrier.
• The functionality of the previous point could technically be provided by either an external tool, or even a plugin of sorts but this would vastly increase the complexity, in addition to not benefiting from our testing capabilities. In the author's view this is not a worth while idea.
• Some comments have also been made about the readability of toml vs yaml when concerning deeply nested structures. as an illustration consider the following structure (Path taken from Deltares data catalog in HydroMT)

a:
    b:
        c:
            d:
                e:
                    path: 'p:/11205028-c3s_435/01_data/01_Timeseries/timeseries2/{variable}/reanalysis_{variable}_{freq}_{year}_{month:02d}_v1.nc'

In toml there are two options for this

[a]
b = { 
	c= { 
		d={
			e={
				path= "p:/11205028-c3s_435/01_data/01_Timeseries/timeseries2/{variable}/reanalysis_{variable}_{freq}_{year}_{month:02d}_v1.nc"
			}
		}
	}
}

While this is not much better in toml this can also be presented as

[a.b.c.d.e]
path = "p:/11205028-c3s_435/01_data/01_Timeseries/timeseries2/{variable}/reanalysis_{variable}_{freq}_{year}_{month:02d}_v1.nc"

This somewhat hides the nested structure, which can either be a up or downside depending on the readers considerations. One thing that should also be noted there is that this can impose quite significant constraints on the yaml given that for readability purposes, line length usually has an upper limit imposed on it (as it was until recently in HydroMT). In many cases this can be elevated by splitting values across multiple lines, but this is not always possible (as is the case with paths like above). Here the indents count towards the character limit, especially since usually spaces are preferred over tabs, this limitation can be quite significant. In toml there is at least the option of mitigating this by using the syntax used in the second snippet.

Finally, this last syntax could compose quite well with changes currently under review in #438 Here we introduce the options of version which apply diffs to their base catalog entry. Consider the possible alternatives in both yaml and toml:

esa_worldcover:
  crs: 4326
  data_type: RasterDataset
  driver: raster
  meta:
    category: landuse
    source_license: CC BY 4.0
    source_url: https://doi.org/10.5281/zenodo.5571936
    source_version: v100
  versions:
    - legacy_esa_worldcover:
        path: landuse/esa_worldcover/esa-worldcover.vrt
        filesystem: local
        driver_kwargs:
          chunks:
            x: 36000
            y: 36000
    - aws_esa_worldcover:
        path: s3://esa-worldcover/v100/2020/ESA_WorldCover_10m_2020_v100_Map_AWS.vrt
        rename:
          ESA_WorldCover_10m_2020_v100_Map_AWS: landuse
        filesystem: s3
        driver_kwargs:
          storage_options:
            anon: true

[esa_worldcover]
crs = 4_326
data_type = "RasterDataset"
driver = "raster"

[esa_worldcover.meta]
category = "landuse"
source_license = "CC BY 4.0"
source_url = "https://doi.org/10.5281/zenodo.5571936"
source_version = "v100"

[esa_worldcover.versions.legacy_esa_worldcover]
path = "landuse/esa_worldcover/esa-worldcover.vrt"
filesystem = "local"
driver_kwargs.chunks = { x = 36_000, y = 36_000 }

[esa_worldcover.versions.aws_esa_worldcover]
path = "s3://esa-worldcover/v100/2020/ESA_WorldCover_10m_2020_v100_Map_AWS.vrt"
filesystem = "s3"
rename = { ESA_WorldCover_10m_2020_v100_Map_AWS = "landuse" }
driver_kwargs = { storage_options = { anon = true } }

While the yaml maintains more of a visual hierarchy toml provides more flexibility in how to define these mappings. Note that even in the above example, this is not the only way to represent this structure within the toml format, which can be either an advantage or a disadvantage depending on the reader's viewpoints.

Prior art

While yaml is a widely supported format by big industry tools such as (but not limited too)

• Github Actions
• Kubernetes
• Docker-Compose
• Ansible
• Conda

It has also received some criticism:

• https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell
• https://dev.to/jessekphillips/stop-using-yaml-3kec
• https://noyaml.com/

Yaml aims to be mostly human readable. Some criticism has focused on the complexities that this can introduce. In addition the significant whitespace can be bad for it's compressibility making it more error propone and less portable between editors. Yet it still remains a very widely accepted format.

There are two main competitors to Yaml:

• Toml, which is discussed in this RFC. This language format has gained more popularity in recent years, being adopted by tools such as
  • Pypi (pyproject.toml) and with it tools such as Poetry
  • Rust/Cargo
  • Julia
  • Jekyll
  • Hugo
  • Zola
  • Pixi
• JSON. JSON is also used for configuration in many cases, but is considered less human readable, and lacks support for things like comments. To the author's knowledge this is why JSON has not been considered for this or previous proposals.

One well known fact that poses a risk here is the fact that once adopted it can be very hard to convince users to leave it behind, and thus to deperacte it down the line for any reason. Some examples that come to mind are:

• Our own ini file format.
• Python 2.7
• get-poetry.py

Therefore the decision to support another language should not be taken lightly. However, given the adoption of toml in the above mentioned tools and communities it is very unlikely that support for toml will wain in the coming years. It is equally unlikely that an objective winner will emerge from the yaml vs. toml debate, therefore there is a case to be made for supporting both.

Unresolved questions

• What should the CLI options for supporting toml be? (see above discussion for more detail)
• Should we provide migration options from yaml to toml in HydroMT itself?

Future possibilities

While outside the scope of this RFC, one could imagine a future where, due to the aforementioned uniformity across the Deltares software suite, HydroMT is encouraged to move towards Toml entirely. While this is not imminently happening to anyone's knowledge, a transitional period where we support both yaml and toml would make this transition easier to say the least.

[DirkEilander]

In my experience each software uses one file format for configuration, for instance Wflow and Ribasim use TOML; SFINCS uses its own input file format. For HydroMT we have decided to use yaml because of a few reasons: 1) it is an easy readable format where the key of each section corresponds to a method of a Model class, 2) it is widely used for workflows and catalog configurations, 3) it provides a more condensed format than toml for nested arguments. In my opinion, supporting different formats would only add confusion for users and to my knowledge their is no demand for TOML from users at this point. Furthermore, pro users can go around the CLI and use toml in their scripts in combination with the DataCatalog.from_dict method and the Model.build method if they want to. I therefore suggest to only implement toml support in the Model.read_config/write_config methods to support models like Wflow and Ribasim that use TOML, and not as possible configuration file format for the HydroMT configuration & datacatalog files.

[savente93]

Given that we haven't had any new perspectives added than the one Dirk added, I think we can safely assume that that is the consensus (i.e. only support toml for model configuration), so I'm going to close this RFC. The PR for the toml support is already there so I'll open that up for review. The ADR PR will come later.