Notes on History TNG

Notes on History, The Next Generation

These are @WilliamJamieson's notes on History TNG to help document the current progress on this effort.

Overview

History TNG is intended to be a replacement for the current GEOS history component, which utilizes NUOPC's "asynchronous" capabilities. In theory, NUOPC should (or will be extended to) allow us to perform similar IO functions as our IO-server without the need for our current IO-server. Moreover, this process affords us the opportunity to "reimagine" how GEOS history operates and works.

Currently, GEOS history has accumulated lots of different use cases/configurations some of which have been added in a very ad hoc fashion over its lifespan. Supporting all of this while allowing for the ability to add new features is becoming increasingly difficult. Especially, due to the fact that choices in the configuration format are starting to become self-limiting. Hence, part of creating History TNG includes a thorough survey of the current GEOS history features, so that History TNG can be built so that it can accommodate all of these features in a sustainable way. In particular, this includes a redesign of the History input file from the HISTORY.rc file into a History.yaml file including the creation of a flexible infrastructure to parse the new file format in an easily extendible fashion.

I see three central goals of the History TNG project:

1. Creating an NUOPC based history component (or set of components).
2. Migration of the IO-server's capabilities into NUOPC.
3. Creation of an easily extendable yaml import format together with solid documentation of its limitations.

Note that I consider a functional history component to be a configurable component which can take in a collection (or set of collections) and write all the fields in that collection to a file.

Roadmap

Rocky Dunlap (@rsdunlapiv) has proposed what he calls the "scooter", "car", etc. method of developing History TNG. By this, he means that after every major developmental step (and hopefully at every point in between) we maintain an operational history component (not full featured, but a functional history component).

So the steps in this roadmap are as follows:

1. Create a "scooter" which is a history component which takes in an input file containing a single collection, and then writes that collection to a file. There should be no fancy features, basically writing all the fields in the collection to a file with no re-gridding, interpolation, or other "fancy" operations. This will be a severely limited history component, with the basic goal of creating a clean NUOPC based History Writer component together with the necessary changes to GEOS to allow NUOPC to facility the coupling of the components.
2. Create "scooter 2.0" which is the same as "scooter" accept that the model allows one to add multiple collections, via the creation of additional History writer components.
3. Create "Motorcycle". This would introduce the basic re-gridding and interpolation functionality that history currently supports. This could be done by adding NUOPC mediator components between the History Writer Components and GEOS. NUOPC mediators are intended to allow NUOPC to implement automatic re-gridding, interpolation, and other model interoperability features.
4. Create "Car". Here one might introduce a "main" history component which exists on both the history GEOS pet lists. This main history would attempt to orchestrate how all the History Writers are operating by assigning collections to different writers as needed. Here is where we start introducing IO-server capabilities into History TNG.

Note that this is a rough road map, which should be refined over time. In particular, all of the features that history currently supports should be implemented when appropriate in this timeline (I have a feeling that "motorcycle" will go through several iterations). The big question for me is if the NUOPC connectors are flexible enough to support the features that "car" will need.

Current status

Currently the project is between 50% and 60% on the basic "scooter" step. Here the goals were:

1. Create a functional NUOPC cap for GEOS which would allow the correct fields to make it to history (complete)
2. Create a History Writer which could write a collection to a netcdf file using MAPL's PFIO utilities.
3. Create the start of a new yaml format to configure the NUOPC cap and writer (complete).
4. Create a standalone test program to test the current state of History TNG (partially complete).

Overview of Current state of History TNG code

All history TNG code is currently contained on the feature/wjamieson/History_TNG branch of MAPL. The majority of the new code is contained in the gridcomps/History_TNG directory. The exceptions to this are the components of History TNG which require changes to the MAPL cap (the NUOPC cap), meaning those parts are contained within the gridcomps/Cap](https://github.com/GEOS-ESM/MAPL/tree/feature/wjamieson/History_TNG/gridcomps/Cap) directory.

Components of History TNG

In the History_TNG directory contains all the major objects that keep track of the collection configurations. In particular, I have taken care to carefully encapsulate all of the information needed by a history component so that in the future features should be both simple and have a natural home. Moreover, this approach allows the possibility of code reuse via inheritance with minimal refactoring.

The class structure of History TNG is nested starting with handling individual fields all the way through handling sets of collections. The majority of the initial work for History TNG involves the introduction of NUOPC, which requires us to figure out how to pass individual fields from MAPL to NUOPC. This has lead to an initial focus on carefully handling how History TNG stores information on the fields it needs to work with.

History TNG's handling of Individual Fields

For the "scooter" version of History TNG, it was extremely important to describe how to handle each field the History TNG will need to deal with. This is so that the NUOPC cap has enough information so that it can reach into GEOS through MAPL and extract the necessary fields for History TNG to then write to files and likewise have enough information for the History Writer to tell NUOPC which fields it needs passed to it. The primary class for describing this is the FieldEntry class.

FieldEntry requires two primary pieces of information for a given field, namely:

1. The MAPL short name
2. The MAPL component name

These are needed so that the NUOPC cap and MAPL can find the correct fields in MAPL in order to pass them up into NUOPC. It has more configurable information, so that one could configure the NUOPC configuration, namely:

1. The field units (NUOPC requires that one registers the units for a field, and at the current time it is easier to let the user specify these or allow the default NUOPC unit than it was to derive this information from MAPL. Doing so was on my TODO list for future versions of History TNG. Moreover, allowing the user to specify units could allow for automatic unit conversions in History TNG.) Currently, this defaults to "1" if not specified.
2. TransferOfferGeomObject This is a NUOPC option which allows components to agree on which component or components can specify the geometry (grid) that the field uses so that NUOPC can attempt to use its automatic re-gridding capability. Currently, this defaults to the NUOPC default which is "will provide".
3. SharePolicyField This is a NUOPC option which allows for direct pointer sharing between components. Currently, this defaults to the NUOPC default which is "not share".
4. SharePolicyGeomObject This is a NUOPC option which allows a field to set the geometry (grid) for another NUOPC component. Currently, this defaults to the NUOPC default which is the value SharePolicyField is set to.

Note that the NUOPC configuration options are how NUOPC negotiates among components to allow for fields to be transferred. The NUOPC defaults will work for the current "scooter" version, but they may need to be tweaked for later versions of History TNG. Moreover, a more general NUOPC cap for all of GEOS will need to be able to individually configure all these options.

FieldEntry contains all the methods needed to handle a single field with History TNG such as the necessary advertise and realize steps that NUOPC needs to perform in addition to other book-keeping tasks. In the future, I expect that per-field history configuration options would be implemented at this level.

The FieldEntryMap is simply a gFTL string-FieldEntry map.

For the purposes of the NUOPC cap, I needed to introduce a collection of all fields that History TNG will need to write across all active collections. Thus I wrapped a FieldEntryMap inside a class to form the FieldRegistry. The field registry handles all the configurable portions of History TNG which requires special behavior by the NUOPC cap. This primarily is advertising and realizing all the fields that GEOS needs to export into NUOPC for the current history configuration.

In addition at the driver level that spans the PETs of both the model and writer the registry must add the fields to the NUOPC_FieldDictionary. These are currently registered as the variable_short_name.component_name.

Groups

@tclune noted that History really needed a notion of sets of fields which could be configured in different ways for different collections, in order to reduce the duplication of information in the configuration file. This lead to the notion of a Group, which is a set of fields that one can define together.

In order to implement this I need to break the it into a few smaller parts. First, we needed a FieldGroupEntry which wraps the FieldEntry class together with the additional information that the History writer will need to know about the field, in this case the field's optional alias_name. However, more information can be included at this point. The FieldGroupEntryMap is simply a gFTL string-FieldGroupEntry map. While a FieldGroup is a class to contain all the collective operations that one would want to do on a set of fields inside a Group.

The Group class itself currently consists of two FieldGroups. This is because ultimately Groups of fields should be able to contain expressions of fields and fields defined via regular expressions. This means a group needs to be able to define fields that need to be output by history directly, the fields variable, and the fields that History TNG needs access to so that it can perform operations which do not need to be output directly, the aux_fields variable. Note that when combining groups it is possible to have fields that appear in both categories by happenstance; this is automatically taken care of by removing fields from aux_fields which are added to fields (this is necessary for providing accurate accounting for NUOPC advertise and realize).

The layers needed to construct the Group class should allow enough flexibility to handle all the actions one might wish to define in history on a field or set of fields.

Note that since Groups are meant to be defined outside of collections, I needed to introduce a GroupMap class, which is a gFTL string-Group map, and then a GroupRegistry class to wrap the map with the collective group operations (primarily reading all the groups out of the yaml file).

Collections

In the current version of history we are primarily concerned with collections, which are a set of fields together with the information on how to write these fields to a file. In History TNG this is implemented by defining a collection as a list of pre-defined groups, a set of fields (defined as a group unique to the collection), and then any other necessary information for creating the file.

All of this is performed by the Collection class. Note that in particular, the yaml configuration of a collection will reuse the notation used for that of a group, but then adds additional keys to define the rest of the information needed by the collection.

Note that currently, the History TNG collection contains a minimal set of information to create very simple file using the PFIO utilities. This means many features will need to be added to the Collection class in order to fully support all current history features. My current plan is to introduce a wrapper class for each new feature's information which will perform the necessary operations where the interface is passed up and through the Collection class. Currently the Frequency and Template classes are just wrappers on the information being read from the yaml file, the next steps will be to add methods to these classes to produce the data to configure PFIO correctly.

Note that to fully define the History Configuration, I needed to introduce a CollectionMap and CollectionRegistry which serve the same purposes as the GroupMap and GroupRegistry classes respectively.

HistoryConfig

The is HistoryConfig class is simply the class that orchestrates reading and parsing the History.yaml file in order to produce the FieldRegistry for the NUOPC cap and a Collection for each of the history writer components.

Notably in addition to containing a GroupRegistry and a CollectionRegistry the HistoryConfig also lists the enabled collections (the ones history will actually output). There are plans to add more options to this object including:

1. A versioning system
2. A list of possible grids for use in collections
3. A list of "vertical" (non-grid dimension) defined interpolation/sampling methods for us in collections.
4. Any other global options for all history outputs.

The HistoryWriter

Currently the History Writer component has not been implemented. In theory it should be implemented in the HistoryWriter class which should wrap a single collection generically so that it can be sub-classed to support different file writing utilities one of which being PFIO.

Since ESMF has very strict interfaces, it is necessary to create a "dual" module to the HistoryWriter class which accesses the underlying class and then wraps the HistoryWriter methods in the necessary subroutines to form a proper GridComp. I intend for this to go into the HistoryWriter_GridComp.F90 file.

The HistoryCap

The HistoryCap class acts as the NUOPC cap for History TNG. Like the HistoryWriter it requires a "dual" module to handle being a "true" NUOPC cap, see NUOPC_HistoryCap.F90. The HistoryCap implements all the necessary phases and registrations necessary to pass the fields required by a History TNG configuration out of GEOS and into NUOPC.

Note that I have marked in several places some TODOs concerning ESMF 8.1.0. Once GEOS moves to ESMF 8.1.0, significant cleanup of the HistoryCap can occur because NUOPC_CompSpecialize will support specializing NUOPC to custom created advertise and realize routines, instead of the current method of carefully overriding some of the generic NUOPC initialization phases via the NUOPC_CompSetEntryPoint methods. These changes will ultimately be safer and easier to read. When properly introduced, the changes laid out in the HistoryCap TODOs will allow one to also eliminate the need for the NUOPCmap class, which handles this careful overriding of NUOPC initialization phases.

Note that it was found that the %export_state of the MAPL_CapGridComp needs to be passed down the FieldRegistry and ultimately to the FieldEntry. The HistoryCap was passing down the export state from the "model" or ESMF_GriddedComponent used by the NUOPC_Cap. However, the fields we care about are in the %export_state of CapGridComp. So this needs some thought and makes the interfaces less elegant. I do not believe we can put these in the export_state of the main GC as advertise is creating empty fields and realize replaces this empty field with a new field of the same name you pass. So we really can't mess with this.

Changes to MAPL_CapGridComp

I have made a few changes to MAPL_CapGridComp in order to clean up the interfaces that allow NUOPC to reach into GEOS. These interfaces were originally created for the online-ctm project, and were quite cumbersome and prone to error (a lot of programming by coincidence). Most of the changes center on a refactoring of the initialize_extdata method by splitting the NUOPC related portions into their own methods which are then just wrapped by initialize_extdata. I have taken care to still support the original NUOPC interfaces, which rely on information being present in CAP.rc. The code should fall back on these if one does not allocate the field registries that have been added to the MAPL_CapGridComp. Since the HistoryCap creates and holds a pointer to the MAPL_CapGridComp it can then allocate and fill these registries for MAPL_CapGridComp so that when initialize is called the new cleaner NUOPC interfaces are used.

Note that we may want to do a merge of History TNG into MAPL develop soon, as I keep needing to hand merge the changes to the MAPL_CapGridComp due to me extending the interfaces of some of the subroutines.

The testing_app Directory

The 'testing_app` directory contains a shell of a program to test History TNG. Currently this program has not been modified from the state that I was using it when I was testing the NUOPC cap for GOCART and the NUOPC cap for the Online-CTM. In spite of this, it should be relatively easy to modify the current code in order to produce a GEOS independent (only using MAPL) testing application to demonstrate History TNG.

The basic idea to produce a testing application is to use the Provider_GridComp as a root GridComp in MAPL with the intention of it creating and filling fields (providing fields) which can then be written to files by History TNG. Indeed, one should create a child component for Provider_GridComp so that we can test History TNG writing from non-root components as well. Then the HistoryWriter_GridComp can directly replace the function of the Reciever_GridComp in this program, as the Reciever_GridComp was intended to test if a NUOPC component could receive information via NUOPC from MAPL. The UFS_TestingCap.F90 file is currently redundant; however, I have left it in place as an example of a NUOPC compliant GridComp component (though for the GOCART case it was replacing the Provider_GridComp not the Reciever_GridComp).

The tests Directory

The tests contains all the unit tests that I have written to cover History TNG. These should be fairly complete and currently run without any issues. Note that this directory also includes some example yaml files for how History TNG can currently be configured.

Next Steps in History TNG Code Development

The next major step is that "scooter" version of History TNG needs to be completed, this is primarily finishing the HistoryWriter class so that it can actually write a collection to a file. Once this is finished the testing_app should be completed so that all of History TNG can be easily exercised.

Once the "scooter" and testing_app are complete, I would create a driver and application version for the GEOSgcm so that a realistic system test can be performed on History TNG. Once this is done, I think the next "scooter" version should be attempted where one adds multiple History writers so that multiple collections can be written. A good test would be to then write all the collections normally associated with a GEOS run using History TNG. Note that for this test I would still strip out all the non-implemented features and instead just try to replicate History TNG outputting all the collections in terms of their basic fields in a simplistic fashion, which simply exercises the scale of a typical GEOS History configuration.

Once the basic "scooter" is completed, I think there should be a basic evaluation of what history features can be added at this point, which do not increase the complexity of the NUOPC architecture, that is require additional components to implement. These should be implemented and fully tested at this point.

After this initial survey, I think the "motorcycle" version should be implemented, in which NUOPC mediator components should be introduced between each writer and GEOS. These mediators handle the negotiation of fields from GEOS that require re-gridding, interpolation, and other operations in order to meet the criteria for what the History Writers want to write. This keeps the History Writer components themselves pretty simple because they simply write the fields they are given by NUOPC in the way the user has defined. In other words the History Writer should not have to touch the data passed to it by NUOPC; instead, the mediator component would be responsible for altering the fields to meet the users configuration before that field reaches the History Writer. In NUOPC this makes the most sense as mediator components are intended to allow components which have differing definitions and needs for fields to have an intermediate component to which accounts for these differences.

Once a mediator, history writer component pair have been introduced and tested, I believe that the majority of non-IO related features (those features that do not pertain to how History manages its resources and disk) of History can be implemented in History TNG. This should be done carefully and the initial version of a History.yaml file should be finalized. At this point automated conversion scripts may need to be written to convert HISTORY.rc files to History.yaml files.

At this point History TNG should have a rich feature set and closely replicate how History currently operates. The IO-server related tasks should now be addressed for the "car" version of History TNG. This will involve attempting to manipulate the NUOPC connectors so that the writers and their Mediators can be made more flexible. I believe a lot of IO-server can be implemented via careful manipulation of the NUOPC connector's state using the setting and checking of attributes. This is because a connector's state only exists for user customization, such as controlling a writer via its connector from some main component. Alternate approaches may be better, but they will require input from the ESMF's NUOPC developers.

Notes from current attempt to take over project

These are my observations and things I have had to fix to get the testing app working with the provider grid comp and a fake writer that imports a field.

1. The FieldRegistry did not have a routine to actually register the fields with the nuopc dictionary which must be done at the driver level.
2. The Realizing taking place in FieldEntry on the GEOS side needs to ultimately have the %export_state from CapGridComp passed as those are where the fields are that need to be realized, not in the state associated with the model GC from the top.
3. Really the writer should get the grid from the model and have nuopc transfer it. This means that the advertise and realize are different on the model and writer side as the advertise on the writer and 'cannot provide' for the geometry object and the realize allocates the field unlike on the model side.
4. The writer will need an acceptTransfer method to make a new grid with a new delist so that there are not multiple DE's on the writer side since they will in general not be on the same number of PEs.
5. When moving to 8.1 the initialization of GEOS must be called from one of the specialized methods in the NUOPC_HistoryCap so it goes into advertise.