Create external timeseries table specific for concentration timeseries results

[evetion]

The output from Delwaq is a concentration timeseries (possibly multiple per substance) per basin. We should be able to access it from a control node.

Relates to #520, but in discussion with Martijn, I propose to expand the Basin with an extra table, instead of having a new External node that somehow links to the same basin for a better user experience.

Added complexity, these fields would ideally be re-used in the online coupling via BMI later on.

[evetion]

After a discussion with @visr, this is an attempt to have a generic design for all issues involved (#1123), with a scope of waterquality; so substance + concentrations.

Need

For any substance/concentration calculations (whether in Delwaq or internal), we need:

1. A table defining initial conditions in the basins (some can start out salty, also needed for a continuity tracer Add continuity tracer  #1105)

2. Table(s) defining conditions on the boundary nodes, which can be static or timeseries. Note that uniquely, a basin has multiple boundaries.

3. A table for concentration(s) timeseries on basins which we can use for control (partial implementation of External data (for DiscreteControl) #520)

   • (a) in memory when we calculate concentration(s) ourselves
   • (b) as a table of results from Delwaq

Design

To add any number of substances and their concentrations, we think a long format works best, as we can hardcode the column names and thus use them in our code. Any table (1,2 or 3) would thus have at least the following columns.

node_id	time	substance	concentration
1	2020-01-01 00:00:00	Cl	3.14
1	2020-01-01 00:00:00	Continuity	1

Any substance name will do, it can be a valid substance code used in Delwaq, where any other name is accepted as a tracer like substance. As long as we do passive tracing, this (and the unit of concentration) doesn't matter. For active tracers, we might need to be clearer about the allowed substances (and prefix tracers or similar solutions) and the unit of concentration (SI).

The above table is enough to implement 1 and 3b, and could be merged(!) as one in a name like BasinConcentrationState (happy to take suggestions). For the initial conditions in 1, one would just supply a single (or empty?) time equal to the start timestep. For 3(b), the table would be overwritten by the results from Delwaq. If this is confusing, we can split into two tables: Case 1 is then BasinConcentrationState (without the time column) and 3b is BasinConcentrationTime.

Case 2 requires more schemas. We can repeat the above table as FlowBoundaryConcentrationState and LevelBoundaryConcentrationState, with the values linearly interpolated across the time domain. For the Basins we can do the same BasinConcentrationTime (help?), but we either add a type column with precipitation, drainage and urban_runoff, or add these names suffixed with concentration instead of the concentration column.

[gijsber]

Hi @evetion,
Having one BasinConcentrationState table with all timestamps and states in it is not very handy when we want to run Ribasim in a Fews Operational System. For that situation, where you need to do state updating to match the current situation, you would like to have a BasinConcentrationState (or BasinConcentrationInitialState table which holds a single state (for the start of the run) while a BasinConcentrationTime (or BasinConcentrationOverTime table holds the concentration over time during the simulation period. One of the tasks of the Ribasim-Fews adapter would be to pick the state at the end of the run and update the BasinConcentrationState table with this last value to become the new initial state for the next run.
For the boundaries, I would think that FlowBoundaryConcentration and the LevelBoundaryConcentration could work as table name (although ...BoundaryConcentrationState or ConcentrationTime could do as well).

For the (vertical) basin fluxes, I would recommend a table BasinForcingConcentrationTime (or BasinFluxConcentrationTime with additional column named forcing_type (or flux_type).

[Huite]

Having one BasinConcentrationState table with all timestamps and states in it is not very handy when we want to run Ribasim in a Fews Operational System. For that situation, where you need to do state updating to match the current situation, you would like to have a BasinConcentrationState (or BasinConcentrationInitialState table which holds a single state (for the start of the run) while a BasinConcentrationTime (or BasinConcentrationOverTime table holds the concentration over time during the simulation period. One of the tasks of the Ribasim-Fews adapter would be to pick the state at the end of the run and update the BasinConcentrationState table with this last value to become the new initial state for the next run.

For what it's worth, I'm having trouble following this. The Fews adapter could just append to the Time table, with newer timestamps? Ribasim will figure out which ones to use based on start and end time.

(This is a bit stream-of-consciousness-y, please indulge me:)

I feel it's probably smart to make a clear distinction between the concentration as a boundary for the control to act upon, and actually simulating conservative solute /tracer transport in Ribasim.

As a boundary, there's no need for an initial condition right? The concentration at any given moment is just defined by the transient table.

In case of simulating transport, it's another state. In that case, we do need an initial state (just like the basin levels). And indeed, we need to concentrations per boundary. But do you really need to mix both use cases?

Dumb question maybe: Why not support the external concentrations (Delwaq results) as just an abstracted ExternalTimeSeries (or whatever). It's just a lever for control; solute concentrations are obviously relevant, but it could also be something else (electricity price).
I can think of a strong argument against this, which is: if you switch from Ribasim solutes to Delwaq solutes, you have change your control schematization. Instead of listening to the external time series, it'll have to listen to a Basin concentration instead.

Anyway, I guess my suggestions are:

• Go ahead with implementing an external time series node. It's useful anyway and it would be sufficient to represent the Delwaq concentrations.
• Forget about Delwaq temporarily, and focus on what's necessary to get conservative solute transport working in Ribasim. The design of this should just follow what's going on for the water balance -- i.e. you need an initial state; and most pressingly, you need a table to reflect the concentrations of the incoming (also outgoing?) flows.

E.g. for the basin concentrations:

fid	node_id	time	substance	time	drainage	evaporation	infiltration	precipitation	runoff

(And bikeshed about whether "substance" should be called "species" or w/e, and the table name)

[gijsber]

For what it's worth, I'm having trouble following this. The Fews adapter could just append to the Time table, with newer timestamps? Ribasim will figure out which ones to use based on start and end time.
(This is a bit stream-of-consciousness-y, please indulge me:)

For sure, Ribasim will be able to pick the right moment in time from a BasinConcentrationStatefile. But from the Operational application perspective, we do not want an ever growing file where no one is taking care that history gets chopped off once in a while. Fews knows nothing about model files (including model specific state files) and will not chop off anything.

An alternative could be that we skip the notion of a state file and transfer the State as a (set of) timeseries value handed over by Fews (via NetCDF-->Arrow). This is pretty easy for Fews config to cope with, and it simplifies the adapter (now it conly becomes a file converter NetCDF-->Arrow, Shp+CSV-->gpkg). In that case we can have a BasinConcentrationState table with a time column included

I feel it's probably smart to make a clear distinction between the concentration as a boundary for the control to act upon, and actually simulating conservative solute /tracer transport in Ribasim.
As a boundary, there's no need for an initial condition right? The concentration at any given moment is just defined by the transient table.

Indeed boundary concentrations can just be a transient table. I would recommend to call these something like BoundaryConcentrationTime or BoundaryConcentrationTransient.