Incompatible with Precompiled DFM

[rqwang]

I found I cannot directly import the model created by dfm_tools in the precompiled DFM (I am using the version of Delft3D FM Suite 2023.01 HMWQ). I feel this is due to the different parameter names in the files. If there is no such a tool to resolve the conflict, I plan to write a script to do it. Please comment.

Thanks.

[veenstrajelmer]

Thanks for creating this issue. We are aware of it and people are working on a fix. The issue is with the D-Flow FM DeltaShell GUI, since it only supports very specific file formats, quite a bit stricter than the FM kernel. Therefore, the model should run properly with the executable that is shipped with the DFM GUI, but I am not sure if that helps in your case.

You can use the code below to import/visualize the model in the D-FlowFM GUI without errors (updated 12-12-2023), but the comments in the code describe what the DFM GUI can not successfully read even after these changes. It is tested for Delft3D FM Suite HMWQ versions 2023.01 to 2023.03. The code also contains links to (internal) JIRA issues in which the remaining issues are documented:

import os
import glob
import shutil
from hydrolib.core.dflowfm import FMModel
import pandas as pd

dir_model = r"c:\DATA\dfm_tools\docs\notebooks\Vietnam_model"

# copy the model files to a new DS directory to avoid messing up the original files
dir_model_ds = f"{dir_model}_DS"
os.makedirs(dir_model_ds, exist_ok=True)
for file_name in os.listdir(dir_model):
    # construct full file path
    source = os.path.join(dir_model, file_name)
    destination = os.path.join(dir_model_ds, file_name)
    # copy only files
    if os.path.isfile(source):
        shutil.copy(source, destination)

# remaining warnings
# Could not parse quantity uxuyadvectionvelocitybnd into a valid flow boundary condition	2023-12-08 23:35:42.713
# Omitting line 5 not starting with [forcing]	2023-12-08 23:35:42.660
# Omitting line 4 not starting with [forcing]	2023-12-08 23:35:42.660
# Omitting line 3 not starting with [forcing]	2023-12-08 23:35:42.660
# salinitybnd/temperaturebnd (t3d) are not visible in DeltaShell
# JIRA epic about import/export improvements: https://issuetracker.deltares.nl/browse/D3DFMIQ-3223

# first we make all paths relative to make our copied model work
# hydrolib-core always uses forward slashes in directories so that is what we will replace here
# https://github.com/Deltares/HYDROLIB-core/issues/532
# temporary fix added in dfm_tools: https://github.com/Deltares/dfm_tools/pull/723
file_list1 = glob.glob(os.path.join(dir_model_ds, "*.mdu"))
file_list2 = glob.glob(os.path.join(dir_model_ds, "*.ext"))
file_list = file_list1 + file_list2
dir_model_forward = dir_model.replace("\\","/") + "/"
for file_one in file_list:
    with open(file_one) as f:
        file_contents = f.read()
    file_contents = file_contents.replace(dir_model_forward, "")
    with open(file_one, "w") as f:
        f.write(file_contents)

# disabling oldextfile, because
# Unexpected line "VARNAME=msl u10n v10n chnk" on line 101 in file Vietnam_old.ext and will be ignored.	2023-12-08 22:27:07.240
# Unexpected line "VARNAME=initialwaterlevel" on line 94 in file Vietnam_old.ext and will be ignored.	2023-12-08 22:27:07.220
# System.ArgumentException: Cannot construct spatial operation for file initialwaterlevel_2022-11-01_00-00-00.nc with file type 11
# if we try to comment the things that raise issues, the extfile is sometimes even deleted.
# unexpected varname: https://issuetracker.deltares.nl/browse/D3DFMIQ-3204 (expected in 2024.01 release)
# netcdf forcing in itself also results in a separate error: https://issuetracker.deltares.nl/browse/D3DFMIQ-3513
file_list = glob.glob(os.path.join(dir_model_ds, "*.mdu"))
for file_one in file_list:
    # convert datetime strings to tstart/tstop nums
    # was resolved in https://issuetracker.deltares.nl/browse/D3DFMIQ-3206 (expected in 2024.01 release)
    mdu_hcdfm = FMModel(file_one, recurse=False)
    tstart_dt = pd.Timestamp(mdu_hcdfm.time.startdatetime)
    tstop_dt = pd.Timestamp(mdu_hcdfm.time.stopdatetime)
    refdate = pd.Timestamp(str(mdu_hcdfm.time.refdate))
    tunit = mdu_hcdfm.time.tunit
    assert tunit=="S"
    tstart_num = (tstart_dt - refdate).total_seconds()
    tstop_num = (tstop_dt - refdate).total_seconds()
    mdu_hcdfm.time.tstart = tstart_num
    mdu_hcdfm.time.tstop = tstop_num
    mdu_hcdfm.save(file_one)
    
    with open(file_one) as f:
        file_contents = f.read()
    file_contents = file_contents.replace("\nextForceFile ", "\n# extForceFile ")
    with open(file_one, "w") as f:
        f.write(file_contents)

# replace in bc files
# JIRA epic for bc file parsing: https://issuetracker.deltares.nl/browse/D3DFMIQ-3487
file_list = glob.glob(os.path.join(dir_model_ds, "*.bc"))
for file_one in file_list:
    with open(file_one) as f:
        file_contents = f.read()
    file_contents = file_contents.replace("[Forcing]", "[forcing]")
    file_contents = file_contents.replace("vertPositions", "Vertical Position Specification")
    file_contents = file_contents.replace("vertInterpolation", "Vertical Interpolation")
    file_contents = file_contents.replace("vertPositionType", "Vertical Position Type")
    file_contents = file_contents.replace("timeInterpolation", "Time Interpolation")
    file_contents = file_contents.replace("vertPositionIndex", "Vertical Position")
    
    if not "= t3d" in file_contents.lower():
        # this renaming must not be done for function=t3d
        low_to_caps_list = ["name", "function", "offset", "factor", "quantity", "unit"]
        for low_to_caps in low_to_caps_list:
            file_contents = file_contents.replace(low_to_caps, low_to_caps.capitalize())
    
    with open(file_one, "w") as f:
        f.write(file_contents)

If you have any additions to this code, please leave them in the comments and I will update the code above again. It will probably not become a dfm_tools function, since the model is valid already. Also, I hope that the next version of the DFM GUI does support the file formats of HYDROLIB-core package that dfm_tools uses to write the model files.

[rqwang]

Thanks for the update. I found the time section in the mdu file also need some work. Here is my code:

## -----------------------------------------------------------------------
# Update the MDU file
# Open the file and read the lines
filename = 'xx.mdu'
with open(input_path+filename, 'r') as file:
    lines = file.readlines()

# Update the headlines
lines = [line.replace('[Geometry]', '[geometry]') for line in lines]
lines = [line.replace('[Numerics]', '[numerics]') for line in lines]
lines = [line.replace('[Physics]', '[physics]') for line in lines]
lines = [line.replace('[Wind]', '[wind]') for line in lines]
lines = [line.replace('[Waves]', '[waves]') for line in lines]
lines = [line.replace('[Time]', '[time]') for line in lines]
lines = [line.replace('[Restart]', '[restart]') for line in lines]
lines = [line.replace('[External Forcing]', '[external forcing]') for line in lines]
lines = [line.replace('[Trachytopes]', '[trachytopes]') for line in lines]
lines = [line.replace('[Output]', '[output]') for line in lines]

# Update the lines
lines = [line[0].upper() + line[1:] if line else line for line in lines]

# Define the new value for TStop
start_line = [line for line in lines if line.strip().startswith('StartDateTime')][0]
stop_line = [line for line in lines if line.strip().startswith('StopDateTime')][0]


# Extract the datetimes from the lines
start_datetime_str = start_line.split('=')[1].split('#')[0].strip()
stop_datetime_str = stop_line.split('=')[1].split('#')[0].strip()

# Parse the datetimes
start_datetime = datetime.strptime(start_datetime_str, '%Y%m%d%H%M%S')
stop_datetime = datetime.strptime(stop_datetime_str, '%Y%m%d%H%M%S')

# Calculate the difference in seconds
difference = (stop_datetime - start_datetime).total_seconds()

new_value = str(difference)

# Update the lines
lines = [line.split('=')[0] + '= ' + new_value + '      #' + line.split('=')[1].split('#')[1] if line.strip().startswith('TStop') else line for line in lines]

# Write the updated lines back to the file
with open(output_path+filename, 'w') as file:
    file.writelines(lines)

## ------------------------------------------------------------------------
# Update the bc file
filename = 'tide_NJCoast_tpxo80_opendap.bc'
with open(input_path+filename, 'r') as file:
    lines = file.readlines()

# Update the headlines
lines = [line.replace('[Forcing]', '[forcing]') for line in lines]
lines = [line[0].upper() + line[1:] if '=' in line else line for line in lines]

# Write the updated lines back to the file
with open(output_path+filename, 'w') as file:
    file.writelines(lines)

[veenstrajelmer]

I see indeed. I have updated my code to parse the tstart/tstop datetimes with hydrolib-core and convert them to the numeric representation in seconds. I also looked at the rest of your changes for the mdu, but I was able to import the model without the capitalization/lowering of the keywords/headers in 2023.01 to 2023.03. I would guess that also works on your end. I will close this issue for now, since it is not a dfm_tools problem. But feel free to add new comments if you have additional insights.

I have added some links to JIRA issues, even though they are not public. Please know that this issue is actively being worked on. Part of the problems are already resolved in version 2024.01 of the GUI. I expect the next version (or maybe the one after that) will directly support importing models generated with dfm_tools (HYDROLIB-core). I hope this helps.