[EPIC] LOFI Probabilistic FIM Project

[LorneLeonard-NOAA]

This is an EPIC card. As items from this list are addressed, their active cards will be linked.

Earlier Prototypes 2023/2024

Some of the earlier HV prototypes can not be re-used. We will keep an eye on those PR.

Big Picture

Source Data -> Process & Prepare Data ->Publish Services 

Phase 1: Start with example dataset

• Step 3: Source
• Step 4: Process
• Step 5: Publish
• Feedback from leadership
• Feedback from end-users

After Phase 1

• Scale the amount of data (1/3 of total)
• Fine tuning hardware and software 
• Increase the amount of data (2/3 of total)
• Fine tuning hardware and software 
• Full scale testing

DEV(TI) -> UAT -> PRD

NOTE: This is the working name “Probabilistic Flood Inundation Mapping” will be modified in the future.

[1] Sensitive Information Locations [WILL NOT BE visible in github]

• Secret manager
• Google Drive

[2] Infrastructure Setup for Development Environment

S3 Bucket(s)

• Folder naming conventions
• Optimizing Performance
• Measuring Performance

EC2 Instance(s)

• Windows
• [ In Progress] Linux for Viz and FIM dev.
• Optimizing Performance
• Measuring Performance

Folder structure conventions

• Data
• Measuring
• Optimizing 
• Software

Software conventions

• Install Locations
• Version control

[3] Source Data

Gregory Petrochenkov is responsible for generating the Probabilistic Flood Inundation Mapping. 
Greg’s PR code is currently here. 

• Probabilistic Flood Inundation Mapping needs to be merged with the FIM/HAND
• Estimated time when this will be official?
• HydroVis needs this first.

Greg’s PR code for inundation here

Hardware: 

• Will be run on XXX EC2 instances.

Big Question: Where do we run Greg’s code? With existing FIM/Hand EC2, or separate?

Main Steps:

[Part 1 in FIM git code repo]

• Create Params.csv file 
  • Depends on NWM version, not HAND version
  • • Needs testing
  • Not merged into FIM
  • Ran against 4.5.2.11 (i.e. behind current HAND version), for dev will not update.
  • • Long term will be run per HAND release.
• Execute probabilistic_distribution_parameters.py
• Requires FIM/HAND output directory location
  • • Need to confirm access
• Name of the output files.
  • • Need to confirm folder and naming structures.

[Part 2 HAND distribution output]

💯 Priority-1

• # FIRST Task Greg, Robert, and Lorne will run Greg’s work on EC2 to understand all data steps.
  Running Greg’s work on EC2 to understand all data steps #1040

- How often is this run?  We will run it on all HAND (PROD / Release) versions but not necessarily all HAND versions. (not counting FIM test UAT runs)

- TBD: Likely run on the FIM side once per HAND prod set (BED) but it might be changed to be run in the HV side (costs, disk space, etc)

- Jan 6, 2025 - status:code is done.. needs review for this part of the script. Rob and Greg will be tightening it up. aka) good naming conventions, patterns, etc.

[Part 3 Prob FIM processing (Inundation)]

Creating presentation vector features (percents).
Greg is building a new mosaic engine.

inundate_mosaic_wrapper_optimized.py
inundate_gms_optimized.py
inundate_optimized.py

mosaic_inundation.py [LINK]

overlapping_inundatiion.py

Currently the mosaic creates a raster dataset.

    overlap.merge_rasters(project_path + "/merged_final5.tif", threaded=False, workers=4)

• [TODO] Evaluate adding code to create geometry file here.

• Compute costs
• Network costs
• Disk costs

• [TODO] Confirm location of S3 bucket output files

[4] Process Data

• Windows

  • Relying on ESRI tools for processing

• [In Progress] Linux Flavor

  • Using QGIS and open source for processing

• [In Progress ] Convert Raster datasets to Polygon datasets 
  [LoFI] Convert Raster datasets to Polygon datasets with sample dataset #1031

  • Benchmark
    • • Disk Speeds
    • • Memory Usage
    • • Network Usage
    • • Disk Size

• Schema requirements

  • Time
  • Probability Value
  • Version?
  • ???

• Any spatial joins required? If so, where is the data stored? 

• Are joins required via database?

• Can/Should the polygon geometry be simplified?

  • Needed for zoom scale, ESRI’s map server services do this to a point.
    * No control of simplification using ESRI services.
  • Could implement custom filtering and simplification to improve performance.
    * Would mean more layers within the published services. 
    • Is this a problem for end-users?
      * Benchmarks
    • Compute and storage costs
  • If spatial joins are required, this means geometry simplification will be the later steps.

• What type of visualizations?

  • Heat maps?
    • Need to convert polygons to points.
      • Criteria for centroids?

[Step One] For early DEV testing

Manually publish Map via Pro to k8/enterprise services.

• PostgreSQL as a DataStore 
  • Benchmarks
    • • How are the dbs configured?
    • • Backup expectations?
    • • Network Usage
    • • Disk Size
  • Using ArcGIS Server provide PostgreSQL connection information
  • Register the connection file
  • Publish Service Layers from Pro connection file

[Step Two] For UAT and PRD

• Modify the HV workflows. 
  • Follow Dynamic Vector Services patterns
    • • Duplicate LAMBDA/viz_function templates
    • • Configure database setup
    • • Create SQL queries for spatial joins etc
    • • Setup maps files for publishing.

[5] Publish Data

Part One: Testing workflows to process data for publication

• [Ripple and LOFI] Testing workflows to process data for publication #1051

• Windows

  • Relying on ESRI tools for processing

• Ubuntu

  • Using QGIS and open source for processing

• [] Convert Raster datasets to Polygon datasets 

  • Benchmark
    • • Disk Speeds
    • • Memory Usage
    • • Network Usage
    • • Disk Size

• Name of Service

• Vector

• Raster

• Multiple layers

Types of Services:

• MapServer
• FeatureServer
• OGC

Service publication requirements

• Symbology
• MetaData
• Query Capabilities
• Service Description
• Zoom Scale
• Tags

One Layer only?
Break up intervals into Layers??

ESRI enterprise environment?

• Kubenetes? HV is currently migrating TI to k8.
• Check license requirements
  • Image services
• [In Progress] Asked Robert Van and Shawn Crawley for confirmation of data store setup