Documentation (#20)

* added readme and cleaned requirements

* linted

* updating docs

* added publish.md

* added rabbit mq docs

* added post processing docs

* added the last of the docs/ files

* added the rest of the README.mds

* updated compose

Author: taddyb

Source/RnR/Dockerfile.docs

@@ -9,7 +9,7 @@ Key Features of Replace and Route:
 - Uses docker compose to manage architecture
 - Uses a jupyter-notebook to view and manage code through a container
 
-<img src="docs/API_spec.png" alt="isolated" width="750"/>
+<img src="docs/photos/API_spec.png" alt="isolated" width="750"/>
 
 ## Installation
 

Source/RnR/INSTALL.md

@@ -68,7 +68,6 @@ services:
     build: 
       context: .
       dockerfile: Dockerfile.app
-    # image: ghcr.io/taddyb33/hydrovis/rnr:0.0.1
     volumes:
       - type: bind
         source: ./data

Source/RnR/README.md

@@ -1,3 +1,19 @@
 # Data dir
 
-This dir's purpose is to store data files generated by replace and route
+This dir's purpose is to store inputs and data files generated by replace and route. The following folder structure is:
+```
+- plots/
+  - All hydrograph plots generated by RnR
+- logs/
+  - internal service logs for the publisher-app and consumer
+- replace_and_route/
+  - post processed outputs for replace and route
+- rfc_channel_forcings/
+  - T-Route inputs generated from RFC forecasts
+- rfc_geopackage_data/
+  - T-Route gpkg files containing spatial domain information for each RFC point
+- troute_output/
+  - T-Route output .nc files
+- troute_restart/
+  - restart files generated by T-Route for warmstarts
+```

Source/RnR/compose.yaml

@@ -0,0 +1,16 @@
+# Assimilation
+
+Based on the PWS requirements (2.3.4.1.4) we must assimilat forecasts that have flooding. 
+
+```
+Assimilate RFC forecasts that have values within the forecast horizon that are at or above
+flood stage as defined by the local NWS field offices. RFC forecasts shall be assimilated
+upstream of the RFC forecast location at a distance greater than 2 miles, but not to exceed 5
+miles
+```
+
+To account for this, we are inserting flow using the upstream catchment of the RFC point using enterprise hydrofabric connectivity. Since there is a many to one relationship between catchments to nexus points, we just have to use the direct catchment upstream. 
+
+Below is a photo of the assimilation in action. The grey catchments are areas where there is no flow, and the green catchments are areas where there is flow. 
+
+![alt text](photos/assimilation.png)

Source/RnR/data/README.md

@@ -0,0 +1,18 @@
+# Common Errors:
+
+## NoDownstreamLID Error
+
+If there is no downstream RFC point detected, we cannot determine how far to route the flow
+
+<img src="photos/error_example_1.png" alt="isolated" width="750"/>
+
+## NoForecastError
+
+If there is an Empty NWPS response, we can't route flow
+
+<img src="photos/error_example_2.png" alt="isolated" width="750"/>
+
+## LID not detected
+
+Since we are using a mock database with only 170/3000+ RFC points, it is very common for a downstream RFC point to not be included in the mock database. Thus, when running RnR with docker compose you will often see a PydanticValidationError for LID not detected (there being a None where there should be rfc downstream data)
+

Source/RnR/docs/assimilation.md

@@ -0,0 +1,44 @@
+# Consumer 
+
+![alt text](photos/consumer.png)
+
+## What is its job?
+
+The data consumer pulls messages from the message queue and runs a series of microservices on the message json body:
+1. Reads in the message
+2. Processes the forecast into T-Route inputs
+3. Determines the HYFeatures ID
+4. Runs T-Route
+5. Post-processes and plots the data
+
+## How is this accessed
+
+The consumer is an asynchronous task that is spun up using the docker compose and will await messages from the queue
+
+```yaml
+  consumer:
+    build:
+      context: .
+      dockerfile: Dockerfile.app
+    restart: always
+    volumes:
+      - type: bind
+        source: ./data
+        target: /app/data
+    environment:
+      - PIKA_URL=rabbitmq
+      - RABBITMQ_HOST=rabbitmq
+      - SQLALCHEMY_DATABASE_URL=postgresql://{}:{}@{}/{}
+      - DB_HOST=mock_db
+      - REDIS_URL=redis
+      - SUBSET_URL=http://hfsubset:8000/api/v1
+      - TROUTE_URL=http://troute:8000/api/v1
+    command: sh -c ". /app/.venv/bin/activate && python src/rnr/app/consumer_manager.py"
+    depends_on:
+      redis:
+        condition: service_started
+      troute:
+        condition: service_healthy
+      rabbitmq:
+        condition: service_healthy
+```

Source/RnR/docs/common-errors.md

@@ -0,0 +1,5 @@
+# Front end Web Services
+
+A web frontend will be set up as part of the Docker compose process, with the endpoint /frontend/v1/plot/, which can be accessed in a browser. This allows searching of all the PNG plot files and the Replace and Route NetCDF data generated by the API. First a LID needs to be selected. Then any files that fall within the date range selected by the user will display on screen, in a tabbed interface grouping them by feature ID, and then by forecast date. The search can be narrowed down to any date range desired (though result sets may take significantly longer to load for large date ranges or for LIDs with forecasts for numerous feature IDs). There is also a "Download Zip" option that will take the currently displayed result set and package it in a Zip file for the end user.
+
+![alt text](photos/front_end.png)

Source/RnR/docs/consumer.md

@@ -0,0 +1,33 @@
+# Hydrofabric usage
+
+We are using flat files generated by the HFsubset endpoint prior to RnR being run. The HFsubset endpoint, similar to T-Route, is managed by the following compose entry:
+```yaml
+  hfsubset:
+    image: ghcr.io/taddyb33/hfsubset-legacy:0.0.4
+    ports:
+      - "8008:8000"
+    volumes:
+      - type: bind
+        source: ./data/rfc_geopackage_data
+        target: /app/data
+    command: sh -c ". /app/.venv/bin/activate && uvicorn src.hfsubset.app.main:app --host 0.0.0.0 --port 8000"
+    healthcheck:
+      test: curl --fail -I http://localhost:8000/health || exit 1
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 5s
+```
+
+## What is its job?
+
+Hfsubset is used to create subsets of the hydrofabric v20.1 geopackages that end at a specified location. We make both upstream RFC and downstream RFC subsets, then use them for determining catchments of origin, connectivity, etc. These flat files are fed to T-route using a shared volume. 
+
+## How is this accessed
+
+Subsets are generated by the following localhost endpoints on localhost:8008/docs:
+- /api/v1/subset/
+- /api/v1/downstream
+
+If you want the whole dataset, you can ping the following endpoint from localhost:8000/docs:
+- /api/v1/rfc/build_rfc_geopackages

Source/RnR/docs/docker-compose.md

@@ -0,0 +1,3 @@
+# IaC
+
+For all information regarding IaC visit `Source/RnR/terraform` and read their README.md file

Source/RnR/docs/front-end.md

@@ -0,0 +1,23 @@
+# Message Broker and Cache
+
+The message broker is a piece of software that is used to take message bodies from the Publisher, sort them, then post the messages. Redis caching is used to make sure we are only running the workflow as required when there are new forecasts
+
+![alt text](photos/message_broker_and_cache.png)
+
+## What is its job?
+
+Hold messages in a queue based on priority so the consumer can successfully route them, or cache data that has already been routed. There are three queues:
+1. Priority:
+- For locations that are experiencing flooding
+2. Base:
+- For all other locations
+3. Error: For all Locations that cause errors / trigger exceptions. 
+
+## How is this accessed
+
+The publisher calls the message broker internally, same with caching. 
+
+## What is the port?
+- To view the rabbit MQ portal, go to localhost:15672
+
+![alt text](photos/rabbit_mq.png)

Source/RnR/docs/hydrofabric-usage.md

@@ -0,0 +1,32 @@
+# Post Processing
+
+Post processing is performed on all t-route outputs to include required metadata. 
+
+## What metadata is added:
+
+Outside what is generated by T-Route, and stage, we are adding the following metadata
+
+```python
+ds = ds.assign_attrs(assimilated_rfc_point=assimilated_point)
+ds = ds.assign_attrs(
+    observed_flood_status=json_data["status"]["observed"]["floodCategory"]
+)
+ds = ds.assign_attrs(
+    forecasted_flood_status=json_data["status"]["forecast"]["floodCategory"]
+)
+ds = ds.assign_attrs(RFC_location_id=json_data["lid"])
+ds = ds.assign_attrs(upstream_RFC_location_id=json_data["upstream_lid"])
+ds = ds.assign_attrs(downstream_RFC_location_id=json_data["downstream_lid"])
+ds = ds.assign_attrs(RFC=json_data["rfc"]["abbreviation"])
+ds = ds.assign_attrs(WFO=json_data["wfo"]["abbreviation"])
+ds = ds.assign_attrs(USGS=json_data["usgs_id"])
+ds = ds.assign_attrs(county=json_data["county"])
+ds = ds.assign_attrs(state=json_data["state"]["abbreviation"])
+ds = ds.assign_attrs(Latitude=json_data["latitude"])
+ds = ds.assign_attrs(Longitude=json_data["longitude"])
+ds = ds.assign_attrs(Last_Forecast_Time=json_data["times"][stage_idx])
+```
+
+## How is this accessed
+
+The consumer app calls the post processing when running the ReplaceAndRoute service. 

Source/RnR/docs/iac.md

@@ -0,0 +1,24 @@
+# Publisher
+
+The data publisher section of the RnR event driven architecture is shown below. 
+
+![alt text](photos/data_publisher.png)
+
+## What is its job?
+
+This application is spun up by docker compose and is tasked with:
+1. Requesting RFC information from a DB
+2. Pulling Forecasts from the RFC points
+3. Formatting the forecasts in a json message body
+4. Posting the forecasts to the Rabbit MQ message queue to be processed by the consumer
+
+## How is this accessed
+
+The publisher is pinged by the following localhost endpoints:
+- /api/v1/publish/start
+    - Runs the publish endpoint for all RFC 
+- /api/v1/publish/{lid}
+    - Runs the publish endpoint for a specific RFC location ID
+
+## What is the port?
+- localhost:8000/docs

Source/RnR/docs/message_broker_cache.md

@@ -0,0 +1,52 @@
+# T-Route Usage 
+
+Our docker compose file uses a prebuilt docker image from T-Route to call T-Route as a service for river routing:
+
+```yaml
+troute:
+image: ghcr.io/taddyb33/t-route-dev:0.0.2
+ports:
+    - "8004:8000"
+volumes:
+    - type: bind
+    source: ./data/troute_output
+    target: /t-route/output
+    bind:
+        selinux: z
+    - type: bind
+    source: ./data
+    target: /t-route/data
+    bind:
+        selinux: z
+command: sh -c ". /t-route/.venv/bin/activate && uvicorn app.main:app --host 0.0.0.0 --port 8000"
+healthcheck:
+    test: curl --fail -I http://localhost:8000/health || exit 1
+    interval: 30s
+    timeout: 5s
+    retries: 3
+    start_period: 5s
+```
+
+This searches the `ghcr.io/taddyb33/` container registry for a t-route-dev image. 
+
+## Why an API?
+
+T-Route is used in many contexts for hydrological river routing:
+- NGEN 
+- Scientific Python 
+- Replace and Route (RnR)
+
+In the latest PR for RnR, there is a requirement to run T-Route as a service. This service requires an easy way to dynamically create config files, restart flow from Initial Conditions, and run T-Route. To satisfy this requirement, a FastAPI endpoint was created in `/src/app` along with code to dynamically create t-route endpoints. 
+
+## Why use shared volumes?
+
+Since T-Route is running in a docker container, there has to be a connection between the directories on your machine and the directories within the container. We're sharing the following folders by default:
+- `data/rfc_channel_forcings`
+  - For storing RnR RFC channel domain forcing files (T-Route inputs)
+- `data/rfc_geopackage_data`
+  - For storing HYFeatures gpkg files 
+  - Indexed by the NHD COMID, also called hf_id. Ex: 2930769 is the hf_id for the CAGM7 RFC forecast point. 
+- `data/troute_restart`
+  - For storing TRoute Restart files
+- `data/troute_output`
+  - For outputting results from the T-Route container

Source/RnR/docs/API_spec.png Source/RnR/docs/photos/API_spec.png

@@ -0,0 +1,19 @@
+# Mock DB:
+- This dir contains information from PI-2 to setup a mock database to read RFC information from. 
+- There is a missing `rnr_schema.dump` file that is missing as it is too large for Git
+
+To create the mock DB, you can run
+`docker build -t mock_db -f Dockerfile.mock_db . `
+
+or, you can reference the github container registry image similar to how compose does it:
+```yaml
+  mock_db:
+    image: ghcr.io/taddyb33/hydrovis/mock_database:0.0.1
+    environment:
+      - POSTGRES_PASSWORD=pass123
+      - POSTGRES_USER=postgres
+      - POSTGRES_DB=vizprocessing
+    ports:
+      - "5432:5432"
+
+```

Source/RnR/docs/photos/assimilation.png

@@ -176,7 +176,7 @@ A security group is created to allow SSH access and open the necessary ports for
 The EC2 instance is configured via the instance user_date with necessary software including Docker, Docker Compose and the AWS CLI. It will automatically clone the RnR application code, sync data from S3, and start the services defined in the Docker Compose configuration via a systemd service called rnr-app.
 
 Example of checking the status via systemd. The app(s) can be stopped and started similarly via systemctl stop and start.
-```sh
+```shell
 systemctl status rnr-app
 
 ● rnr-app.service - Docker Compose Application
@@ -196,4 +196,6 @@ Aug 29 20:38:29 ip-10-6-0-139.ngwpc.com docker[39123]:  Container rnr-app-1  Sta
 Aug 29 20:38:30 ip-10-6-0-139.ngwpc.com docker[39123]:  Container rnr-jupyterlab-1  Started
 Aug 29 20:38:30 ip-10-6-0-139.ngwpc.com docker[39123]:  Container rnr-consumer-1  Started
 Aug 29 20:38:30 ip-10-6-0-139.ngwpc.com docker[39123]:  Container rnr-app-1  Started
-```sh
+```
+
+*Note:* If you would like to see the files that are being generated from RnR, go to the `/app/hydrovis/Source/RnR/data/` directory on the EC2. `/app/hydrovis/Source/RnR` is the location for all RnR code delivered as IaC