Resolve Merge Conflicts With Upstream

.gitignore

@@ -6,4 +6,5 @@ cov.xml
 .venv
 cov_html
 htmlcov
-.pytest_cache
+.pytest_cache
+local_blob_storage

docker-compose-addons.yml

@@ -151,7 +151,8 @@ services:
       LOGGING_LEVEL: ${FIRMWARE_MANAGER_LOGGING_LEVEL}
     volumes:
       - ${GOOGLE_APPLICATION_CREDENTIALS}:/google/gcp_credentials.json
+      - ${HOST_BLOB_STORAGE_DIRECTORY}:/mnt/blob_storage
     logging:
       options:
         max-size: '10m'
-        max-file: '5'
+        max-file: '5'

resources/kubernetes/README.md

@@ -8,7 +8,7 @@ The webapp and API both utilize a K8s Ingress to handle external access to the a
 
 The YAML files use GCP specific specifications for various values such as "networking.gke.io/managed-certificates". These values will not work on AWS and Azure but there should be equivalent fields that these specifications can be updated to if needing to deploy in another cloud environment.
 
-The environment variables must be set according to the README documentation for each application. The iss-health-check application only supports GCP.
+The environment variables must be set according to the README documentation for each application. The iss-health-check application supports GCP or postgres for storing keys. The environment variables for the iss-health-check application must be set according to the README documentation for the iss-health-check application.
 
 ## Useful Links
 

resources/kubernetes/iss-health-check.yaml

@@ -1,4 +1,4 @@
-# This deployment is only usable in a GCP environment due to the GCP Secret Manager dependency
+# If GCP is being used to store keys, this deployment will only be usable in a GCP environment due to the GCP Secret Manager dependency
 apiVersion: 'apps/v1'
 kind: 'Deployment'
 metadata:
@@ -27,6 +27,8 @@ spec:
           ports:
             - containerPort: 8080
           env:
+            - name: STORAGE_TYPE
+              value: GCP
             - name: GOOGLE_APPLICATION_CREDENTIALS
               value: '/home/secret/cv_credentials.json'
             - name: PROJECT_ID
@@ -41,6 +43,8 @@ spec:
               value: ''
             - name: ISS_SCMS_VEHICLE_REST_ENDPOINT
               value: ''
+            - name: ISS_KEY_TABLE_NAME
+              value: ''
             - name: DB_USER
               value: ''
             - name: DB_PASS

resources/sql_scripts/CVManager_CreateTables.sql

@@ -325,6 +325,21 @@ SELECT ro.rsu_id, org.name
 FROM public.rsu_organization AS ro
 JOIN public.organizations AS org ON ro.organization_id = org.organization_id;
 
+-- Create iss keys table (id, iss_key, creation_date, expiration_date)
+CREATE SEQUENCE public.iss_keys_iss_key_id_seq
+   INCREMENT 1
+   START 1
+   MINVALUE 1
+   MAXVALUE 2147483647
+   CACHE 1;
+
+CREATE TABLE IF NOT EXISTS public.iss_keys
+(
+   iss_key_id integer NOT NULL DEFAULT nextval('iss_keys_iss_key_id_seq'::regclass),
+   common_name character varying(128) COLLATE pg_catalog.default NOT NULL,
+   token character varying(128) COLLATE pg_catalog.default NOT NULL
+);
+
 -- Create scms_health table
 CREATE SEQUENCE public.scms_health_scms_health_id_seq
    INCREMENT 1

sample.env

@@ -130,9 +130,11 @@ KAFKA_BIGQUERY_TABLENAME = ''
 # ---------------------------------------------------------------------
 
 # Firmware Manager Addon:
-BLOB_STORAGE_PROVIDER=GCP
+BLOB_STORAGE_PROVIDER=DOCKER
 BLOB_STORAGE_BUCKET=
 GCP_PROJECT=
+## Docker volume mount point for BLOB storage (if using Docker)
+HOST_BLOB_STORAGE_DIRECTORY=./local_blob_storage
 # ---------------------------------------------------------------------
 
 # Geo-spatial message query Addon:

services/addons/images/firmware_manager/README.md

@@ -16,7 +16,7 @@ This directory contains a microservice that runs within the CV Manager GKE Clust
 
 An RSU is determined to be ready for upgrade if its entry in the "rsus" table in PostgreSQL has its "target_firmware_version" set to be different than its "firmware_version". The Firmware Manager will ignore all devices with incompatible firmware upgrades set as their target firmware based on the "firmware_upgrade_rules" table. The CV Manager API will only offer CV Manager webapp users compatible options so this generally is a precaution.
 
-Hosting firmware files is recommended to be done via the cloud. GCP cloud storage is the currently supported method. Alternatives can be added via the [download_blob.py](download_blob.py) script. Firmware storage must be organized by: `vendor/rsu-model/firmware-version/install_package`.
+Hosting firmware files is recommended to be done via the cloud. GCP cloud storage is the currently supported method, but a directory mounted as a docker volume can also be used. Alternative cloud support can be added via the [download_blob.py](download_blob.py) script. Firmware storage must be organized by: `vendor/rsu-model/firmware-version/install_package`.
 
 Firmware upgrades have unique procedures based on RSU vendor/manufacturer. To avoid requiring a unique bash script for every single firmware upgrade, the Firmware Manager has been written to use vendor based upgrade scripts that have been thoroughly tested. An interface-like abstract class, [base_upgrader.py](base_upgrader.py), has been made for helping create upgrade scripts for vendors not yet supported. The Firmware Manager selects the script to use based off the RSU's "model" column in the "rsus" table. These scripts report back to the Firmware Manager on completion with a status of whether the upgrade was a success or failure. Regardless, the Firmware Manager will remove the process from its tracking and update the PostgreSQL database accordingly.
 
@@ -40,7 +40,7 @@ Available REST endpoints:
 
 To properly run the firmware_manager microservice the following services are also required:
 
-- Cloud based blob storage
+- Blob storage (cloud-based or otherwise)
   - Firmware storage must be organized by: `vendor/rsu-model/firmware-version/install_package`.
 - CV Manager PostgreSQL database with data in the "rsus", "rsu_models", "manufacturers", "firmware_images", and "firmware_upgrade_rules" tables
 - Network connectivity from the environment the firmware_manager is deployed into to the blob storage and the RSUs
@@ -70,6 +70,9 @@ GCP Required environment variables:
 - GCP_PROJECT - GCP project for the firmware cloud storage bucket
 - GOOGLE_APPLICATION_CREDENTIALS - Service account location. Recommended to attach as a volume.
 
+Docker volume required environment variables:
+- HOST_BLOB_STORAGE_DIRECTORY - Directory mounted as a docker volume for firmware storage. A relative path can be specified here.
+
 ## Vendor Specific Requirements
 
 ### Commsignia

services/addons/images/firmware_manager/download_blob.py

@@ -3,17 +3,67 @@
 import os
 
 
-# Only supports GCP Bucket Storage for downloading blobs
 def download_gcp_blob(blob_name, destination_file_name):
+    """Download a file from a GCP Bucket Storage bucket to a local file.
+
+    Args:
+        blob_name (str): The name of the file in the bucket.
+        destination_file_name (str): The name of the local file to download the bucket file to.
+    """
+
+    if not validate_file_type(blob_name):
+        return False
+
     gcp_project = os.environ.get("GCP_PROJECT")
     bucket_name = os.environ.get("BLOB_STORAGE_BUCKET")
     storage_client = storage.Client(gcp_project)
     bucket = storage_client.get_bucket(bucket_name)
     blob = bucket.blob(blob_name)
+
     if blob.exists():
         blob.download_to_filename(destination_file_name)
         logging.info(
             f"Downloaded storage object {blob_name} from bucket {bucket_name} to local file {destination_file_name}."
         )
         return True
     return False
+
+
+def download_docker_blob(blob_name, destination_file_name):
+    """Copy a file from a directory mounted as a volume in a Docker container to a local file.
+
+    Args:
+        blob_name (str): The name of the file in the directory.
+        destination_file_name (str): The name of the local file to copy the directory file to.
+    """
+
+    if not validate_file_type(blob_name):
+        return False
+
+    directory = "/mnt/blob_storage"
+    source_file_name = f"{directory}/{blob_name}"
+    os.system(f"cp {source_file_name} {destination_file_name}")
+    logging.info(
+        f"Copied storage object {blob_name} from directory {directory} to local file {destination_file_name}."
+    )
+    return True
+
+
+def validate_file_type(file_name):
+    """Validate the file type of the file to be downloaded.
+
+    Args:
+        file_name (str): The name of the file to be downloaded.
+    """
+    if not file_name.endswith(".tar"):
+        logging.error(
+            f"Unsupported file type for storage object {file_name}. Only .tar files are supported."
+        )
+        return False
+    return True
+
+
+class UnsupportedFileTypeException(Exception):
+    def __init__(self, message="Unsupported file type. Only .tar files are supported."):
+        self.message = message
+        super().__init__(self.message)

services/addons/images/firmware_manager/sample.env

@@ -7,9 +7,11 @@ PG_DB_NAME=""
 PG_DB_USER=""
 PG_DB_PASS=""
 
-# Blob storage variables
-BLOB_STORAGE_PROVIDER="GCP"
-BLOB_STORAGE_BUCKET=""
+# Blob storage variables (only 'GCP' and 'DOCKER' are supported at this time)
+BLOB_STORAGE_PROVIDER=DOCKER
+BLOB_STORAGE_BUCKET=
+## Docker volume mount point for BLOB storage (if using DOCKER)
+HOST_BLOB_STORAGE_DIRECTORY=./local_blob_storage
 
 # For users using GCP cloud storage
 GCP_PROJECT=""

services/addons/images/firmware_manager/upgrader.py

@@ -34,15 +34,22 @@ def download_blob(self, blob_name=None, local_file_name=None):
         # Create parent rsu_ip directory
         path = self.local_file_name[: self.local_file_name.rfind("/")]
         Path(path).mkdir(exist_ok=True)
+        blob_name = self.blob_name if blob_name is None else blob_name
+        local_file_name = (
+            self.local_file_name if local_file_name is None else local_file_name
+        )
 
         # Download blob, defaults to GCP blob storage
-        bsp = os.environ.get("BLOB_STORAGE_PROVIDER", "GCP")
-        if bsp == "GCP":
-            blob_name = self.blob_name if blob_name is None else blob_name
-            local_file_name = self.local_file_name if local_file_name is None else local_file_name
+        bspCaseInsensitive = os.environ.get(
+            "BLOB_STORAGE_PROVIDER", "DOCKER"
+        ).casefold()
+        if bspCaseInsensitive == "gcp":
             return download_blob.download_gcp_blob(blob_name, local_file_name)
+        elif bspCaseInsensitive == "docker":
+            return download_blob.download_docker_blob(blob_name, local_file_name)
         else:
             logging.error("Unsupported blob storage provider")
+            raise StorageProviderNotSupportedException
 
     # Notifies the firmware manager of the completion status for the upgrade
     # success is a boolean
@@ -72,7 +79,7 @@ def wait_until_online(self):
             iter += 1
         # 3 minutes pass with no response
         return -1
-    
+
     def check_online(self):
         iter = 0
         # Ping once every second for 5 seconds to verify RSU is online
@@ -86,12 +93,16 @@ def check_online(self):
             time.sleep(1)
         # 5 seconds pass with no response
         return False
-    
+
     def send_error_email(self, type="Firmware Upgrader", err=""):
         try:
             email_addresses = os.environ.get("FW_EMAIL_RECIPIENTS").split(",")
 
-            subject = f"{self.rsu_ip} Firmware Upgrader Failure" if type == "Firmware Upgrader" else f"{self.rsu_ip} Firmware Upgrader Post Upgrade Script Failure"
+            subject = (
+                f"{self.rsu_ip} Firmware Upgrader Failure"
+                if type == "Firmware Upgrader"
+                else f"{self.rsu_ip} Firmware Upgrader Post Upgrade Script Failure"
+            )
 
             for email_address in email_addresses:
                 emailSender = EmailSender(
@@ -115,3 +126,8 @@ def send_error_email(self, type="Firmware Upgrader", err=""):
     @abc.abstractclassmethod
     def upgrade(self):
         pass
+
+
+class StorageProviderNotSupportedException(Exception):
+    def __init__(self):
+        super().__init__("Unsupported blob storage provider")

services/addons/images/iss_health_check/README.md

@@ -11,23 +11,25 @@
 
 This directory contains a microservice that runs within the CV Manager GKE Cluster. The iss_health_checker application populates the CV Manager PostGreSQL database's 'scms_health' table with the current ISS SCMS statuses of all RSUs recorded in the 'rsus' table. These statuses are queried by this application from a provided ISS Green Hills SCMS API endpoint.
 
-The application schedules the iss_health_checker script to run every 6 hours. A new SCMS API access key is generated every run of the script to ensure the access never expires. This is due to a limitation of the SCMS API not allowing permanent access keys. Access keys are stored in GCP Secret Manager to allow for versioning and encrypted storage. The application removes the previous access key from the SCMS API after runtime to reduce clutter of access keys on the API service account.
+The application schedules the iss_health_checker script to run every 6 hours. A new SCMS API access key is generated every run of the script to ensure the access never expires. This is due to a limitation of the SCMS API not allowing permanent access keys. Access keys can be stored in GCP Secret Manager to allow for versioning and encrypted storage. The application removes the previous access key from the SCMS API after runtime to reduce clutter of access keys on the API service account.
 
-Currently only GCP is supported to run this application due to a reliance on the GCP Secret Manager. Storing the access keys on a local volume is not recommended due to security vulnerabilities. Feel free to contribute to this application for secret manager equivalent support for other cloud environments.
+Currently GCP & Postgres are the only supported storage solutions to run this application. Storing the access keys on a local volume is not recommended due to security vulnerabilities. Feel free to contribute to this application to support other storage solutions.
 
 ## Requirements <a name = "requirements"></a>
 
 To properly run the iss_health_checker microservice the following services are also required:
 
-- GCP project and service account with GCP Secret Manager access
+- GCP project and service account with GCP Secret Manager access (only required if STORAGE_TYPE is set to 'gcp')
 - CV Manager PostgreSQL database with at least one RSU inserted into the 'rsus' table
 - Service agreement with ISS Green Hills to have access to the SCMS API REST service endpoint
 - iss_health_checker must be deployed in the same environment or K8s cluster as the PostgreSQL database
 - iss_health_checker deployment must have access to the internet or at least the SCMS API endpoint
 
 The iss_health_checker microservice expects the following environment variables to be set:
 
-- GOOGLE_APPLICATION_CREDENTIALS - file location for GCP JSON service account key.
+- STORAGE_TYPE - Storage solution for the SCMS API access keys. Currently only 'gcp' & 'postgres' are supported.
+- GOOGLE_APPLICATION_CREDENTIALS - File location for GCP JSON service account key. Only required if STORAGE_TYPE is set to 'gcp'.
+- ISS_KEY_TABLE_NAME - Postgres table name for the ISS SCMS API access keys. Only required if STORAGE_TYPE is set to 'postgres'.
 - PROJECT_ID - GCP project ID.
 - ISS_API_KEY - Initial ISS SCMS API access key to perform the first run of the script. This access key must not expire before the first runtime.
 - ISS_API_KEY_NAME - Human readable reference for the access key within ISS SCMS API. Generated access keys will utilize this same name.