Sync updates from upstream, add permissions boundary to created roles…

… if needed (#8)

* Point README to veda-docs (NASA-IMPACT#171)

* Upgrade titiler and titiler-pgstac

* Fix br/gzip header test

* Remove factory (it's included in new titiler-pgstac version)
NASA-IMPACT#148 (comment)

* Remove pin on cramjam

* Load test data in container (actions not updated)

* fix actions tests

* adjust container strategy

* actions connections fix

* remove -it flag (it broke actions)

* re-enable lint, propagate test changes to other actions

* docker-compose -> docker compose

* Fix template import

* include private subnet config

* fix subnet type

* format changes

* pub accessible default true

* remove publicly accessible variable

* Add sql connection and execution abstraction

* Add sql logic to fix projection extension types

* add script to delete null stac_extensions

* Lint and move sql command scripts to support_scripts dir

* Use pythonic naming conventions

* pgstac readme change

* docker compose pgstac version

* pre-deploy action fix

* RDS Proxy initial implementation

* proxy secret + urlllib fix

* use pgstac 0.7.9

* Update set environment

* Add support for permissions boundary

* use titiler custom JSONResponse to handle NaN values

---------

Co-authored-by: Julia Signell <jsignell@gmail.com>
Co-authored-by: ividito <isayah@developmentseed.org>
Co-authored-by: smohiudd <saadiq@developmentseed.org>
Co-authored-by: Saadiq Mohiuddin <34844565+smohiudd@users.noreply.github.com>
Co-authored-by: Nathan Zimmerman <npzimmerman@gmail.com>
Co-authored-by: Caden Helbling <caden.helbling@gmail.com>
Co-authored-by: vincentsarago <vincent.sarago@gmail.com>
Co-authored-by: Alexandra Kirk <alexandra@developmentseed.org>

.example.env

@@ -4,7 +4,7 @@ CDK_DEFAULT_REGION=[REQUIRED IF DEPLOYING TO EXISTING VPC]
 
 STAGE=[FILL ME IN]
 
-VEDA_DB_PGSTAC_VERSION=0.7.6
+VEDA_DB_PGSTAC_VERSION=0.7.9
 VEDA_DB_SCHEMA_VERSION=0.1.0
 VEDA_DB_SNAPSHOT_ID=[OPTIONAL BUT **REQUIRED** FOR ALL DEPLOYMENTS AFTER BASING DEPLOYMENT ON SNAPSHOT]
 
@@ -18,6 +18,7 @@ VEDA_DOMAIN_ALT_HOSTED_ZONE_NAME=[OPTIONAL SECOND DOMAIN]
 VEDA_RASTER_ENABLE_MOSAIC_SEARCH=TRUE
 VEDA_RASTER_DATA_ACCESS_ROLE_ARN=[OPTIONAL ARN OF IAM ROLE TO BE ASSUMED BY RASTER API] 
 
-VEDA_RASTER_PATH_PREFIX=[OPTIONAL PATH PREFIX TO ADD TO TITILER ENDPOINTS]
+VEDA_DB_PUBLICLY_ACCESSIBLE=TRUE
 
+VEDA_RASTER_PATH_PREFIX=[OPTIONAL PATH PREFIX TO ADD TO TITILER ENDPOINTS]
 VEDA_STAC_PATH_PREFIX=[OPTIONAL PATH PREFIX TO ADD TO TITILER ENDPOINTS]

.github/workflows/pr.yml

@@ -60,7 +60,7 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: '3.9'
-    
+
       - uses: actions/cache@v3
         with:
           path: ${{ env.pythonLocation }}
@@ -69,23 +69,24 @@ jobs:
       - name: Install python dependencies
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -e .[dev,deploy,test]
+          python -m pip install -e .[dev,deploy,test] 
+
+      - name: Launch services
+        run: AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY=${{secrets.AWS_SECRET_ACCESS_KEY}} docker compose up --build -d
 
       - name: Ingest Stac Items/Collection
         run: |
-          pypgstac pgready --dsn postgresql://username:password@0.0.0.0:5432/postgis
-          pypgstac load collections .github/workflows/data/noaa-emergency-response.json --dsn postgresql://username:password@0.0.0.0:5432/postgis --method insert
-          pypgstac load items .github/workflows/data/noaa-eri-nashville2020.json --dsn postgresql://username:password@0.0.0.0:5432/postgis --method insert
+          ./scripts/load-data-container.sh
 
       - name: Sleep for 10 seconds
         run: sleep 10s
         shell: bash
-
+  
       - name: Integrations tests
         run: python -m pytest .github/workflows/tests/ -vv -s
 
       - name: Stop services
-        run: docker-compose stop
+        run: docker compose stop
 
   predeploy:
     needs: [test]

.github/workflows/tests/test_raster.py

@@ -13,7 +13,7 @@ def test_raster_api():
     )
     assert resp.status_code == 200
     assert resp.headers["content-type"] == "application/json"
-    assert resp.headers["content-encoding"] == "br"
+    assert resp.headers["content-encoding"] in ["br", "gzip"]
 
 
 def test_mosaic_api():

README.md

@@ -40,7 +40,7 @@ To retrieve the variables for a stage that has been previously deployed, the sec
 | --- | --- |
 | `APP_NAME` | Optional app name used to name stack and resources, defaults to `veda-backend` |
 | `STAGE` | **REQUIRED** Deployment stage used to name stack and resources, i.e. `dev`, `staging`, `prod` |
-| `VEDA_DB_PGSTAC_VERSION` | **REQUIRED** version of PgStac database, i.e. 0.5 |
+| `VEDA_DB_PGSTAC_VERSION` | **REQUIRED** version of PgStac database, i.e. 0.7.9 |
 | `VEDA_DB_SCHEMA_VERSION` | **REQUIRED** The version of the custom veda-backend schema, i.e. 0.1.1 |
 | `VEDA_DB_SNAPSHOT_ID` | **Once used always REQUIRED** Optional RDS snapshot identifier to initialize RDS from a snapshot |
 > **Note** See [Advanced Configuration](docs/advanced_configuration.md) for details about custom configuration options.
@@ -114,11 +114,11 @@ Support scripts are provided for manual system operations.
 | [**veda-ui**](https://github.com/NASA-IMPACT/veda-ui) | Dashboard UI for viewing and analysing VEDA assets |
 | [**veda-stac-ingestor**](https://github.com/NASA-IMPACT/veda-stac-ingestor) |  Entry-point for users/services to add new records to database |
 | [**veda-data-pipelines**](https://github.com/NASA-IMPACT/veda-data-pipelines) | Cloud optimize data assets and submit records for publication to veda-stac-ingestor |
-| [**veda-documentation**](https://github.com/NASA-IMPACT/veda-documentation) | Documentation repository for end users of VEDA ecosystem data and tools |
+| [**veda-docs**](https://github.com/NASA-IMPACT/veda-docs) | Documentation repository for end users of VEDA ecosystem data and tools |
 
 ## VEDA usage examples
 
-### [VEDA documentation](https://nasa-impact.github.io/veda-documentation/)
+### [VEDA documentation](https://nasa-impact.github.io/veda-docs)
 
 ### [VEDA dashboard](https://www.earthdata.nasa.gov/dashboard)
 
@@ -128,4 +128,4 @@ Support scripts are provided for manual system operations.
 Radiant Earth's [stac-browser](https://github.com/radiantearth/stac-browser) is a browser for STAC catalogs. The demo version of this browser [radiantearth.github.io/stac-browser](https://radiantearth.github.io/stac-browser/#/) can be used to browse the contents of the veda-backend STAC catalog, paste the veda-backend stac-api URL deployed by this project in the demo and click load. Read more about the recent developments and usage of stac-browser [here](https://medium.com/radiant-earth-insights/the-exciting-future-of-the-stac-browser-2351143aa24b).
 
 # License
-This project is licensed under **Apache 2**, see the [LICENSE](LICENSE) file for more details.
+This project is licensed under **Apache 2**, see the [LICENSE](LICENSE) file for more details.

app.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """ CDK Configuration for the veda-backend stack."""
 
-from aws_cdk import App, Stack, Tags, aws_iam
+from aws_cdk import App, Aspects, Stack, Tags, aws_iam
 from constructs import Construct
 
 from config import veda_app_settings
@@ -22,13 +22,17 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
         super().__init__(scope, construct_id, **kwargs)
 
         if veda_app_settings.permissions_boundary_policy_name:
-            permission_boundary_policy = aws_iam.Policy.from_policy_name(
+            permission_boundary_policy = aws_iam.ManagedPolicy.from_managed_policy_name(
                 self,
                 "permission-boundary",
                 veda_app_settings.permissions_boundary_policy_name,
             )
             aws_iam.PermissionsBoundary.of(self).apply(permission_boundary_policy)
 
+            from permission_boundary import PermissionBoundaryAspect
+
+            Aspects.of(self).add(PermissionBoundaryAspect(permission_boundary_policy))
+
 
 veda_stack = VedaStack(
     app,
@@ -71,7 +75,6 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
 
 # TODO this conditional supports deploying a second set of APIs to a separate custom domain and should be removed if no longer necessary
 if veda_app_settings.alt_domain():
-
     alt_domain = DomainConstruct(
         veda_stack,
         "alt-domain",

database/infrastructure/config.py

@@ -34,11 +34,9 @@ class vedaDBSettings(BaseSettings):
             "**Once used always REQUIRED**"
         ),
     )
-    private_subnets: Optional[bool] = Field(
-        False,
-        description="Boolean deploy database to private subnets",
+    publicly_accessible: Optional[bool] = Field(
+        True, description="Boolean if the RDS should be publicly accessible"
     )
-
     # RDS custom postgres parameters
     max_locks_per_transaction: Optional[str] = Field(
         "1024",
@@ -55,6 +53,10 @@ class vedaDBSettings(BaseSettings):
         description="maximum number of temporary buffers used by each session",
         regex=r"^[1-9]\d*$",
     )
+    use_rds_proxy: Optional[bool] = Field(
+        False,
+        description="Boolean if the RDS should be accessed through a proxy",
+    )
 
     class Config:
         """model config."""

database/infrastructure/construct.py

@@ -10,6 +10,7 @@
     RemovalPolicy,
     Stack,
     aws_ec2,
+    aws_iam,
     aws_lambda,
     aws_logs,
     aws_rds,
@@ -35,6 +36,7 @@ def __init__(
         new_username: str,
         secrets_prefix: str,
         stage: str,
+        host: str,
     ) -> None:
         """."""
         super().__init__(scope, construct_id)
@@ -69,7 +71,7 @@ def __init__(
                         "dbname": new_dbname,
                         "engine": "postgres",
                         "port": 5432,
-                        "host": database.instance_endpoint.hostname,
+                        "host": host,
                         "username": new_username,
                     }
                 ),
@@ -129,10 +131,9 @@ def __init__(
         stack_name = Stack.of(self).stack_name
 
         # Configure accessibility
-        publicly_accessible = False if veda_db_settings.private_subnets else True
         subnet_type = (
             aws_ec2.SubnetType.PRIVATE_ISOLATED
-            if veda_db_settings.private_subnets is True
+            if veda_db_settings.publicly_accessible is False
             else aws_ec2.SubnetType.PUBLIC
         )
 
@@ -171,7 +172,7 @@ def __init__(
                 vpc_subnets=aws_ec2.SubnetSelection(subnet_type=subnet_type),
                 deletion_protection=True,
                 removal_policy=RemovalPolicy.RETAIN,
-                publicly_accessible=publicly_accessible,
+                publicly_accessible=veda_db_settings.publicly_accessible,
                 credentials=credentials,
                 parameter_group=parameter_group,
             )
@@ -190,10 +191,12 @@ def __init__(
                 vpc_subnets=aws_ec2.SubnetSelection(subnet_type=subnet_type),
                 deletion_protection=True,
                 removal_policy=RemovalPolicy.RETAIN,
-                publicly_accessible=publicly_accessible,
+                publicly_accessible=veda_db_settings.publicly_accessible,
                 parameter_group=parameter_group,
             )
 
+        hostname = database.instance_endpoint.hostname
+
         # Use custom resource to bootstrap PgSTAC database
         self.pgstac = BootstrapPgStac(
             self,
@@ -203,8 +206,35 @@ def __init__(
             new_username=veda_db_settings.user,
             secrets_prefix=stack_name,
             stage=stage,
+            host=hostname,
         )
 
+        self.proxy = None
+        if veda_db_settings.use_rds_proxy:
+            # secret for non-admin user
+            proxy_secret = self.pgstac.secret
+
+            proxy_role = aws_iam.Role(
+                self,
+                "RDSProxyRole",
+                assumed_by=aws_iam.ServicePrincipal("rds.amazonaws.com"),
+            )
+            self.proxy = aws_rds.DatabaseProxy(
+                self,
+                proxy_target=aws_rds.ProxyTarget.from_instance(database),
+                id="RdsProxy",
+                vpc=vpc,
+                secrets=[database.secret, proxy_secret],
+                db_proxy_name=f"veda-backend-{stage}-proxy",
+                role=proxy_role,
+                require_tls=False,
+                debug_logging=False,
+            )
+
+            # Allow connections to the proxy from the same security groups as the DB
+            for sg in database.connections.security_groups:
+                self.proxy.connections.add_security_group(sg)
+
         CfnOutput(
             self,
             "pgstac-secret-name",

docker-compose.yml

@@ -95,7 +95,7 @@ services:
   database:
     container_name: veda.db
     platform: linux/amd64
-    image: ghcr.io/stac-utils/pgstac:v0.7.1
+    image: ghcr.io/stac-utils/pgstac:v0.7.9
     environment:
       - POSTGRES_USER=username
       - POSTGRES_PASSWORD=password
@@ -105,9 +105,11 @@ services:
       - PGDATABASE=postgis
     ports:
       - "5432:5432"
-    command: postgres -N 500
+    command: "postgres -N 500"
     volumes:
       - ./.pgdata:/var/lib/postgresql/data
+      - ./scripts:/tmp/scripts
+      - ./.github/workflows/data:/tmp/data
 
 networks:
   default:

network/infrastructure/construct.py

@@ -48,7 +48,7 @@ def __init__(
             )
             private_subnet = aws_ec2.SubnetConfiguration(
                 name="private",
-                subnet_type=aws_ec2.SubnetType.PRIVATE_WITH_NAT,
+                subnet_type=aws_ec2.SubnetType.PRIVATE_ISOLATED,
                 cidr_mask=veda_vpc_settings.private_mask,
             )
 
@@ -68,7 +68,7 @@ def __init__(
                 "dynamodb": aws_ec2.GatewayVpcEndpointAwsService.DYNAMODB,
             }
 
-            for (id, service) in vpc_endpoints.items():
+            for id, service in vpc_endpoints.items():
                 if isinstance(service, aws_ec2.InterfaceVpcEndpointAwsService):
                     self.vpc.add_interface_endpoint(id, service=service)
                 elif isinstance(service, aws_ec2.GatewayVpcEndpointAwsService):

permission_boundary.py

@@ -0,0 +1,56 @@
+"""Class that applies permissions boundary to all the roles created within a Stack"""
+from typing import Union
+
+import jsii
+from aws_cdk import IAspect, aws_iam
+from constructs import IConstruct
+from jsii._reference_map import _refs
+from jsii._utils import Singleton
+
+
+@jsii.implements(IAspect)
+class PermissionBoundaryAspect:
+    """
+    This aspect finds all aws_iam.Role objects in a node (ie. CDK stack) and sets permission boundary to the given ARN.
+    """
+
+    def __init__(self, permission_boundary: Union[aws_iam.ManagedPolicy, str]) -> None:
+        """
+        :param permission_boundary: Either aws_iam.ManagedPolicy object or managed policy's ARN string
+        """
+        self.permission_boundary = permission_boundary
+
+    def visit(self, construct_ref: IConstruct) -> None:
+        """
+        construct_ref only contains a string reference to an object. To get the actual object, we need to resolve it using JSII mapping.
+        :param construct_ref: ObjRef object with string reference to the actual object.
+        :return: None
+        """
+        if isinstance(construct_ref, jsii._kernel.ObjRef) and hasattr(
+            construct_ref, "ref"
+        ):
+            kernel = Singleton._instances[
+                jsii._kernel.Kernel
+            ]  # The same object is available as: jsii.kernel
+            resolve = _refs.resolve(kernel, construct_ref)
+        else:
+            resolve = construct_ref
+
+        def _walk(obj):
+            if isinstance(obj, aws_iam.Role):
+                cfn_role = obj.node.find_child("Resource")
+                policy_arn = (
+                    self.permission_boundary
+                    if isinstance(self.permission_boundary, str)
+                    else self.permission_boundary.managed_policy_arn
+                )
+                cfn_role.add_property_override("PermissionsBoundary", policy_arn)
+            else:
+                if hasattr(obj, "permissions_node"):
+                    for c in obj.permissions_node.children:
+                        _walk(c)
+                if hasattr(obj, "node") and obj.node.children:
+                    for c in obj.node.children:
+                        _walk(c)
+
+        _walk(resolve)

raster_api/runtime/setup.py

@@ -6,10 +6,8 @@
     long_description = f.read()
 
 inst_reqs = [
-    "titiler.pgstac==0.1.0.a9",
-    "titiler.application>=0.5,<0.6",
-    "starlette-cramjam>=0.1.0,<0.2",
-    "fastapi==0.93.0",
+    "titiler.pgstac==0.2.3",
+    "titiler.application>=0.10,<0.11",
     "importlib_resources>=1.1.0;python_version<='3.9'",  # https://github.com/cogeotiff/rio-tiler/pull/379
     "aws_xray_sdk>=2.6.0,<3",
     "aws-lambda-powertools>=1.18.0",
@@ -29,7 +27,7 @@
     description="",
     python_requires=">=3.7",
     packages=find_namespace_packages(exclude=["tests*"]),
-    package_data={"veda": ["raster/templates/*.html"]},
+    package_data={"src": ["templates/*.html"]},
     include_package_data=True,
     zip_safe=False,
     install_requires=inst_reqs,