Merge branch 'bird-house:master' into master

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.22.11
+current_version = 1.23.1
 commit = True
 tag = False
 tag_name = {new_version}
@@ -30,11 +30,11 @@ search = {current_version}
 replace = {new_version}
 
 [bumpversion:file:RELEASE.txt]
-search = {current_version} 2023-02-03T04:58:52Z
+search = {current_version} 2023-02-13T18:31:02Z
 replace = {new_version} {utcnow:%Y-%m-%dT%H:%M:%SZ}
 
 [bumpversion:part:releaseTime]
-values = 2023-02-03T04:58:52Z
+values = 2023-02-13T18:31:02Z
 
 [bumpversion:file(version):birdhouse/config/canarie-api/docker_configuration.py.template]
 search = 'version': '{current_version}'

CHANGES.md

@@ -16,6 +16,110 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[1.23.1](https://github.com/bird-house/birdhouse-deploy/tree/1.23.1) (2023-02-13)
+------------------------------------------------------------------------------------------------------------------
+
+[//]: # (list changes here, using '-' for each new entry, remove this when items are added)
+
+## Fixes
+- Vars in `DELAYED_EVAL` list are not expanded properly outside of `pavics-compose.sh`
+
+  There are other scripts sourcing `default.env` and `env.local` and all those
+  scripts have to expand the vars in `DELAYED_EVAL` list to have their actual
+  values.
+
+  Only scripts using the 3 variables in `DELAYED_EVAL` list are broken.
+
+  `DELAYED_EVAL` was previously introduced in PR https://github.com/bird-house/birdhouse-deploy/pull/272.
+
+  **Sample errors**
+
+  `fix-geoserver-data-dir-perm` (called at the end of `pavics-compose.sh`):
+  ```
+  fix GeoServer data dir permission on first run only, when data dir do not exist yet.
+  + DATA_DIR='${DATA_PERSIST_ROOT}/geoserver'
+  + '[' -n  ]
+  + docker run --rm --name fix-geoserver-data-dir-perm --volume '${DATA_PERSIST_ROOT}/geoserver:/datadir' --env FIRST_RUN_ONLY bash:5.1.4 bash -xc 'if [ -z "$FIRST_RUN_ONLY" -o ! -f /datadir/global.xml ]; \
+      then chown -R 1000:10001 /datadir; else echo "No execute."; fi'
+  docker: Error response from daemon: create ${DATA_PERSIST_ROOT}/geoserver: "${DATA_PERSIST_ROOT}/geoserver" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
+  ```
+
+  `trigger-deploy-notebook` (broke notebook deploy job):
+  ```
+  + TMP_SCRIPT=/tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook
+  + cat
+  + chmod a+x /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook
+  + docker run --rm --name deploy_tutorial_notebooks -u root -v /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/deploy-notebook:/deploy-notebook:ro -v /tmp/notebookdeploy/notebookdeploy.XXXXXXIfafFK/tutorial-notebooks:/tutorial-notebooks:ro -v '${DATA_PERSIST_ROOT}/jupyterhub_user_data:/notebook_dir:rw' --entrypoint /deploy-notebook bash:5.1.4
+  docker: Error response from daemon: create ${DATA_PERSIST_ROOT}/jupyterhub_user_data: "${DATA_PERSIST_ROOT}/jupyterhub_user_data" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
+  ```
+
+  **Explanation of the fix**
+
+  All scripts have to remember to call function `process_delayed_eval` in order
+  to obtain the real value of each vars in `DELAYED_EVAL` list.
+
+  Centralized all logic about reading configs (config files reading order,
+  remember to call `process_delayed_eval`) to avoid mistake and to ease updating
+  logic in the future.  Too many scripts were reading the configs  themselves and
+  some are not doing it properly, ex: forgot to hide password when reading
+  `env.local`.
+
+  **All scripts should do this going forward**
+
+  ```sh
+  # Set variable COMPOSE_DIR to the dir containing pavics-compose.sh and docker-compose.yml.
+
+  # Source the script providing function read_configs.
+  # read_configs uses COMPOSE_DIR to find default.env and env.local.
+  . $COMPOSE_DIR/read-configs.include.sh
+
+  # Call function read_configs to read the various config files in the appropriate order and process delayed eval vars properly.
+  read_configs
+  ```
+
+[1.23.0](https://github.com/bird-house/birdhouse-deploy/tree/1.23.0) (2023-02-10)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes:
+
+- secure-data-proxy: add new [`secure-data-proxy`][secure-data-proxy] optional component.
+
+  When enabled, this component will enforce authentication and authorization to be resolved against the `/wpsoutputs`
+  endpoint prior to accessing the results produced by WPS executions. A Magpie service named `secure-data-proxy` is
+  created to define the resource and permission hierarchy of directories and files the users and groups can access.
+  When disabled, the original behavior to provide open access to `/wpsoutputs` is employed.
+
+  A variable named `SECURE_DATA_PROXY_AUTH_INCLUDE` is dynamically assigned based on the activation or not of this
+  component. Corresponding validation of optional/mandatory/delayed-eval variables used by this component are also
+  applied dynamically, as well as mounting the necessary `nginx` and `docker-compose` extended configurations.
+
+- Weaver: adjust user-context output directory hooks and permissions for [`secure-data-proxy`][secure-data-proxy].
+
+  When a process defined in Weaver (either a WPS provider or a local definition) is executed by a user that was granted
+  authorization to run a job, the corresponding user-context directory under `/wpsoutputs/users/{user-id}` will be used
+  for storing the execution outputs and will have the appropriate permissions set for that user to grant them access to
+  those outputs.
+
+## Fixes:
+
+- Magpie/Twitcher: update minimum version `magpie>=3.31.0` to employ `twitcher>=0.8.0` in `MapgieAdatepr`.
+
+  - Resolve an issue where `response.request` references were not set in OWS proxy responses when handled by Twitcher.
+    This caused `MapgieAdatepr` response hooks to fail, which in turn caused failing requests for any non-WPS
+    service that defined any proxy request hook, such as in the case of [`weaver`][weaver-component] component.
+
+  - Adds the Twitcher ``/ows/verify/{service_name}[/{extra_path}`` endpoint employed for validating authorized access
+    to Magpie service/resources, in the same fashion as the protected proxy endpoint, but without performing the proxied
+    request toward the target service. This is mandatory for using the new [`secure-data-proxy`][secure-data-proxy] 
+    optional component, otherwise the proxy endpoint triggers data download twice, once for authorization and another
+    for actually accessing the data.
+
+  See also [Ouranosinc/Magpie#571](https://github.com/Ouranosinc/Magpie/pull/571)
+  and [bird-house/twitcher#118](https://github.com/bird-house/twitcher/pull/118).
+
+[secure-data-proxy]: birdhouse/optional-components/secure-data-proxy
+[weaver-component]: birdhouse/components/weaver
+
 [1.22.11](https://github.com/bird-house/birdhouse-deploy/tree/1.22.11) (2023-02-03)
 ------------------------------------------------------------------------------------------------------------------
 

Makefile

@@ -1,7 +1,7 @@
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 1.22.11
+override APP_VERSION := 1.23.1
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -14,13 +14,13 @@ for a full-fledged production platform.
     * - releases
       - | |latest-version| |commits-since|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.22.11.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.23.1.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.22.11...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.23.1...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-1.22.11-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-1.23.1-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.22.11
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.23.1
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-1.22.11 2023-02-03T04:58:52Z
+1.23.1 2023-02-13T18:31:02Z

birdhouse/components/cowbird/config/cowbird/config.yml.template

@@ -50,12 +50,16 @@ sync_permissions:
         # the `user` variable name would be matched with `user_xyz` and `synced_file`, with `file_abc`.
         # Also, this key would need to sync permissions with the `thredds_workspace` resource key, considering the
         # `permissions_mapping` defined below. The `thredds_workspace` would be deduced to the resource path
-        # `/catalog/workspaces/user_xyz/dir1/dir2/subdir/file_abc`.
+        # `/thredds/catalog/workspaces/user_xyz/dir1/dir2/subdir/file_abc`.
         # The types of each segment of this target resource path would be deduced
         # from the `thredds_workspace` config below.
         thredds_workspace:
-          - name: catalog
+          - name: thredds
             type: service
+          # not a resource in Magpie
+          # 'catalog' is the file/view format specifier for the rest of the path
+          # - name: catalog
+          #  type: directory
           - name: workspaces
             type: directory
           - name: "{user}"
@@ -102,13 +106,17 @@ sync_permissions:
       - "geoserver_workspace : createStoredQuery <-> thredds_workspace : write"
   weaver_outputs:
     services:
-      api:
+      weaver:
         process_description:
+          - name: weaver
+            type: service
           - name: processes
             type: route
           - name: "{processID}"
             type: route
         process_job_status:
+          - name: weaver
+            type: service
           - name: processes
             type: route
           - name: "{processID}"
@@ -118,28 +126,106 @@ sync_permissions:
           - name: "{jobID}"
             type: route
         job_status:
+          - name: weaver
+            type: service
           - name: jobs
             type: route
           - name: "{jobID}"
             type: route
         job_outputs:
+          - name: weaver
+            type: service
           - name: jobs
             type: route
           - name: "{jobID}"
             type: route
           - name: outputs
             type: route
+        job_output_single:
+          - name: weaver
+            type: service
+          - name: jobs
+            type: route
+          - name: "{jobID}"
+            type: route
+          - name: outputs
+            type: route
+          - name: "{outputID}"
+            type: route
+      # see 'optional-components/secure-data-proxy' for more details on protected WPS-outputs
+      wps_outputs:
+        # /wpsoutputs/weaver/{public|<user-id>}/{job-id}
         weaver_wps_outputs:
+          - name: secure-data-proxy
+            type: service
           - name: wpsoutputs
             type: route
           - name: weaver
             type: route
+          - name: "{user_context_dir}"
+            type: route
           - name: "{jobID}"
             type: route
+        # /wpsoutputs/weaver/{public|<user-id>}/{job-id}/{output-file}
+        weaver_wps_output_single:
+          - name: secure-data-proxy
+            type: service
+          - name: wpsoutputs
+            type: route
+          - name: weaver
+            type: route
+          - name: "{user_context_dir}"
+            type: route
+          - name: "{jobID}"
+            type: route
+          - name: "{outputID}"
+            type: route
+      thredds:
+        # /twitcher/ows/proxy/thredds/catalog/birdhouse/wps_outputs/weaver/catalog.html
+        # /twitcher/ows/proxy/thredds/catalog/birdhouse/wps_outputs/weaver/{public|<user-id>}/catalog.html
+        # /twitcher/ows/proxy/thredds/catalog/birdhouse/wps_outputs/weaver/{public|<user-id>}/{job-id}/catalog.html
+        # /twitcher/ows/proxy/thredds/catalog/birdhouse/wps_outputs/weaver/{public|<user-id>}/{job-id}/{output-file}
+        # note: paths start after ows-proxy portion extracted when Twitcher/Magpie resolve between each other
+        thredds_wps_outputs:
+          - name: thredds
+            type: service
+          # not a resource in Magpie
+          # 'catalog' is the file/view format specifier for the rest of the path
+          # - name: catalog
+          #  type: directory
+          - name: birdhouse
+            type: directory
+          - name: wps_outputs
+            type: directory
+          - name: weaver
+            type: directory
+          - name: "{user_context_dir}"
+            type: directory
+          - name: "{jobID}"
+            type: directory
+        thredds_wps_output_single:
+          - name: thredds
+            type: service
+          # not a resource in Magpie
+          # 'catalog' is the file/view format specifier for the rest of the path
+          # - name: catalog
+          #  type: directory
+          - name: birdhouse
+            type: directory
+          - name: wps_outputs
+            type: directory
+          - name: weaver
+            type: directory
+          - name: "{user_context_dir}"
+            type: directory
+          - name: "{jobID}"
+            type: directory
+          - name: "{outputID}"
+            type: file
     permissions_mapping:
       # When user is granted access to an output (either side),
       # output retrieval is allowed from both endpoints (wps-outputs/weaver).
-      - "weaver_wps_outputs : read <-> job_outputs : read"
+      - "weaver_wps_outputs : read -> job_outputs : read"
       # When output can be retrieved, access to details about the process and
       # the job are also provided (to understand what each output represents),
       # but getting read access to a process description should not grant
@@ -149,7 +235,16 @@ sync_permissions:
       - "weaver_wps_outputs : read -> job_status : read"
       # process-prefixed items can be only one-way since wps-outputs does not
       # encode the 'processID' information ('jobID' directly the top-level dir)
-      - "process_job_status : read -> weaver_wps_outputs : read"
       - "process_job_status : read -> job_status : read"
+      # NOTE:
+      #   missing 'user_context_dir' information not defined in path of process execution request
+      #   this permission must be set using the magpie/twitcher pre/post request hook to extract the authorized user
+      ###- "process_job_status : read -> weaver_wps_outputs : read"
       # different permission (match), otherwise all jobs/outputs become available.
       - "process_job_status : read -> process_description : read-match"
+      # corresponding outputs retrieved under wps-outputs or thredds share access
+      - "weaver_wps_outputs : read <-> thredds_wps_outputs : read"
+      # permissions if outputs are shared one-by-one in case of multiple files produced by the process
+      - "weaver_wps_output_single : read <-> thredds_wps_output_single : read"
+      - "weaver_wps_output_single : read -> job_output_single : read"
+      - "thredds_wps_output_single : read -> job_output_single : read"

birdhouse/components/cowbird/default.env

@@ -4,9 +4,9 @@
 
 # All env in this default.env can be overridden by env.local.
 
-# All env in this default.env must NOT depend on any other env.  If they do,
-# must use single quote to avoid early expansion before overrides in local.env
-# are applied and must add to the list of DELAYED_EVAL.
+# All env in this default.env must NOT depend on any other env.  If they do, they
+# must use single quotes to avoid early expansion before overrides in env.local
+# are applied and must be added to the list of DELAYED_EVAL.
 
 # add any new variables not already in 'VARS' or 'OPTIONAL_VARS' that must be replaced in templates here
 # single quotes are important in below list to keep variable names intact until 'pavics-compose' parses them

birdhouse/components/cowbird/docker-compose-extra.yml

@@ -73,10 +73,15 @@ services:
 
   # extend Magpie permissions to grant access to Cowbird API via secured Twitcher proxy
   magpie:
+    links:
+      # must have link to send webhook requests directly though internal network
+      - cowbird
     volumes:
       # NOTE:
-      #   Although file uses the "config.yml" format, it is very important to pass it as independent/duplicate reference
-      #   provider/permissions config files. This is because 'MAGPIE_CONFIG_PATH' is not used to allow parsing multiple
-      #   config files for each extendable service, using loading of all configuration files found in mount directories.
+      #   Although the file uses the combined "config.yml" format, it is very important to pass it as independent and
+      #   duplicate references for providers/permissions/webhooks config files. This is because 'MAGPIE_CONFIG_PATH' is
+      #   not used to allow parsing additive per-component config files for each extendable service, using loading of
+      #   all configuration files found in mounted directories.
       - ./components/cowbird/config/magpie/config.yml:/opt/local/src/magpie/config/permissions/cowbird.yml:ro
       - ./components/cowbird/config/magpie/config.yml:/opt/local/src/magpie/config/providers/cowbird.yml:ro
+      - ./components/cowbird/config/magpie/config.yml:/opt/local/src/magpie/config/webhooks/cowbird.yml:ro

birdhouse/components/monitoring/default.env

@@ -1,8 +1,8 @@
 # All env in this default.env can be overridden by env.local.
 
-# All env in this default.env must NOT depend on any other env.  If they do,
-# must use single quote to avoid early expansion before overrides in local.env
-# are applied and must add to the list of DELAYED_EVAL.
+# All env in this default.env must NOT depend on any other env.  If they do, they
+# must use single quotes to avoid early expansion before overrides in env.local
+# are applied and must be added to the list of DELAYED_EVAL.
 
 export GRAFANA_ADMIN_PASSWORD="changeme!"
 export ALERTMANAGER_ADMIN_EMAIL_RECEIVER=""  # "user1@example.com,user2@example.com"