Improve linkage of PR CL items for admin (#6706, PR #6712)

.github/ISSUE_TEMPLATE/upgrade.md

@@ -13,28 +13,28 @@ _period: 14 days
   - [ ] Remove unused dependencies with high or critical CVEs
   - [ ] Push commit to GitHub (directly to `master` branch, no PR needed)
   - [ ] GH Action workflow succeeded
-  - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm) 
+  - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm/tags) 
 - [ ] Update [Elasticsearch image](https://github.com/DataBiosphere/azul-docker-elasticsearch)
   - [ ] Bump [base image](https://hub.docker.com/_/elasticsearch/tags) tag (only minor and patch versions), if possible
   - [ ] Bump internal version 
   - [ ] Remove unused dependencies with high or critical CVEs
   - [ ] Push commit to GitHub (directly to `main` branch, no PR needed)
   - [ ] GH Action workflow succeeded
-  - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch) 
+  - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch/tags) 
 - [ ] Update [BigQuery Emulator image](https://github.com/DataBiosphere/azul-bigquery-emulator)
   - [ ] Bump [base image](https://hub.docker.com/_/debian/tags?name=bookworm) tag, if possible
   - [ ] Bump internal version 
   - [ ] Push commit to GitHub (directly to `azul` branch, no PR needed)
   - [ ] GH Action workflow succeeded
-  - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator) 
+  - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator/tags) 
 - [ ] Create Azul PR, connected to this issue, with … 
     - [ ] … changes to `requirements*.txt` from open Dependabot PRs, one commit per PR
     - [ ] … upgrade direct Python dependencies, [reference the operator manual](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#upgrade-direct-python-dependencies) for instructions <sub>or not applicable</sub>
     - [ ] … update to [Python](https://hub.docker.com/_/python/tags) (only patch versions) <sub>or no update available</sub>
     - [ ] … update to [Terraform](https://hub.docker.com/r/hashicorp/terraform/tags) (only patch versions) <sub>or no update available</sub>
-    - [ ] … new [PyCharm image](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm)
-    - [ ] … new [Elasticsearch image](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch)
-    - [ ] … new [BigQuery Emulator image](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator)
+    - [ ] … new [PyCharm image](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm/tags)
+    - [ ] … new [Elasticsearch image](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch/tags)
+    - [ ] … new [BigQuery Emulator image](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator/tags))
     - [ ] … update to [Docker images](https://hub.docker.com/_/docker/tags) (only minor and patch versions) <sub>or no update available</sub>
     - [ ] … update to [GitLab](https://hub.docker.com/r/gitlab/gitlab-ce/tags) & [GitLab runner images](https://hub.docker.com/r/gitlab/gitlab-runner/tags) <sub>or no update available</sub>
     - [ ] … update to [ClamAV image](https://hub.docker.com/r/clamav/clamav/tags) <sub>or no update available</sub>

.github/PULL_REQUEST_TEMPLATE/anvilprod-promotion.md

@@ -53,7 +53,7 @@ Connected issue: #0000
 
 ### System administrator
 
-- [ ] Background migrations for `anvilprod.gitlab` are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
+- [ ] Background migrations for [`anvilprod.gitlab`](https://gitlab.explore.anvilproject.org/admin/background_migrations) are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
 - [ ] PR is assigned to only the operator
 
 
@@ -100,9 +100,9 @@ Connected issue: #0000
 
 ### System administrator
 
-- [ ] Removed unused image tags from [pycharm image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm) <sub>or this promotion does not alter references to that image</sub>
-- [ ] Removed unused image tags from [elasticsearch image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch) <sub>or this promotion does not alter references to that image</sub>
-- [ ] Removed unused image tags from [bigquery_emulator image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator) <sub>or this promotion does not alter references to that image</sub>
+- [ ] Removed unused image tags from [pycharm image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm/tags) <sub>or this promotion does not alter references to that image</sub>
+- [ ] Removed unused image tags from [elasticsearch image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch/tags) <sub>or this promotion does not alter references to that image</sub>
+- [ ] Removed unused image tags from [bigquery_emulator image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator/tags) <sub>or this promotion does not alter references to that image</sub>
 - [ ] PR is assigned to no one
 
 

.github/PULL_REQUEST_TEMPLATE/prod-promotion.md

@@ -53,7 +53,7 @@ Connected issue: #0000
 
 ### System administrator
 
-- [ ] Background migrations for `prod.gitlab` are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
+- [ ] Background migrations for [`prod.gitlab`](https://gitlab.azul.data.humancellatlas.org/admin/background_migrations) are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
 - [ ] PR is assigned to only the operator
 
 
@@ -95,9 +95,9 @@ Connected issue: #0000
 
 ### System administrator
 
-- [ ] Removed unused image tags from [pycharm image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm) <sub>or this promotion does not alter references to that image</sub>
-- [ ] Removed unused image tags from [elasticsearch image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch) <sub>or this promotion does not alter references to that image</sub>
-- [ ] Removed unused image tags from [bigquery_emulator image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator) <sub>or this promotion does not alter references to that image</sub>
+- [ ] Removed unused image tags from [pycharm image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm/tags) <sub>or this promotion does not alter references to that image</sub>
+- [ ] Removed unused image tags from [elasticsearch image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch/tags) <sub>or this promotion does not alter references to that image</sub>
+- [ ] Removed unused image tags from [bigquery_emulator image on DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator/tags) <sub>or this promotion does not alter references to that image</sub>
 - [ ] PR is assigned to no one
 
 

.github/PULL_REQUEST_TEMPLATE/upgrade.md

@@ -65,8 +65,8 @@ Connected issue: #0000
 
 ### System administrator
 
-- [ ] Background migrations for `dev.gitlab` are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
-- [ ] Background migrations for `anvildev.gitlab` are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
+- [ ] Background migrations for [`dev.gitlab`](https://gitlab.dev.singlecell.gi.ucsc.edu/admin/background_migrations) are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
+- [ ] Background migrations for [`anvildev.gitlab`](https://gitlab.anvil.gi.ucsc.edu/admin/background_migrations) are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
 - [ ] PR is assigned to only the operator
 
 

.github/pull_request_template.md

@@ -118,8 +118,8 @@ title is `Fix: ` followed by the issue title
 
 ### System administrator
 
-- [ ] Background migrations for `dev.gitlab` are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
-- [ ] Background migrations for `anvildev.gitlab` are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
+- [ ] Background migrations for [`dev.gitlab`](https://gitlab.dev.singlecell.gi.ucsc.edu/admin/background_migrations) are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
+- [ ] Background migrations for [`anvildev.gitlab`](https://gitlab.anvil.gi.ucsc.edu/admin/background_migrations) are complete <sub>or this PR is not labeled `deploy:gitlab`</sub>
 - [ ] PR is assigned to only the operator
 
 

.github/pull_request_template.md.template.py

@@ -32,6 +32,9 @@
 from azul.collections import (
     OrderedSet,
 )
+from azul.modules import (
+    load_script,
+)
 from azul.strings import (
     back_quote as bq,
     join_grammatically,
@@ -235,6 +238,16 @@ def main():
             emit(t, target_branch)
 
 
+def env_var(deployment: str, variable: str) -> str:
+    """
+    Return an environment variable value from the given deployment.
+    """
+    script = load_script('export_environment')
+    env, warning = script.load_env(deployment)
+    resolved_env = script.resolve_env(env)
+    return resolved_env[variable]
+
+
 def emit(t: T, target_branch: str):
     emit_checklist(
         [
@@ -715,7 +728,9 @@ def emit(t: T, target_branch: str):
                 *[
                     {
                         'type': 'cli',
-                        'content': f'Background migrations for `{d}.gitlab` are complete',
+                        'content': f'Background migrations for '
+                                   f'[`{d}.gitlab`](https://gitlab.{env_var(d, "AZUL_DOMAIN_NAME")}'
+                                   f'/admin/background_migrations) are complete',
                         'alt': 'or this PR is not labeled `deploy:gitlab`'
                     }
                     for d in t.target_deployments(target_branch)
@@ -1012,7 +1027,7 @@ def emit(t: T, target_branch: str):
                 *[
                     {
                         'type': 'cli',
-                        'content': f'Removed unused image tags from [{name} image on DockerHub]({url})',
+                        'content': f'Removed unused image tags from [{name} image on DockerHub]({url}/tags)',
                         'alt': 'or this promotion does not alter references to that image'
                     }
                     for name, url in custom_images.items()

scripts/export_environment.py

@@ -73,6 +73,14 @@ def env(self) -> DraftEnvironment:
         }
 
 
+class InvalidDeployment(RuntimeError):
+
+    def __init__(self, dir_: Path) -> None:
+        super().__init__(
+            f"{dir_} does not exist or is not a symbolic link to a directory."
+        )
+
+
 class InvalidActiveDeployment(RuntimeError):
 
     def __init__(self, dir_: Path) -> None:
@@ -91,45 +99,55 @@ def __init__(self, parent: Path, component: Path) -> None:
         )
 
 
-def load_env() -> Tuple[Environment, Optional[str]]:
+def load_env(deployment: Optional[str] = None
+             ) -> Tuple[Environment, Optional[str]]:
     """
     Load environment.py and environment.local.py modules from the project
-    root and the current active deployment directory, call their env()
-    function to obtain the environment dictionary and merge the dictionaries.
-    The entries from an environment.local.py take precedence over those from
-    a corresponding environment.py in the same directory. The modules from
-    the deployment directory take precedence over ones in the project root.
+    root and either the specified deployment or the current active deployment
+    directory, call their env() function to obtain the environment dictionary
+    and merge the dictionaries. The entries from an environment.local.py take
+    precedence over those from a corresponding environment.py in the same
+    directory. The modules from the deployment directory take precedence over
+    ones in the project root.
     """
 
     deployments_dir = root_dir / 'deployments'
     active_deployment_dir = deployments_dir / '.active'
-    if active_deployment_dir.exists():
-        if not active_deployment_dir.is_dir() or not active_deployment_dir.is_symlink():
-            raise InvalidActiveDeployment(active_deployment_dir)
-
-        # If active deployment is a component of another one, also load the parent
-        # deployments (like dev.gitlab).
-        active_deployment_dir = Path(os.readlink(str(active_deployment_dir)))
-        if not active_deployment_dir.is_absolute():
-            active_deployment_dir = deployments_dir / active_deployment_dir
-        if not active_deployment_dir.is_dir():
-            raise InvalidActiveDeployment(active_deployment_dir)
-        relative_active_deployment_dir = active_deployment_dir.relative_to(deployments_dir)
-        prefix, _, suffix = str(relative_active_deployment_dir).partition('.')
-        if suffix and suffix != 'local':
-            parent_deployment_dir = deployments_dir / prefix
-            if not parent_deployment_dir.exists():
-                raise BadParentDeployment(parent_deployment_dir, active_deployment_dir)
-        else:
-            parent_deployment_dir = None
+
+    if deployment is not None:
+        deployment_dir = deployments_dir / deployment
+        if not deployment_dir.is_dir():
+            raise InvalidDeployment(deployments_dir)
+        warning = None
+    elif active_deployment_dir.is_dir() and active_deployment_dir.is_symlink():
+        deployment_dir = Path(os.readlink(str(active_deployment_dir)))
+        if not deployment_dir.is_absolute():
+            deployment_dir = deployments_dir / deployment_dir
+        if not deployment_dir.is_dir():
+            raise InvalidActiveDeployment(deployment_dir)
         warning = None
+    elif active_deployment_dir.exists():
+        raise InvalidActiveDeployment(active_deployment_dir)
     else:
         warning = (
             f'No active deployment (missing {str(active_deployment_dir)!r}). '
             f'Loaded global defaults only.'
         )
-        active_deployment_dir = None
+        deployment_dir = None
+
+    if deployment_dir is None:
         parent_deployment_dir = None
+    else:
+        # If the deployment is a component of another one (e.g. `dev.gitlab`),
+        # also get the parent deployment's directory.
+        relative_deployment_dir = deployment_dir.relative_to(deployments_dir)
+        prefix, _, suffix = str(relative_deployment_dir).partition('.')
+        if suffix and suffix != 'local':
+            parent_deployment_dir = deployments_dir / prefix
+            if not parent_deployment_dir.exists():
+                raise BadParentDeployment(parent_deployment_dir, deployment_dir)
+        else:
+            parent_deployment_dir = None
 
     def _load(dir_path: Path, local: bool = False) -> Optional[EnvironmentModule]:
         """
@@ -153,10 +171,10 @@ def _load(dir_path: Path, local: bool = False) -> Optional[EnvironmentModule]:
             return None
 
     modules = [
-        active_deployment_dir and _load(active_deployment_dir, local=True),
+        deployment_dir and _load(deployment_dir, local=True),
         parent_deployment_dir and _load(parent_deployment_dir, local=True),
         _load(root_dir, local=True),
-        active_deployment_dir and _load(active_deployment_dir),
+        deployment_dir and _load(deployment_dir),
         parent_deployment_dir and _load(parent_deployment_dir),
         _load(root_dir)
     ]