Keycloak integration first pass (#848)

* keycloak helm and ingress only

* admin password

* attempt to update realm

* install keycloak operator

* keycloak in own namespace to see if operators work

* keycloak namespace

* Mrparkers keycloak provider for terraform attempt 2

* realm by provider

* no operator

* Keycloak helm/config as new standalone tf deploy stages

* specify mrparkers in module

* qhub-bot password

* Users and Groups in Keycloak via Provider

* User group membership

* forwardauth uses keycloak (jh does not yet)

* jh auth uses keycloak (but uid/gid not correct)

* smtp can be set in qhub realm

* keycloak initial-root-password

* tls_insecure_skip_verify in keycloak for local k8s

* remove uids and gids (import and save not working)

* migration working, no save

* saves and loads state

* bring uidgid up-to-date in jhub config

* Attempt at logout_redirect_url logout of keycloak

* nfsuserinfo readme

* Rename nfsuserinfo to userinfo

* adapt userinfo to use keycloak

* remove old json files

* keycloak userinfo working for yaml users

* manually added users/groups from keycloak

* admin hook; users as default group

* group/user validation and preservation

* attempt to use OAUTH_LOGOUT_REDIRECT_URL from oauthenticator 14.2.0

* oauthenticator 14.1.0 is latest on conda-forge

* logout redirect uri

* black/flake8

* tf fmt, remove obsolete nfsuserinfo config map

* temp passwords for users

* qhub-jupyterhub-theme 0.3.2 in jupyterhub image

* fix conda-store build; allow list in jh template vars

* Arbitrary Extension pods in QHub

* keycloak env vars in extensions

* fix forwardauth

* keycloak client for extensions (attempt)

* oauth client secret

* f-string escape

* fix redirect base

* groups membership mapper in keycloak client

* JWT_SECRET

* logout chaining

* remove smtp settings

* black fix

* terraform fmt

* full-only, and Auth0/GitHub keycloak idps

* auth0 and github idps

* fix github

* always generate keycloak root password

* auth0/github working on qhub init

* tf fmt

* allow very simple users/groups in yaml

* black fmt

* first attempt at direct mount shared

* switch userinfo for keycloak

* fix shared link

* only chown once

* remove userinfo from tf

* remove userinfo code

* profiles attribute of groups in keycloak

* Flake8 fix

* cypress keycloak login

* Fix safe username / groups

* remove old keycloak module

* tf 1.0.5, tls skip verify in oauth

* OAUTH2_TLS_VERIFY var

* encourage auth exec order for keycloak

* update to latest yaml processing including comments

* keycloak qhub upgrade

docs/source/admin_guide/faq.md

@@ -22,3 +22,19 @@ terraform](https://github.com/Quansight/qhub/issues/786)
 To disable the AZRebalance service, follow the steps in this [AWS
 documentation](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html)
 to suspend the AZRebalance service.
+
+## Can I deploy an arbitrary pod?
+
+Yes, add extensions as follows:
+
+```
+extensions:
+  - name: echo-test
+    image: inanimate/echo-server:latest
+    urlslug: echo
+    private: true
+```
+
+This will deploy a simple service based on the image provided. name must be a simple terraform-friendly string.
+
+It will be available on your QHub site at the /echo URL (or whatever urlslug you provide). Users will be required to be logged in if private is true.

docs/source/installation/configuration.md

@@ -208,8 +208,6 @@ security:
     config:
       client_id: ...
       client_secret: ...
-      oauth_callback_url: 'http[s]://[your-host]/hub/oauth_callback'
-      scope: ["openid", "email", "profile"]
       auth0_subdomain: ...
 ```
 
@@ -226,7 +224,6 @@ security:
     config:
       client_id: ...
       client_secret: ...
-      oauth_callback_url: 'http[s]://[your-host]/hub/oauth_callback'
 ```
 
 #### Password Based Authentication

environment-dev.yaml

@@ -3,13 +3,11 @@ channels:
   - conda-forge
 dependencies:
   - cookiecutter ==1.7.2
-  - gitignore-parser==0.0.8
   - ruamel.yaml
   - cloudflare
   - auth0-python
   - pydantic
   - pynacl
-  - bcrypt
 
   # dev dependencies
   - flake8 ==3.8.4

qhub/cli/deploy.py

@@ -1,11 +1,10 @@
 import pathlib
 import logging
 
-from ruamel import yaml
-
 from qhub.deploy import deploy_configuration
 from qhub.schema import verify
 from qhub.render import render_template
+from qhub.utils import load_yaml
 
 logger = logging.getLogger(__name__)
 
@@ -41,6 +40,11 @@ def create_deploy_subcommand(subparser):
         action="store_true",
         help="Disable auto-rendering in deploy stage",
     )
+    subparser.add_argument(
+        "--full-only",
+        action="store_true",
+        help="Only carry out one full pass instead of targeted sections (for development purposes)",
+    )
     subparser.set_defaults(func=handle_deploy)
 
 
@@ -51,8 +55,7 @@ def handle_deploy(args):
             f"passed in configuration filename={config_filename} must exist"
         )
 
-    with config_filename.open() as f:
-        config = yaml.safe_load(f.read())
+    config = load_yaml(config_filename)
 
     verify(config)
 
@@ -65,4 +68,5 @@ def handle_deploy(args):
         args.dns_auto_provision,
         args.disable_prompt,
         args.skip_remote_state_provision,
+        args.full_only,
     )

qhub/cli/destroy.py

@@ -1,11 +1,10 @@
 import pathlib
 import logging
 
-from ruamel import yaml
-
 from qhub.destroy import destroy_configuration
 from qhub.schema import verify
 from qhub.render import render_template
+from qhub.utils import load_yaml
 
 logger = logging.getLogger(__name__)
 
@@ -34,8 +33,7 @@ def handle_destroy(args):
             f"passed in configuration filename={config_filename} must exist"
         )
 
-    with config_filename.open() as f:
-        config = yaml.safe_load(f.read())
+    config = load_yaml(config_filename)
 
     verify(config)
 

qhub/cli/initialize.py

@@ -1,7 +1,6 @@
-from ruamel import yaml
-
 from qhub.initialize import render_config
 from qhub.schema import ProviderEnum
+from qhub.utils import yaml
 
 
 def create_init_subcommand(subparser):
@@ -80,9 +79,7 @@ def handle_init(args):
 
     try:
         with open("qhub-config.yaml", "x") as f:
-            yaml.dump(
-                config, f, default_flow_style=False, Dumper=yaml.RoundTripDumper
-            )  # RoundTripDumper avoids alphabetical sorting of yaml file
+            yaml.dump(config, f)
     except FileExistsError:
         raise ValueError(
             "A qhub-config.yaml file already exists. Please move or delete it and try again."

qhub/cli/render.py

@@ -1,9 +1,8 @@
 import pathlib
 
-from ruamel import yaml
-
 from qhub.render import render_template
 from qhub.schema import verify
+from qhub.utils import load_yaml
 
 
 def create_render_subcommand(subparser):
@@ -22,8 +21,7 @@ def handle_render(args):
             f"passed in configuration filename={config_filename} must exist"
         )
 
-    with config_filename.open() as f:
-        config = yaml.safe_load(f.read())
+    config = load_yaml(config_filename)
 
     verify(config)
 

qhub/cli/validate.py

@@ -1,8 +1,8 @@
 import pathlib
 
-from ruamel import yaml
 from qhub.schema import verify
 from qhub.provider.cicd.linter import comment_on_pr
+from qhub.utils import load_yaml
 
 
 def create_validate_subcommand(subparser):
@@ -39,8 +39,7 @@ def handle_validate(args):
             f"passed in configuration filename={config_filename} must exist"
         )
 
-    with config_filename.open() as f:
-        config = yaml.safe_load(f.read())
+    config = load_yaml(config_filename)
 
     if args.enable_commenting:
         # for PR's only

qhub/constants.py

@@ -1 +1 @@
-TERRAFORM_VERSION = "1.0.0"
+TERRAFORM_VERSION = "1.0.5"

qhub/deploy.py

@@ -17,6 +17,7 @@ def deploy_configuration(
     dns_auto_provision,
     disable_prompt,
     skip_remote_state_provision,
+    full_only,
 ):
     logger.info(f'All qhub endpoints will be under https://{config["domain"]}')
 
@@ -28,6 +29,7 @@ def deploy_configuration(
                 dns_auto_provision,
                 disable_prompt,
                 skip_remote_state_provision,
+                full_only,
             )
         except CalledProcessError as e:
             logger.error(e.output)
@@ -40,6 +42,7 @@ def guided_install(
     dns_auto_provision,
     disable_prompt=False,
     skip_remote_state_provision=False,
+    full_only=False,
 ):
     # 01 Check Environment Variables
     check_cloud_credentials(config)
@@ -59,65 +62,85 @@ def guided_install(
 
     # 3 kubernetes-alpha provider requires that kubernetes be
     # provisionioned before any "kubernetes_manifests" resources
+    logger.info("Running terraform init")
     terraform.init(directory="infrastructure")
-    terraform.apply(
-        directory="infrastructure",
-        targets=[
-            "module.kubernetes",
-            "module.kubernetes-initialization",
-        ],
-    )
 
-    # 04 Create qhub initial state (up to nginx-ingress)
-    terraform.init(directory="infrastructure")
-    terraform.apply(
-        directory="infrastructure",
-        targets=[
+    if not full_only:
+        targets = [
             "module.kubernetes",
             "module.kubernetes-initialization",
-            "module.kubernetes-ingress",
-        ],
-    )
-
-    cmd_output = terraform.output(directory="infrastructure")
-    # This is a bit ugly, but the issue we have at the moment is being unable
-    # to parse cmd_output as json on Github Actions.
-    ip_matches = re.findall(r'"ip": "(?!string)(.+)"', cmd_output)
-    hostname_matches = re.findall(r'"hostname": "(?!string)(.+)"', cmd_output)
-    if ip_matches:
-        ip_or_hostname = ip_matches[0]
-    elif hostname_matches:
-        ip_or_hostname = hostname_matches[0]
-    else:
-        raise ValueError(f"IP Address not found in: {cmd_output}")
-
-    # 05 Update DNS to point to qhub deployment
-    if dns_auto_provision and dns_provider == "cloudflare":
-        record_name, zone_name = (
-            config["domain"].split(".")[:-2],
-            config["domain"].split(".")[-2:],
+        ]
+
+        logger.info(f"Running Terraform Stage: {targets}")
+        terraform.apply(
+            directory="infrastructure",
+            targets=targets,
+        )
+
+        # 04 Create qhub initial state (up to nginx-ingress)
+        targets = ["module.kubernetes-ingress"]
+        logger.info(f"Running Terraform Stage: {targets}")
+        terraform.apply(
+            directory="infrastructure",
+            targets=targets,
         )
-        record_name = ".".join(record_name)
-        zone_name = ".".join(zone_name)
-        if config["provider"] in {"do", "gcp", "azure"}:
-            update_record(zone_name, record_name, "A", ip_or_hostname)
-            if config.get("clearml", {}).get("enabled"):
-                add_clearml_dns(zone_name, record_name, "A", ip_or_hostname)
-        elif config["provider"] == "aws":
-            update_record(zone_name, record_name, "CNAME", ip_or_hostname)
-            if config.get("clearml", {}).get("enabled"):
-                add_clearml_dns(zone_name, record_name, "CNAME", ip_or_hostname)
+
+        cmd_output = terraform.output(directory="infrastructure")
+        # This is a bit ugly, but the issue we have at the moment is being unable
+        # to parse cmd_output as json on Github Actions.
+        ip_matches = re.findall(r'"ip": "(?!string)(.+)"', cmd_output)
+        hostname_matches = re.findall(r'"hostname": "(?!string)(.+)"', cmd_output)
+        if ip_matches:
+            ip_or_hostname = ip_matches[0]
+        elif hostname_matches:
+            ip_or_hostname = hostname_matches[0]
         else:
-            logger.info(
-                f"Couldn't update the DNS record for cloud provider: {config['provider']}"
+            raise ValueError(f"IP Address not found in: {cmd_output}")
+
+        # 05 Update DNS to point to qhub deployment
+        if dns_auto_provision and dns_provider == "cloudflare":
+            record_name, zone_name = (
+                config["domain"].split(".")[:-2],
+                config["domain"].split(".")[-2:],
+            )
+            record_name = ".".join(record_name)
+            zone_name = ".".join(zone_name)
+            if config["provider"] in {"do", "gcp", "azure"}:
+                update_record(zone_name, record_name, "A", ip_or_hostname)
+                if config.get("clearml", {}).get("enabled"):
+                    add_clearml_dns(zone_name, record_name, "A", ip_or_hostname)
+            elif config["provider"] == "aws":
+                update_record(zone_name, record_name, "CNAME", ip_or_hostname)
+                if config.get("clearml", {}).get("enabled"):
+                    add_clearml_dns(zone_name, record_name, "CNAME", ip_or_hostname)
+            else:
+                logger.info(
+                    f"Couldn't update the DNS record for cloud provider: {config['provider']}"
+                )
+        elif not disable_prompt:
+            input(
+                f"Take IP Address {ip_or_hostname} and update DNS to point to "
+                f'"{config["domain"]}" [Press Enter when Complete]'
             )
-    elif not disable_prompt:
-        input(
-            f"Take IP Address {ip_or_hostname} and update DNS to point to "
-            f'"{config["domain"]}" [Press Enter when Complete]'
+
+        # Now Keycloak Helm chart
+        targets = ["module.kubernetes-keycloak-helm"]
+        logger.info(f"Running Terraform Stage: {targets}")
+        terraform.apply(
+            directory="infrastructure",
+            targets=targets,
+        )
+
+        # Now Keycloak realm and config
+        targets = ["module.kubernetes-keycloak-config"]
+        logger.info(f"Running Terraform Stage: {targets}")
+        terraform.apply(
+            directory="infrastructure",
+            targets=targets,
         )
 
-    # 06 Full deploy QHub
+    # Full deploy QHub
+    logger.info("Running Terraform Stage: FULL")
     terraform.apply(directory="infrastructure")