Keycloak Export

docs/source/admin_guide/backup.md

@@ -1,19 +1,31 @@
-# Manual backups
+# Manual Backups
 
-Manual backups requires amount to:
+Your cloud provider may have native ways to backup your Kubernetes cluster and volumes.
+
+This guide describes how you would manually obtain the data you need to repopulate your QHub if your cluster is lost and you wish to start it up again from the `qhub-config.yaml` file.
+
+There are three main locations that you need to backup:
+
+1. The Network File System (NFS) volume where all JupyterLab workspace files are stored
+2. The Keycloak user/group database
+3. The JupyterHub database (for Dashboard configuration)
+
+## Network File System
+
+This amounts to:
 
 - Tarballing the /home directory
 - Saving to block storage [s3, google cloud storage, etc]
 - Downloading and untaring to new cluster
 
-This specific guide shows how to do this on a cluster on AWS.
+This specific guide shows how to do this on an AWS cluster and upload to AWS S3.
 
-## Pre-requisites
+### Pre-requisites
 
 - [Install kubectl](<https://kubernetes.io/docs/tasks/tools/>)
 - [Install AWS command-line tool](<https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html>)
 
-## Kubectl configuration
+### Kubectl configuration
 
 To setup kubectl, obtain the name of the cluster. If the user knows the deployment region of the current cluster, this is straightforward:
 
@@ -27,7 +39,7 @@ Copy the relevant name from this output, and run this command:
 aws eks update-kubeconfig  --region us-west-2 --name <relevant-name>
 ```
 
-## Pod deployment
+### Pod deployment
 
 With kubectl configured, the user now needs to deploy the pod that allows the user to access the cluster files. First save the follow pod specification to a file named `pod.yaml`
 
@@ -41,7 +53,7 @@ spec:
   volumes:
     - name: volume-to-debug-ubuntu
       persistentVolumeClaim:
-        claimName: "<mount-share-drive>"
+        claimName: "nfs-mount-dev-share"
   containers:
     - name: debugger
       image: ubuntu
@@ -51,9 +63,7 @@ spec:
           name: volume-to-debug-ubuntu
 ```
 
-To determine what should replace `<mount-share-drive>` run `kubectl get pvc -n dev`. This is the volume that doesn't has conda in the name, and has the same storage space as specified by the `shared<sub>filesystem</sub>` line in `qhub-config.yaml`. In the example the name is `nfs-mount-dev-share`
-
-With the name of the nfs volume saved in the file, run:
+Run:
 
 ```shell
 kubectl apply -f pod.yaml -n dev
@@ -67,7 +77,7 @@ kubectl exec -n dev --stdin --tty volume-debugger-ubuntu -- /bin/bash
 
 Again replacing the `dev` namespace as needed.
 
-## Installations
+### Installations
 
 The user must install several `apt` packages, as the pod spun up is a basic pod. The following commands installs them:
 
@@ -77,7 +87,7 @@ apt install curl -y
 apt install unzip -y
 ```
 
-Because the user is on AWS, the AWS command-line tool is also installed.:
+Because the user is on AWS, the AWS command-line tool is also installed:
 
 ```shell
 curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
@@ -86,9 +96,9 @@ unzip awscliv2.zip
 aws configure
 ```
 
-The last command from preceding prompts for your AWS public/private key, and default region. Past each of these and press enter. To ignore and skip the output, press enter.
+The last line in the command above prompts for your AWS public/private key and default region. Paste each of these and press enter. To ignore and skip the output, press enter.
 
-## Backups
+### Backups
 
 To backup the file system, run:
 
@@ -99,18 +109,18 @@ tar -cvf <custom_name>.tar /home
 
 The preferred naming scheme includes a year-month-day, example `2021-04-23<sub>home</sub><sub>backup.tar</sub>`. The user can utilize multi-backups through this step. This step takes several minutes depending on the size of the home directories.
 
-## Upload to block storage
+### Upload to block storage
 
 Once this is complete, the user uploads the tar file to S3 using the AWS command-line tool:
 
 ```shell
 aws s3 cp 2021-04-23.tar s3://<your_bucket_name>/backups/2021-04-23.tar
 ```
 
-Replacing your <your<sub>bucket</sub><sub>name</sub>> with a bucket you have created. If you don't have an existing bucket, instructions are here:
+Replacing `your_bucket_name` with a bucket you have created. If you don't have an existing bucket, instructions are here:
 <https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html>
 
-## Download from block storage and decompress
+### Download from block storage and decompress
 
 Now that the data backed up, perform the same steps preceding for the new cluster. This includes:
 
@@ -135,7 +145,7 @@ tar -xvf 2021-04-23.tar
 
 The file permissions for the default tar is same as the original files.
 
-## Google cloud provider
+### Google cloud provider
 
 To use the Google Cloud provider, install [gsutil](https://cloud.google.com/storage/docs/gsutil_install). The instructions are same as the preceding steps. Additionally, use these commands for copy/download of the backup:
 
@@ -147,7 +157,64 @@ cd /data
 gsutil cp gs://<your_bucket_name>/backups/2021-04-23.tar .
 ```
 
-## Digital Ocean
+### Digital Ocean
 
 Similar instructions, but use Digital Ocean spaces. This guide explains installation of the command-line tool:
 https://www.digitalocean.com/community/tutorials/how-to-migrate-from-amazon-s3-to-digitalocean-spaces-with-rclone
+
+## Keycloak User/Group Database
+
+QHub provides a simple script to export the important user/group database. Your new QHub cluster will recreate a lot of Keycloak config (including new Keycloak clients which will have new secrets), so only the high-level Group and User info is exported.
+
+If you have a heavily customized Keycloak configuration, some details may be omitted in this export.
+
+### Export Keycloak
+
+The export script is at [`qhub/scripts/keycloak-export.py`](https://github.com/Quansight/qhub/blob/main/scripts/keycloak-export.py).
+
+Locate your `qhub-config.yaml` file, for example by checking out of your Git repo for you QHub. Activate a virtual environment with the `qhub` Python package installed.
+
+This assumes that the password visible in the `qhub-config.yaml` file under the `security.keycloak.initial_root_password` field is still valid for the root user.
+
+If not, first set the `KEYCLOAK_ADMIN_PASSWORD` environment variable to the new value.
+
+Run the following to create the export file:
+
+```
+python qhub/scripts/keycloak-export.py -c qhub-config.yaml > exported-keycloak.json
+```
+
+You may wish to upload the Keycloak export to the same S3 location where you uploaded the TAR file in the NFS section.
+
+### Import Keycloak
+
+To re-import your users and groups, [login to the /auth/ URL](../installation/login.md) using the root username and password.
+
+Under 'Manage' on the left-hand side, click 'Import'. Locate the `exported-keycloak.json` file and select it. Then click the 'Import' button.
+
+All users and groups should now be present in Keycloak. Note that password will not have been restored so may need to be reset.
+
+## JupyterHub Database
+
+The JupyterHub database will mostly be recreated when you start a new cluster anyway, but should be backed up to save Dashboard configurations.
+
+You can do something very similar to the NFS backup, above - this time you need to back up a file on the PersistentVolume `hub-db-dir`. First, you might think you can just make a new `pod.yaml` file, this time specifying `claimName: "hub-db-dir"` instead of `claimName: "nfs-mount-dev-share"`. However, `hub-db-dir` is 'Read Write Once' - the 'Once' meaning it can only be mounted to one pod at a time, but the JupyterHub pod will already have this mounted!
+
+So instead of mounting to a new 'debugger pod' we have to access the JupyterHub pod directly and see what we can do from there.
+
+Look up the JupyterHub pod:
+```
+kubectl get pods -n dev
+```
+
+It will be something like `hub-765c9488d6-8z4nj`.
+
+Get a shell into that pod:
+
+```
+kubectl exec -n dev --stdin --tty hub-765c9488d6-8z4nj -- /bin/bash
+```
+
+There is no need to TAR anything up since the only file required to be backed up is `/srv/jupyterhub/jupyterhub.sqlite`.
+
+So we just need to upload the file to S3. You might want to install the AWS CLI tool as we did before, but unfortunately the Hub container is quite locked down and it isn't straightforward to install that... You might need to upload to S3 using curl directly as [explained in this article](https://www.gyanblog.com/aws/how-upload-aws-s3-curl/).

qhub/keycloak.py

@@ -19,6 +19,21 @@ def do_keycloak(config_filename, *args):
             "Only keycloak command is 'keycloak adduser username [password]'"
         )
 
+    keycloak_admin = get_keycloak_admin_from_config(config_filename)
+
+    new_user_dict = {"username": args[1], "enabled": True}
+    if len(args) >= 3:
+        new_user_dict["credentials"] = [
+            {"type": "password", "value": args[2], "temporary": False}
+        ]
+    else:
+        print("Not setting any password (none supplied)")
+
+    print(f"Adding user {args[1]}")
+    keycloak_admin.create_user(new_user_dict)
+
+
+def get_keycloak_admin_from_config(config_filename):
     config = load_yaml(config_filename)
 
     verify(config)
@@ -51,13 +66,4 @@ def do_keycloak(config_filename, *args):
     ) as e:
         raise ValueError(f"Failed to connect to Keycloak server: {e}")
 
-    new_user_dict = {"username": args[1], "enabled": True}
-    if len(args) >= 3:
-        new_user_dict["credentials"] = [
-            {"type": "password", "value": args[2], "temporary": False}
-        ]
-    else:
-        print("Not setting any password (none supplied)")
-
-    print(f"Adding user {args[1]}")
-    keycloak_admin.create_user(new_user_dict)
+    return keycloak_admin

scripts/keycloak-export.py

@@ -0,0 +1,51 @@
+import logging
+import sys
+import pathlib
+import argparse
+import json
+
+from qhub.keycloak import get_keycloak_admin_from_config
+
+logging.basicConfig(level=logging.INFO)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Export users and groups from QHub.")
+    parser.add_argument("-c", "--config", help="qhub configuration", required=True)
+    args = parser.parse_args()
+
+    handle_keycloak_export(args)
+
+
+def handle_keycloak_export(args):
+    config_filename = pathlib.Path(args.config)
+    if not config_filename.is_file():
+        raise ValueError(
+            f"passed in configuration filename={config_filename} must exist"
+        )
+
+    keycloak_admin = get_keycloak_admin_from_config(config_filename)
+
+    realm = {"id": "qhub", "realm": "qhub"}
+
+    def process_user(u):
+        uid = u["id"]
+        memberships = keycloak_admin.get_user_groups(uid)
+
+        del u["id"]
+        u["groups"] = [g["name"] for g in memberships]
+        return u
+
+    realm["users"] = [process_user(u) for u in keycloak_admin.get_users()]
+
+    realm["groups"] = [
+        {"name": g["name"], "path": g["path"]}
+        for g in keycloak_admin.get_groups()
+        if g["name"] not in {"users", "admin"}
+    ]
+
+    json.dump(realm, sys.stdout, indent=2)
+
+
+if __name__ == "__main__":
+    main()