OGC Disaster Pilot 2022 Sprint Meta Issue

[iameskild]

Context

Quansight is conducting a sprint as part of the 2022 OGC Disaster Pilot. The aim of the sprint is to demonstrate how QHub/Nebari could be used to quickly spin up a data science platform on the cloud of choice to provide practitioners the computational tools needed to respond to a disaster. The Sprint is scheduled for July

Audience & Structure

The event will be focused on scientists and engineers in the geospatial-ocean-met space. There will be two tracks/tutorials.

1. Tutorial demonstrating the use of the (Pangeo stack)[https://pangeo.io/) on QHub/Nebari
2. Tutorial walking through the installation of QHub/Nebari

It is expected that some participants will not want to install QHub themselves and will only be interested in learning about the Pangeo stack. The plan is to give them accounts on a hosted QHub/Nebari. Most probably via ESIP or via the OGC.

Currently, the plan is for a two day sprint event starting with two tutorials on the first day taking not more than the morning, after this there will be an async mechanism for folks to ask questions as they either try out the Pangeo stack in the hosted QHub/Nebari OR try installing QHub/Nebari for themselves.

High Priority Issues

Installation

• Clear working instructions on how to install QHub/Nebari on the clouds we decide to support
• Make sure we have sane instance sizes for the clouds we use in the demo
  • Small/Large/High Mem/Cheap GPU etc
  • Is our default conda store pod size reasonable
  • Are our Dask workers configured correctly for our instance sizes
  • We may be able to use the ESIP deployment as an example.
• Clear documentation of how to use Keycloak
  • How do groups and roles work. Is there a difference
  • Explain special groups i.e. currently Admin/Developer/Analyst
  • How do I add/remove people to QHub. (ideally this should be doable by anyone in the admin group) not just with the root password.
  • How are groups and shared folders connected.
• conda-store
  • fix for user created environments conda-environments not showing up for CDS-Dashboards
  • fix namespace clashes between filesystem conda-envs and user created environments
  • documentation around how to create/delete environments and namespaces and conda-store configuration
  • Rename default and filesystem namespaces and also explain the different between the envs that come from git and user created environments

Demo List

@dharhas will split this out in a new issue but at a high level. We will be demonstrating:

• Dask (Monitoring of cluster stats via grafana and prometheus #39)
• Xarray
• CDSDashboards (Create a method to use secrets in QHub config #13)
• Panel/Holoviz (related to CDS Create a method to use secrets in QHub config #13)
• VSCode (Test config file for DO has validation errors #55)
• Long running workflows via kbatch (Applying PR corrections from https://github.com/Quansight/qhub-ops/pull/79 #46)
• Scheduled cronjobs via kbatch (Create github-action for automatically deployment qhub.dev #47)
• using jupyter-ssh & tmux to access your pod and run things *
• using a token to connect to dask-gateway from local machine *
• multistep workflow via argo* (Fix github action for documentation deployment to readthedocs #48 and Create github-action for automatically deployment qhub.dev #47)

• stretch goals

Open Questions

• There is a requirement to show installation capability on multiple clouds. Do we support all 4 or say for the sprint we will only support 2.
• Which cloud install will we use in the demo.Probably AWS since many @rsignell-usgs demos use AWS hosted datasets
• Are we comfortable with our GPU support. Preference is to
• Do we want to refer to everything as Nebari. This is my preference.
• some of the datasets like 'sentinel-1' etc require AWS credentials. how do we handle this in the sprint.

Out of Scope

• Renaming of code to use Nebari is not required for this. We can explain we are in the middle of a rebranding.

References

2021 Disaster Pilot

Original Issue description from @iameskild below.

Clear deployment instructions

For the items listed below, most of these docs need to be validated / improved.

• Installation
• 🔴 Getting setup
• 🔴 qhub init and qhub deploy to the cloud
• Basic keycloak user / group management overview

I think that although not perfect, much of the team's effort in the past few months has been to stabilize the deployment process and I think with a few improvements and updates to the docs, we're in good shape for the demo. Here are a few items that could be addressed as a stretch goal:

• ENH - Run schema validation during qhub init #1267
• [BUG] - Increase general node default size #1111

Demos and clear documentation for the core services

These docs can be the demos or instructions walking users through some of the core features and services.

• Dask
  • 🔴 Test / clean up current dask-related docs
  • [ENH] - Upgrade Dask-Gateway to v2022.04 #1296
  • [BUG] - qhub-dask hasn't been updated from 0.3.13 and is inconsistent with docs #1297
  • [bug] Unable to connect to dashboard using Dask-Gateway and Labextension #1012
    • install dask lab ext from main #1321
• Conda-Store
  • 🔴 Establish and document that the only way to handle conda environments is by adding them to the qhub-config.yaml.
    • Perhaps even remove the mention that environments can be created any other way...
    • This will change in the future as conda-store is developed but given the issues with namespaces, it makes sense to simplify the messaging.
  • Test / clean up current conda-store docs - user_guide
  • Test / clean up current conda-store docs - management
• CDS Dashboards
  • Test / clean up current CDS Dashboards docs
  • [DOC] - Add an example of how to use panel-serve in CDSDashboards #1285
  • Add panel-serve as a CDS dashboard option #1070 - for me to wrap up
• KBatch
  • 🔴 Upgrade kbatch/kbatch-proxy version #1311
    • Upgrade kbatch version #1335
  • 🔴 [DOC] - Create documentation on how to use kbatch #1322
• Argo (stretch-goal)
  • Validate argo docs
  • Create example workflow tutorial

Given that we have some "bare-bones" examples of much of the above, it might be worthwhile developing a few more complex example notebooks. These could fall into the "tutorial" section of the diataxis documentation framework.

🔴 - deemed highest priority by eskild

[iameskild]

@trallard @dharhas @dhavide I have made an attempt at gathering issues that might need to be addressed as well as references to the docs that should be validated / cleaned up prior to the OGC sprint. As we have about a month remaining, not all of this can be addressed but it gives us a good place from which to start prioritizing.

[dharhas]

@rsignell-usgs feedback welcome. I'm on vacation the next 2 weeks but will try and respond.

[dharhas]

Actually the infracost work #1315 needs to be part of this sprint as well.

[kcpevey]

updates to the above list made by @kcpevey and @iameskild, see current state below:

Clear deployment instructions

Installation

• Docs issues:

  • Brainstorm web gui / rest server for qhub operations #36
    • Sane curated default environment, jupyterlab extensions, and theme for QHub #45)
  • Investigate use of Pulumi #38

• Dev issues:

  • [BUG] - Increase general node default size #1111
    • Increase the default instance sizes #1338

Infracost

• Docs issues:

  • Ensure that terraform version is >= 0.12.x #57

• Dev issues:

  • [ENH] - Add Infracost to QHub CI and CLI #1315
    • Generate QHub Costs via infracost #1340

Demos and clear documentation for the core services

Dask

• Docs issues: (https://docs.qhub.dev/en/latest/source/user_guide/dask_gateway.html)
  • 🔴 Monitoring of cluster stats via grafana and prometheus #39
• Dev issues:
  • [ENH] - Upgrade Dask-Gateway to v2022.04 #1296
    • Bump qhub-dask version to 0.4.3 #1341
  • [BUG] - qhub-dask hasn't been updated from 0.3.13 and is inconsistent with docs #1297
  • [bug] Unable to connect to dashboard using Dask-Gateway and Labextension #1012
    • install dask lab ext from main #1321 (Merged)

Conda-Store

• Docs Issues:
  • Adding Azure support #34
  • Switch to miniforge and remove 'default' environment. #14
    • 🔴 Establish and document that the only way to handle conda environments is by adding them to the qhub-config.yaml.

CDS Dashboards

• Docs Issues:
  • Create a method to use secrets in QHub config #13
• Dev Issues:
  • Add panel-serve as a CDS dashboard option #1070 (in review)

KBatch

• Docs Issues:
  • 🔴 Create github-action for automatically deployment qhub.dev #47
  • 🔴 Applying PR corrections from https://github.com/Quansight/qhub-ops/pull/79 #46
• Dev Issues:
  • 🔴 Upgrade kbatch/kbatch-proxy version #1311
    • Upgrade kbatch version #1335

Argo (stretch-goal)

• Docs Issues:
  • Fix github action for documentation deployment to readthedocs #48
  • Create gitter channel with integration to Quansight Slack channel #49

🔴 - deemed highest priority by eskild

[rsignell-usgs]

I'm looking forward to understanding how to use Argo Workflows!

I think the sprint should kick off with something like the CONUS404 Demo

One thing I've found awkward is explaining to users all the things you might want to do when you start up a gateway cluster, and we've an (ugly) custom routine to do this:

import os
import sys
sys.path.append(os.path.join(os.environ['HOME'],'shared','users','lib'))
import ebdpy as ebd

profile = 'nhgf-s3'
region = 'us-west-2'
endpoint = f's3.{region}.amazonaws.com'
ebd.set_credentials(profile=profile, region=region, endpoint=endpoint)
worker_max = 1
client,cluster = ebd.start_dask_cluster(profile=profile, worker_max=worker_max, 
                                      region=region, use_existing_cluster=False,
                                      adaptive_scaling=False, wait_for_cluster=False, 
                                      worker_profile='Medium Worker', 
                                      propagate_env=True)

Is there a better/simpler way?

[rsignell-usgs]

Regarding notebooks to use for the Sprint, do any of these look interesting?
https://gallery.pangeo.io/index.html

With slight modification (creating the Dask cluster), they should all work on Qhub/Nebari

I also tested cloning the Element 84 geo-notebooks repo and this Planetary Computer remote sensing notebook might be nice:
https://jupyter.qhub.esipfed.org/hub/user-redirect/lab/tree/shared/users/rsignell/repos/geo-notebooks/notebooks/odc-planetary-computer.ipynb

[trallard]

Closing as we are tracking the outstanding items elsewhere