diff --git a/.github/workflows/publish.sh b/.github/workflows/publish.sh new file mode 100644 index 000000000..e7cd5b597 --- /dev/null +++ b/.github/workflows/publish.sh @@ -0,0 +1,25 @@ +name: Publish + +on: + push: + branches: [ master ] + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + - name: Set up Python 3.8 + uses: actions/setup-python@v1 + with: + python-version: 3.8 + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install flake8 mkdocs mkdocs-material +# if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + - name: Build and publish + run: | + mkdocs gh-deploy diff --git a/.replace-hostname.sh b/.replace-hostname.sh new file mode 100755 index 000000000..cfd1a4212 --- /dev/null +++ b/.replace-hostname.sh @@ -0,0 +1,6 @@ +#!/bin/bash + +TRUE="not-telling-you-the-hostname.ca" +FAKE="example.ca" + +find . -type f -name '*.md' | tee >(xargs sed -i "s/$TRUE/$FAKE/g") diff --git a/docs/1-Experiments/Jupyter.md b/docs/1-Experiments/Jupyter.md index f1665f2b5..3f42ecb73 100644 --- a/docs/1-Experiments/Jupyter.md +++ b/docs/1-Experiments/Jupyter.md @@ -1,5 +1,66 @@ # Jupyter +## Friendly R and Python experience + +Jupyter gives you **notebooks** to write your code and make visualizations. You +can quickly iterate, visualize, and sahre your analyses. Because it's running on +a server (that you set up in the last section) you can do really big analyses on +centralized hardware! Adding as much horsepower as you need! And because it's on +the cloud, you can share it with your colleagues too. + +![Embed your visualizations](../images/jupyter_scikit.png) + + +## Get started with the examples + +When you started your server, it got loaded with a bunch of example notebooks. +Great notebooks to start with are `R/01-R-Notebook-Demo.ipynb`, or the notebooks +in `scikitlearn`. `pytorch` and `tensorflow` are great if you are familiar with +machine learning. The `mapreduce-pipeline` and `ai-pipeline` are more advanced. + +??? danger "Some notebooks only work in certain server versions" + For instance, `gdal` is only in the geomatics image. So if you use another + image then a notebook using gdal might not work. + +## Adding software + +You do not have `sudo` in Jupyter, but you can use + +```sh +conda install --use-local your_package_name +``` + +or + +```sh +pip install --user your_package_name +``` + +**Don't forget to restart your jupyter kernel afterwards, to make new packages +available.**** + +??? tip "Make sure to restart the Jupyter kernel after installing new software" + If you install software in a terminal, but your jupyter kernel was already + running, then it will not be updated. + +??? tip "Is there something that you can't install?" + If you need something installed, reach us or [open a github issue](https://github.com/StatCan/kubeflow-containers). + We can add it to the default software. + + +# Getting Data in and out of Jupyter + +You can upload and download data to/from Jupyterhub directly in the menu. There +is an upload button at the top, and you can right-click most files or folders to +download them. + +# TODO S3 browser? + + + + + + !!! warning "Phasellus posuere in sem ut cursus" Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod diff --git a/docs/1-Experiments/Kubeflow.md b/docs/1-Experiments/Kubeflow.md index 8eea9e1c9..d58c1885c 100644 --- a/docs/1-Experiments/Kubeflow.md +++ b/docs/1-Experiments/Kubeflow.md @@ -1,8 +1,8 @@ -# Experiments with Kubeflow +# Getting started with Kubeflow ## What does Kubeflow do? -Kubeflow runs your workspaces. You can have notebook servers (called Jupyter +Kubeflow runs your **workspaces**. You can have notebook servers (called Jupyter Servers), and in them you can create analyses in R and Python with interactive visuals. You can save & upload data, download it, and create shared workspaces for your team. @@ -11,9 +11,9 @@ for your team. **Let's get started!** -## Create a Server +# Create a Server -### Log into Kubeflow +## Log into Kubeflow - Log into [the azure portal](https://portal.azure.com) **using your cloud.statcan credentials**. @@ -24,7 +24,7 @@ for your team. ![Azure Portal: Choose the `@cloud.statcan.ca` address](../images/azure-login.png) -- After logging into Azure, log into [kubeflow](https://kubeflow.covid.cloud.statcan.ca) +- After logging into Azure, log into [kubeflow](https://kubeflow.example.ca) ??? failure "Why am I getting "Missing url parameter: code"?" @@ -48,7 +48,7 @@ for your team. - Then click **+ New Server** -### Configuring your server +## Configuring your server - You will get a template to create your notebook server. **Note:** the name must be lowercase letters with hypens. **No spaces, and no underscores.** @@ -65,7 +65,7 @@ for your team. - If you want to use a gpu, check if the image says **cpu** or **gpu**. -### CPU and Memory +## CPU and Memory - At the time of writing (April 21, 2020) there are two types of computers in the cluster @@ -82,7 +82,7 @@ for your team. !!! note "Use GPU machines responsibly" There are fewer GPU machines than CPU machines, so use them responsibly. -### Storing your data +## Storing your data - You'll want to create a data volume! You'll be able to save your work here, and if you shut down your server, you'll be able to just remount your old data @@ -98,7 +98,7 @@ for your team. to find your old disk, but it takes time.) -### And... Create!!! +## And... Create!!! - If you're satisfied with the settings, you can now create the server! It may take a few minutes to spin up depending on the resources you asked for. (GPUs @@ -107,5 +107,26 @@ for your team. !!! success "Your server is running" If all goes well, your server should be running!!! You will now have the option to connect, and [try out Jupyter!](/1-Experiments/Jupyter) + - +# Share your workspace + +In kubeflow every user has a **namespace**. Your namespace belongs to you, and +it's where all your resources live. If you want to collaborate with someone you +need to share a namespace. So you can do that either by sharing your own +namespace, or more preferably, **by creating a team namespace**. + +## Create a new shared namespace + +The link to create a new namespace is here +- TODO THERE IS NO LINK YET. + +## Manage contributors + +You can add or remove people from a namespace you own through the **Manage +Contributors** menu in kubeflow. + +![Contributors Menu](../images/kubeflow_contributors.png) + +!!! success "Now you and your colleagues can share access to a server!" + Now you can share a server with colleagues! Try it out! diff --git a/docs/Getting-Started.md b/docs/Getting-Started.md index 196448a3a..7dd012ded 100644 --- a/docs/Getting-Started.md +++ b/docs/Getting-Started.md @@ -61,7 +61,7 @@ You can also create custom apps. For example: !!! example "See an example NodeJS app we have" The NodeJS pictured above can be found - [here](https://covid19.covid.cloud.statcan.ca). This app is in [the Statcan + [here](https://covid19.example.ca). This app is in [the Statcan Github Repo](https://github.com/statcan/covid19). In the same way, you can submit your own custom apps and we can host them on the platform. @@ -98,7 +98,7 @@ We will give an overview of the technologies here, and in the next sections there will be a more in-depth FAQ of each of them. !!! example "Browse some datasets" - Browse some [datasets](https://datasets.covid.cloud.statcan.ca) here. These + Browse some [datasets](https://datasets.example.ca) here. These data sets are meant to store widely shared data. Either data that has been brought it, or data to be released out as a product. **As always, ensure that the data is not sensitive.** diff --git a/images/azure-login.png b/images/azure-login.png index 63c7a9cdc..d13e7aaba 100644 Binary files a/images/azure-login.png and b/images/azure-login.png differ diff --git a/images/jupyter_scikit.png b/images/jupyter_scikit.png new file mode 100644 index 000000000..ae1cd88f6 Binary files /dev/null and b/images/jupyter_scikit.png differ diff --git a/images/kubeflow_choose_an_image.png b/images/kubeflow_choose_an_image.png new file mode 100644 index 000000000..23b8103d5 Binary files /dev/null and b/images/kubeflow_choose_an_image.png differ diff --git a/images/kubeflow_contributors.png b/images/kubeflow_contributors.png new file mode 100644 index 000000000..2f43a4997 Binary files /dev/null and b/images/kubeflow_contributors.png differ diff --git a/images/kubeflow_volumes.png b/images/kubeflow_volumes.png new file mode 100644 index 000000000..64740a719 Binary files /dev/null and b/images/kubeflow_volumes.png differ diff --git a/images/missing_parameter_code.png b/images/missing_parameter_code.png new file mode 100644 index 000000000..9a7549da5 Binary files /dev/null and b/images/missing_parameter_code.png differ diff --git a/mkdocs.yml b/mkdocs.yml index 16c6d8e79..aa6d397f0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -4,7 +4,6 @@ site_description: User guide for the DAaaS Covid-19 Platform copyright: Copyright © 2020 Statistics Canada theme: name: material - custom_dir: .theme #features: # - tabs palette: