Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CD added; no custom theme #22

Merged
merged 6 commits into from
Apr 29, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/publish.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Publish

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
blairdrummond marked this conversation as resolved.
Show resolved Hide resolved

jobs:
build:

runs-on: ubuntu-latest
# strategy:
# matrix:
# python-version: [3.5, 3.6, 3.7, 3.8]
blairdrummond marked this conversation as resolved.
Show resolved Hide resolved

steps:
- uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 mkdocs mkdocs-material
# if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
# - name: Lint with flake8
# run: |
# # stop the build if there are Python syntax errors or undefined names
# flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
# flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
blairdrummond marked this conversation as resolved.
Show resolved Hide resolved
- name: Build and publish
run: |
mkdocs gh-deploy

6 changes: 6 additions & 0 deletions .replace-hostname.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash

TRUE="not-telling-you-the-hostname.ca"
FAKE="example.ca"

find . -type f -name '*.md' | tee >(xargs sed -i "s/$TRUE/$FAKE/g")
61 changes: 61 additions & 0 deletions docs/1-Experiments/Jupyter.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,66 @@
# Jupyter

## Friendly R and Python experience

Jupyter gives you **notebooks** to write your code and make visualizations. You
can quickly iterate, visualize, and sahre your analyses. Because it's running on
a server (that you set up in the last section) you can do really big analyses on
centralized hardware! Adding as much horsepower as you need! And because it's on
the cloud, you can share it with your colleagues too.

![Embed your visualizations](../images/jupyter_scikit.png)


## Get started with the examples

When you started your server, it got loaded with a bunch of example notebooks.
Great notebooks to start with are `R/01-R-Notebook-Demo.ipynb`, or the notebooks
in `scikitlearn`. `pytorch` and `tensorflow` are great if you are familiar with
machine learning. The `mapreduce-pipeline` and `ai-pipeline` are more advanced.

??? danger "Some notebooks only work in certain server versions"
For instance, `gdal` is only in the geomatics image. So if you use another
image then a notebook using gdal might not work.

## Adding software

You do not have `sudo` in Jupyter, but you can use

```sh
conda install --use-local your_package_name
```

or

```sh
pip install --user your_package_name
```

**Don't forget to restart your jupyter kernel afterwards, to make new packages
available.****

??? tip "Make sure to restart the Jupyter kernel after installing new software"
If you install software in a terminal, but your jupyter kernel was already
running, then it will not be updated.

??? tip "Is there something that you can't install?"
If you need something installed, reach us or [open a github issue](https://github.com/StatCan/kubeflow-containers).
We can add it to the default software.


# Getting Data in and out of Jupyter

You can upload and download data to/from Jupyterhub directly in the menu. There
is an upload button at the top, and you can right-click most files or folders to
download them.

# TODO S3 browser?







!!! warning "Phasellus posuere in sem ut cursus"
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla et euismod
Expand Down
41 changes: 31 additions & 10 deletions docs/1-Experiments/Kubeflow.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Experiments with Kubeflow
# Getting started with Kubeflow

## What does Kubeflow do?

Kubeflow runs your workspaces. You can have notebook servers (called Jupyter
Kubeflow runs your **workspaces**. You can have notebook servers (called Jupyter
Servers), and in them you can create analyses in R and Python with interactive
visuals. You can save & upload data, download it, and create shared workspaces
for your team.
Expand All @@ -11,9 +11,9 @@ for your team.

**Let's get started!**

## Create a Server
# Create a Server

### Log into Kubeflow
## Log into Kubeflow

- Log into [the azure portal](https://portal.azure.com) **using your
cloud.statcan credentials**.
Expand All @@ -24,7 +24,7 @@ for your team.
![Azure Portal: Choose the `@cloud.statcan.ca` address](../images/azure-login.png)


- After logging into Azure, log into [kubeflow](https://kubeflow.covid.cloud.statcan.ca)
- After logging into Azure, log into [kubeflow](https://kubeflow.example.ca)


??? failure "Why am I getting "Missing url parameter: code"?"
Expand All @@ -48,7 +48,7 @@ for your team.

- Then click **+ New Server**

### Configuring your server
## Configuring your server

- You will get a template to create your notebook server.
**Note:** the name must be lowercase letters with hypens. **No spaces, and no underscores.**
Expand All @@ -65,7 +65,7 @@ for your team.
- If you want to use a gpu, check if the image says **cpu** or **gpu**.


### CPU and Memory
## CPU and Memory

- At the time of writing (April 21, 2020) there are two types of computers in
the cluster
Expand All @@ -82,7 +82,7 @@ for your team.
!!! note "Use GPU machines responsibly"
There are fewer GPU machines than CPU machines, so use them responsibly.

### Storing your data
## Storing your data

- You'll want to create a data volume! You'll be able to save your work here,
and if you shut down your server, you'll be able to just remount your old data
Expand All @@ -98,7 +98,7 @@ for your team.
to find your old disk, but it takes time.)


### And... Create!!!
## And... Create!!!

- If you're satisfied with the settings, you can now create the server! It may
take a few minutes to spin up depending on the resources you asked for. (GPUs
Expand All @@ -107,5 +107,26 @@ for your team.
!!! success "Your server is running"
If all goes well, your server should be running!!! You will now have the
option to connect, and [try out Jupyter!](/1-Experiments/Jupyter)



# Share your workspace

In kubeflow every user has a **namespace**. Your namespace belongs to you, and
it's where all your resources live. If you want to collaborate with someone you
need to share a namespace. So you can do that either by sharing your own
namespace, or more preferably, **by creating a team namespace**.

## Create a new shared namespace

The link to create a new namespace is here
- TODO THERE IS NO LINK YET.

## Manage contributors

You can add or remove people from a namespace you own through the **Manage
Contributors** menu in kubeflow.

![Contributors Menu](../images/kubeflow_contributors.png)

!!! success "Now you and your colleagues can share access to a server!"
Now you can share a server with colleagues! Try it out!
4 changes: 2 additions & 2 deletions docs/Getting-Started.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ You can also create custom apps. For example:

!!! example "See an example NodeJS app we have"
The NodeJS pictured above can be found
[here](https://covid19.covid.cloud.statcan.ca). This app is in [the Statcan
[here](https://covid19.example.ca). This app is in [the Statcan
Github Repo](https://github.com/statcan/covid19). In the same way, you can
submit your own custom apps and we can host them on the platform.

Expand Down Expand Up @@ -98,7 +98,7 @@ We will give an overview of the technologies here, and in the next sections
there will be a more in-depth FAQ of each of them.

!!! example "Browse some datasets"
Browse some [datasets](https://datasets.covid.cloud.statcan.ca) here. These
Browse some [datasets](https://datasets.example.ca) here. These
data sets are meant to store widely shared data. Either data that has been
brought it, or data to be released out as a product. **As always, ensure
that the data is not sensitive.**
Expand Down
Binary file modified images/azure-login.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/jupyter_scikit.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/kubeflow_choose_an_image.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/kubeflow_contributors.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/kubeflow_volumes.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/missing_parameter_code.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 0 additions & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ site_description: User guide for the DAaaS Covid-19 Platform
copyright: Copyright © 2020 Statistics Canada
theme:
name: material
custom_dir: .theme
#features:
# - tabs
palette:
Expand Down