Skip to content

Commit e6f0f61

Browse files
luotigerlsxJeffwan
luotigerlsx
authored and
Jeffwan
committed
Add step by step tutorial using mnist as use case (kubeflow#2716)
* add step by step tutorial using mnist as use case * fix mnist typo and change job submit default * add owner file; modify setup and readme about ui deployment statement * Refine notebooks and readme to incorporate reviewers comment * fine tune of the documentation
1 parent 7a702ce commit e6f0f61

7 files changed

+2723
-0
lines changed

‎samples/tutorials/mnist/00_Kubeflow_Cluster_Setup.ipynb

+335
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,335 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "code",
5+
"execution_count": null,
6+
"metadata": {},
7+
"outputs": [],
8+
"source": [
9+
"# Copyright 2019 Google Inc. All Rights Reserved.\n",
10+
"#\n",
11+
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
12+
"# you may not use this file except in compliance with the License.\n",
13+
"# You may obtain a copy of the License at\n",
14+
"#\n",
15+
"# http://www.apache.org/licenses/LICENSE-2.0\n",
16+
"#\n",
17+
"# Unless required by applicable law or agreed to in writing, software\n",
18+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
19+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
20+
"# See the License for the specific language governing permissions and\n",
21+
"# limitations under the License.\n",
22+
"# =============================================================================="
23+
]
24+
},
25+
{
26+
"cell_type": "markdown",
27+
"metadata": {},
28+
"source": [
29+
"# Deploying a Kubeflow Cluster on Google Cloud Platform (GCP)\n",
30+
"This notebook provides instructions for setting up a Kubeflow cluster on GCP using the command-line interface (CLI). For additional help, see the guide to [deploying Kubeflow using the CLI](https://www.kubeflow.org/docs/gke/deploy/deploy-cli/)."
31+
]
32+
},
33+
{
34+
"cell_type": "markdown",
35+
"metadata": {},
36+
"source": [
37+
"## Prerequisites\n",
38+
"- You have a [GCP project setup](https://www.kubeflow.org/docs/gke/deploy/project-setup/) for your Kubeflow Deployment with you having the [owner role](https://cloud.google.com/iam/docs/understanding-roles#primitive_role_definitions) for the project and with the following APIs enabled:\n",
39+
" - [Compute Engine API](https://pantheon.corp.google.com/apis/library/compute.googleapis.com)\n",
40+
" - [Kubernetes Engine API](https://pantheon.corp.google.com/apis/library/container.googleapis.com)\n",
41+
" - [Identity and Access Management(IAM) API](https://pantheon.corp.google.com/apis/library/iam.googleapis.com)\n",
42+
" - [Deployment Manager API](https://pantheon.corp.google.com/apis/library/deploymentmanager.googleapis.com)\n",
43+
" - [Cloud Resource Manager API](https://pantheon.corp.google.com/apis/library/cloudresourcemanager.googleapis.com)\n",
44+
" - [AI Platform Training & Prediction API](https://pantheon.corp.google.com/apis/library/ml.googleapis.com)\n",
45+
"- You have set up [OAuth for Cloud IAP](https://www.kubeflow.org/docs/gke/deploy/oauth-setup/)\n",
46+
"- You have installed and setup [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n",
47+
"- You have installed [gcloud-sdk](https://cloud.google.com/sdk/)"
48+
]
49+
},
50+
{
51+
"cell_type": "markdown",
52+
"metadata": {},
53+
"source": [
54+
"## Running Environment\n",
55+
"\n",
56+
"**This notebook helps in creating the Kubeflow cluster on GCP. You must run this notebook in an environment with Cloud SDK installed, such as Cloud Shell. Learn more about [installing Cloud SDK](https://cloud.google.com/sdk/docs/).**"
57+
]
58+
},
59+
{
60+
"cell_type": "markdown",
61+
"metadata": {},
62+
"source": [
63+
"## Setting up a Kubeflow cluster\n",
64+
"1. Download kfctl\n",
65+
"2. Setup environment variables\n",
66+
"3. Create dedicated service account for deployment\n",
67+
"4. Deploy Kubefow\n",
68+
"5. Install Kubeflow Pipelines SDK\n",
69+
"6. Sanity check\n",
70+
"\n",
71+
"`NOTE` : It is also possible to deploy the kubeflow cluster though [UI](https://www.kubeflow.org/docs/gke/deploy/deploy-ui/)"
72+
]
73+
},
74+
{
75+
"cell_type": "markdown",
76+
"metadata": {},
77+
"source": [
78+
"### Create a working directory\n",
79+
"Create a new working directory in your current directory. The default name is **kubeflow**, but you can change the name."
80+
]
81+
},
82+
{
83+
"cell_type": "code",
84+
"execution_count": null,
85+
"metadata": {},
86+
"outputs": [],
87+
"source": [
88+
"work_directory_name = 'kubeflow'\n",
89+
"\n",
90+
"! mkdir -p $work_directory_name\n",
91+
"\n",
92+
"%cd $work_directory_name"
93+
]
94+
},
95+
{
96+
"cell_type": "markdown",
97+
"metadata": {},
98+
"source": [
99+
"### Download kfctl\n",
100+
"Download kfctl to your working directory. The default version used is v0.7.0, but you can find the latest release [here](https://github.com/kubeflow/kubeflow/releases)."
101+
]
102+
},
103+
{
104+
"cell_type": "code",
105+
"execution_count": null,
106+
"metadata": {},
107+
"outputs": [],
108+
"source": [
109+
"## Download kfctl v0.7.0\n",
110+
"! curl -LO https://github.com/kubeflow/kubeflow/releases/download/v0.7.0/kfctl_v0.7.0_linux.tar.gz\n",
111+
" \n",
112+
"## Unpack the tar ball\n",
113+
"! tar -xvf kfctl_v0.7.0_linux.tar.gz"
114+
]
115+
},
116+
{
117+
"cell_type": "code",
118+
"execution_count": null,
119+
"metadata": {},
120+
"outputs": [],
121+
"source": [
122+
"## Create user credentials\n",
123+
"! gcloud auth application-default login"
124+
]
125+
},
126+
{
127+
"cell_type": "markdown",
128+
"metadata": {},
129+
"source": [
130+
"### Set up environment variables\n",
131+
"Set up environment variables to use while installing Kubeflow. Replace variable placeholders (for example, `<VARIABLE NAME>`) with the correct values for your environment."
132+
]
133+
},
134+
{
135+
"cell_type": "code",
136+
"execution_count": null,
137+
"metadata": {},
138+
"outputs": [],
139+
"source": [
140+
"# Set your GCP project ID and the zone where you want to create the Kubeflow deployment\n",
141+
"%env PROJECT=<ADD GCP PROJECT HERE>\n",
142+
"%env ZONE=<ADD GCP ZONE TO LAUNCH KUBEFLOW CLUSTER HERE>\n",
143+
"\n",
144+
"# google cloud storage bucket\n",
145+
"%env GCP_BUCKET=gs://<ADD STORAGE LOCATION HERE>\n",
146+
"\n",
147+
"# Use the following kfctl configuration file for authentication with \n",
148+
"# Cloud IAP (recommended):\n",
149+
"uri = \"https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml\"\n",
150+
"uri = uri.strip()\n",
151+
"%env CONFIG_URI=$uri\n",
152+
"\n",
153+
"# For using Cloud IAP for authentication, create environment variables\n",
154+
"# from the OAuth client ID and secret that you obtained earlier:\n",
155+
"%env CLIENT_ID=<ADD OAuth CLIENT ID HERE>\n",
156+
"%env CLIENT_SECRET=<ADD OAuth CLIENT SECRET HERE>\n",
157+
"\n",
158+
"# Set KF_NAME to the name of your Kubeflow deployment. You also use this\n",
159+
"# value as directory name when creating your configuration directory. \n",
160+
"# For example, your deployment name can be 'my-kubeflow' or 'kf-test'.\n",
161+
"%env KF_NAME=<ADD KUBEFLOW DEPLOYMENT NAME HERE>\n",
162+
"\n",
163+
"# Set up name of the service account that should be created and used\n",
164+
"# while creating the Kubeflow cluster\n",
165+
"%env SA_NAME=<ADD SERVICE ACCOUNT NAME TO BE CREATED HERE>"
166+
]
167+
},
168+
{
169+
"cell_type": "markdown",
170+
"metadata": {},
171+
"source": [
172+
"Configure gcloud and add kfctl to your path."
173+
]
174+
},
175+
{
176+
"cell_type": "code",
177+
"execution_count": null,
178+
"metadata": {},
179+
"outputs": [],
180+
"source": [
181+
"! gcloud config set project ${PROJECT}\n",
182+
"\n",
183+
"! gcloud config set compute/zone ${ZONE}\n",
184+
"\n",
185+
"\n",
186+
"# Set the path to the base directory where you want to store one or more \n",
187+
"# Kubeflow deployments. For example, /opt/.\n",
188+
"# Here we use the current working directory as the base directory\n",
189+
"# Then set the Kubeflow application directory for this deployment.\n",
190+
"\n",
191+
"import os\n",
192+
"base = os.getcwd()\n",
193+
"%env BASE_DIR=$base\n",
194+
"\n",
195+
"kf_dir = os.getenv('BASE_DIR') + \"/\" + os.getenv('KF_NAME')\n",
196+
"%env KF_DIR=$kf_dir\n",
197+
"\n",
198+
"# The following command is optional. It adds the kfctl binary to your path.\n",
199+
"# If you don't add kfctl to your path, you must use the full path\n",
200+
"# each time you run kfctl. In this example, the kfctl file is present in\n",
201+
"# the current directory\n",
202+
"new_path = os.getenv('PATH') + \":\" + os.getenv('BASE_DIR')\n",
203+
"%env PATH=$new_path"
204+
]
205+
},
206+
{
207+
"cell_type": "markdown",
208+
"metadata": {},
209+
"source": [
210+
"### Create service account\n"
211+
]
212+
},
213+
{
214+
"cell_type": "code",
215+
"execution_count": null,
216+
"metadata": {},
217+
"outputs": [],
218+
"source": [
219+
"! gcloud iam service-accounts create ${SA_NAME}\n",
220+
"! gcloud projects add-iam-policy-binding ${PROJECT} \\\n",
221+
" --member serviceAccount:${SA_NAME}@${PROJECT}.iam.gserviceaccount.com \\\n",
222+
" --role 'roles/owner'\n",
223+
"! gcloud iam service-accounts keys create key.json \\\n",
224+
" --iam-account ${SA_NAME}@${PROJECT}.iam.gserviceaccount.com"
225+
]
226+
},
227+
{
228+
"cell_type": "markdown",
229+
"metadata": {},
230+
"source": [
231+
"### Set GOOGLE_APPLICATION_CREDENTIALS"
232+
]
233+
},
234+
{
235+
"cell_type": "code",
236+
"execution_count": null,
237+
"metadata": {},
238+
"outputs": [],
239+
"source": [
240+
"key_path = os.getenv('BASE_DIR') + \"/\" + 'key.json'\n",
241+
"%env GOOGLE_APPLICATION_CREDENTIALS=$key_path"
242+
]
243+
},
244+
{
245+
"cell_type": "markdown",
246+
"metadata": {},
247+
"source": [
248+
"### Setup and deploy Kubeflow"
249+
]
250+
},
251+
{
252+
"cell_type": "code",
253+
"execution_count": null,
254+
"metadata": {},
255+
"outputs": [],
256+
"source": [
257+
"! mkdir -p ${KF_DIR}\n",
258+
"%cd $kf_dir\n",
259+
"! kfctl apply -V -f ${CONFIG_URI}"
260+
]
261+
},
262+
{
263+
"cell_type": "markdown",
264+
"metadata": {},
265+
"source": [
266+
"### Install Kubeflow Pipelines SDK"
267+
]
268+
},
269+
{
270+
"cell_type": "code",
271+
"execution_count": null,
272+
"metadata": {},
273+
"outputs": [],
274+
"source": [
275+
"%%capture\n",
276+
"\n",
277+
"# Install the SDK (Uncomment the code if the SDK is not installed before)\n",
278+
"! pip3 install 'kfp>=0.1.36' --quiet --user"
279+
]
280+
},
281+
{
282+
"cell_type": "markdown",
283+
"metadata": {},
284+
"source": [
285+
"### Sanity Check: Check the ingress created"
286+
]
287+
},
288+
{
289+
"cell_type": "code",
290+
"execution_count": null,
291+
"metadata": {},
292+
"outputs": [],
293+
"source": [
294+
"! kubectl -n istio-system describe ingress"
295+
]
296+
},
297+
{
298+
"cell_type": "markdown",
299+
"metadata": {},
300+
"source": [
301+
"Access the Kubeflow cluster at **`https://<KF_NAME>.endpoints.<gcp_project_id>.cloud.goog/`**\n",
302+
"\n",
303+
"Note that it may take up to 15-20 mins for the above url to be functional."
304+
]
305+
},
306+
{
307+
"cell_type": "code",
308+
"execution_count": null,
309+
"metadata": {},
310+
"outputs": [],
311+
"source": []
312+
}
313+
],
314+
"metadata": {
315+
"kernelspec": {
316+
"display_name": "Python 3",
317+
"language": "python",
318+
"name": "python3"
319+
},
320+
"language_info": {
321+
"codemirror_mode": {
322+
"name": "ipython",
323+
"version": 3
324+
},
325+
"file_extension": ".py",
326+
"mimetype": "text/x-python",
327+
"name": "python",
328+
"nbconvert_exporter": "python",
329+
"pygments_lexer": "ipython3",
330+
"version": "3.7.3"
331+
}
332+
},
333+
"nbformat": 4,
334+
"nbformat_minor": 2
335+
}

0 commit comments

Comments
 (0)