courses/machine_learning/deepdive/09_sequence/run-on-cloud-ml-engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Covert notebook to raw python\n",
    "To run on ML-Engine, we strip all cells tagged \"display\" to remove the dependency on matplotlib.\n",
    "The remote server/cluster does not have a display anyway, just logs.\n",
    "(To view/edit tags on notebook cells: View>Cell Toolbar>Tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert notebook to raw python format and remove all cells tagged \"display\"\n",
    "NOTEBOOK='02_RNN_generator_temperatures_solution.ipynb'\n",
    "jupyter nbconvert tutorial/${NOTEBOOK} \\\n",
    "    --to python --TagRemovePreprocessor.remove_cell_tags={\\\"display\\\"} \\\n",
    "    --output task.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*** Windows users, please copy-paste this command into your command prompt: ***\n",
    "```bash\n",
    "jupyter nbconvert \"tutorial/02_RNN_generator_temperatures_solution.ipynb\" --to python --output task.py --TagRemovePreprocessor.remove_cell_tags={\\\"display\\\"} \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To run on ML-Engine\n",
    "If you are using your own GCP account you have to first:\n",
    "1. Create a Google Cloud Platform project\n",
    "1. Enable billing\n",
    "1. Create a Google Cloud Storage bucket (put in in region us-central1)\n",
    "1. Enable the necessary APIs and request the necessary quotas\n",
    "\n",
    "If you are using a lab account through Qwiklabs (Available soon):\n",
    "1. Please [register your email address here](https://docs.google.com/forms/d/e/1FAIpQLScDruivAynhrL9XMyEozLZRRCuMLg-X0BFC3ct0VqHs_sW1cg/viewform?usp=sf_link) so that we can white-list you on Qwiklabs.\n",
    "1. Go to Qwiklabs for the last part of the workshop. Qwiklabs will provision a free lab account on Google Cloud Platform with a GPU quota for you:\n",
    "[https://events.qwiklabs.com/classrooms/&lt;available soon&gt;](https://events.qwiklabs.com/classrooms/XXXX)\n",
    "1. Create a Google Cloud Storage bucket (put in in region us-central1)\n",
    "\n",
    "And fill in the info in the variables below.\n",
    "You can try running on a GPU by using --scale-tier=BASIC_GPU or a CPU using --scale-tier=BASIC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET='ml1-demo-martin'\n",
    "PROJECT='cloudml-demo-martin'\n",
    "\n",
    "REGION='us-central1'\n",
    "JOBNAME=sines_$(date -u +%y%m%d_%H%M%S)\n",
    "OUTDIR=\"gs://${BUCKET}/sinejobs/${JOBNAME}\"\n",
    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    " --region=$REGION \\\n",
    " --module-name=tutorial.task \\\n",
    " --package-path=tutorial \\\n",
    " --job-dir=$OUTDIR \\\n",
    " --scale-tier=BASIC_GPU \\\n",
    " --runtime-version=1.6 \\\n",
    " -- \\\n",
    " --data-dir=\"gs://good-temperatures-public\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*** Windows users, please copy-paste this command into your command prompt (replace &lt;PROJECT&gt; and &lt;BUCKET&gt; with your own values and adjust the job name job001 if needed): ***\n",
    "```bash\n",
    "gcloud ml-engine jobs submit training job001 --region=\"us-central1\" --project=<PROJECT> --module-name=\"tutorial.task\" --package-path=\"tutorial\" --job-dir=\"gs://<BUCKET>/sinejobs/job001\" --scale-tier=BASIC_GPU --runtime-version=1.6 -- --data-dir=\"gs://good-temperatures-public\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To test-run locally as if the code was running on ML-Engine\n",
    "Warning: this will use the \"python\" command to run (usually mapped to python 2 on mac, same as ML-Engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gcloud ml-engine local train \\\n",
    " --module-name=tutorial.task \\\n",
    " --package-path=tutorial \\\n",
    " --job-dir=\"checkpoints\" \\\n",
    " -- \\\n",
    " --data-dir=\"gs://good-temperatures-public\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2018 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}