Dciborow/aks (#69)

* Initial commit * Adding First Version of Template (#1) * Initial CODE_OF_CONDUCT.md commit * Initial README.md commit * Initial SECURITY.md commit * Initial LICENSE commit * Adding first template files * Adding first template files * Adding first template files * Update azure-pipelines.yml for Azure Pipelines * Adding first template files * Adding first template files * Adding first template files * Adding first template files * Adding first template files * Adding first template files * adding test notebooks * Adding first template files * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Update README.md * updates Co-authored-by: Microsoft Open Source <microsoftopensource@users.noreply.github.com> * Initial Merge into Master (#2) * Initial CODE_OF_CONDUCT.md commit * Initial README.md commit * Initial SECURITY.md commit * Initial LICENSE commit * Adding first template files * Adding first template files * Adding first template files * Update azure-pipelines.yml for Azure Pipelines * Adding first template files * Adding first template files * Adding first template files * Adding first template files * Adding first template files * Adding first template files * adding test notebooks * Adding first template files * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Update README.md * updates * Update environment.yml * Update environment.yml * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration * Adding Pylint configuration Co-authored-by: Microsoft Open Source <microsoftopensource@users.noreply.github.com> * Update sample_workspace_conf.yml * Adding Test Method * Adding Test Method * Adding Test Method * Adding Test Method * Adding Pylint configuration * Update environment.yml * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update azure-pipelines.yml for Azure Pipelines * Updating to new configuration (#3) * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Adding Realtime Deployment Wrappers * Update 02_TrainOnLocal.ipynb * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Trying to get test 2 to pass * Trying to get test 2 to pass * Working on test 4 * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Create deployment_params.yml * Update azure-pipelines-v2.yml * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py * Merge remote-tracking branch 'template/master' # Conflicts: # .ci/azure-pipelines.yml # .gitignore # LICENSE # README.md # environment.yml # notebooks/00_AMLConfiguration.ipynb # notebooks/__init__.py # sample_workspace_conf.yml # tests/test_notebooks.py Co-authored-by: microsoft-github-operations[bot] <55726097+microsoft-github-operations[bot]@users.noreply.github.com> Co-authored-by: Microsoft Open Source <microsoftopensource@users.noreply.github.com>
microsoft · Feb 28, 2020 · ddc493e · ddc493e
1 parent 84d4490
commit ddc493e
Show file tree

Hide file tree

Showing 16 changed files with 1,137 additions and 1,673 deletions.
diff --git a/.ci/azure-pipelines-v2.yml b/.ci/azure-pipelines-v2.yml
@@ -28,13 +28,20 @@ pr:
     include:
     - master
 
+variables:
+- template: ./vars/deployment_params.yml
+
 stages:
-- template: .ci/stages/deploy_notebooks_stages_v2.yml@aitemplates
+- template: .ci/stages/deploy_notebooks_stages_v5.yml@aitemplates
   parameters:
     Agent: $(Agent_Name)
     jobDisplayName: az-ml-realtime-score
-    DefaultWorkingDirectory: $(System.DefaultWorkingDirectory)
-    workload_vars: ../vars/az-ml-realtime-score.yml
+    TridentWorkloadTypeShort: ${{ variables.TridentWorkloadTypeShort }}
+    DeployLocation: ${{ variables.DeployLocation }}
+    ProjectLocation: ${{ variables.ProjectLocation }}
+    conda: ${{ variables.conda }}
+    post_cleanup: false
+
     flighting_release: false
     flighting_preview: false
     flighting_master: false  
diff --git a/.ci/vars/deployment_params.yml b/.ci/vars/deployment_params.yml
@@ -0,0 +1,5 @@
+variables:
+  TridentWorkloadTypeShort: azmlrts
+  DeployLocation: westus
+  ProjectLocation: "."
+  conda: az-ml-realtime-score
diff --git a/.gitignore b/.gitignore
@@ -4,6 +4,7 @@ workspace_conf.yml
 *.output_ipynb
 .azureml
 pylint-results.xml
+project.yml
 .idea
 score.py
 
@@ -14,16 +15,13 @@ assets/
 scripts/assets/
 .amlignore
 scripts/.amlignore
-__pycache__/
 scripts/__pycache__/
 
 # Environments
 .env
 
-# Jupyter Notebook
-.ipynb_checkpoints
-
 # Byte-compiled / optimized / DLL files
+__pycache__/
 *.py[cod]
 *$py.class
 
@@ -151,3 +149,4 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+/project.yml
diff --git a/.pylintrc b/.pylintrc
@@ -60,7 +60,12 @@ confidence=
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use "--disable=all --enable=classes
 # --disable=W".
-disable=missing-module-docstring,
+disable=global-variable-undefined,
+        global-statement,
+        too-many-arguments,
+        too-many-function-args,
+        pointless-statement,
+        missing-module-docstring,
         trailing-whitespace,
         fixme,
         print-statement,
@@ -302,7 +307,7 @@ variable-naming-style=snake_case
 expected-line-ending-format=
 
 # Regexp for a line that is allowed to be longer than the limit.
-ignore-long-lines=^\s*(# )?<?https?://\S+>?$
+ignore-long-lines=^\s*(# )?<?https?:\S+>?$|^\s*get_ipython\S+
 
 # Number of spaces of indent required inside a hanging or continued line.
 indent-after-paren=4
@@ -454,7 +459,7 @@ signature-mutators=
 
 # List of additional names supposed to be defined in builtins. Remember that
 # you should avoid defining new builtins when possible.
-additional-builtins=
+additional-builtins=get_ipython
 
 # Tells whether unused global variables should be treated as a violation.
 allow-global-unused-variables=yes

diff --git a/notebooks/00_AMLConfiguration.ipynb b/notebooks/00_AMLConfiguration.ipynb
@@ -18,92 +18,19 @@
     "## Prerequisites\n",
     "\n",
     "If you have already completed the prerequisites and selected the correct Kernel for this notebook, the AML Python SDK \n",
-    "is already installed. Let's load the imports and check the AML SDK version."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "import json\n",
+    "is already installed. Let's load the imports and check the AML SDK version.\n",
     "\n",
-    "import azureml.core\n",
-    "from azure_utils.machine_learning.utils import load_configuration, get_or_create_workspace\n",
-    "\n",
-    "print(\"AML SDK Version:\", azureml.core.VERSION)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
     "## Set up your Azure Machine Learning workspace\n",
     "## Load Configurations from file\n",
     "\n",
-    "Configurations are loaded from a file, to prevent accident commits of Azure secerts into source control.\n",
-    "This file name is included in the .gitignore to also prevent accident commits. A template file is included that should\n",
-    "be copied, and each parameter filled in."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "cfg = load_configuration(\"../workspace_conf.yml\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "pycharm": {
-     "name": "#%% md\n"
-    }
-   },
-   "source": [
-    "## Load Configurations into Notebook.\n",
+    "Configurations are loaded by default from a file `project.yml`, to prevent accident commits of Azure secrets into \n",
+    "source control. This file name is included in the `.gitignore` to also prevent accident commits. A template file \n",
+    "is included that should be copied, and each parameter filled in.\n",
+    "\n",
+    "If the file is not present, and UI Prompt will pop up to insert configurations, and save to the file.\n",
     "\n",
-    "The following cell loads the configurations from the local file, into the notebook memory. The following cell is also\n",
-    "marked as a parameter cell. When using this notebook with [papermill](https://github.com/nteract/papermill), these\n",
-    "parameters can be override. See the tests for examples."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    },
-    "tags": [
-     "parameters"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "subscription_id = cfg['subscription_id']\n",
-    "resource_group = cfg['resource_group']\n",
-    "workspace_name = cfg['workspace_name']\n",
-    "workspace_region = cfg['workspace_region']"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
     "## Create the workspace\n",
-    "This cell will create an AML workspace for you in a subscription, provided you have the correct permissions.\n",
+    "This cell will also create an AML workspace for you in a subscription, provided you have the correct permissions.\n",
     "\n",
     "This will fail when:\n",
     "1. You do not have permission to create a workspace in the resource group\n",
@@ -112,32 +39,9 @@
     "subscription\n",
     "\n",
     "If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to \n",
-    "provision the required resources. If this cell succeeds, you're done configuring AML!\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "ws = get_or_create_workspace(workspace_name, subscription_id, resource_group, workspace_region)\n",
-    "ws_json = ws.get_details()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "pycharm": {
-     "name": "#%% md\n"
-    }
-   },
-   "source": [
-    "Let's check the details of the workspace."
+    "provision the required resources. If this cell succeeds, you're done configuring AML!\n",
+    "\n",
+    "After creation we will check the details of the workspace."
    ]
   },
   {
@@ -150,18 +54,17 @@
    },
    "outputs": [],
    "source": [
-    "print(json.dumps(ws_json, indent=2))"
+    "from azure_utils.machine_learning.utils import get_or_create_workspace_from_file\n",
+    "\n",
+    "ws = get_or_create_workspace_from_file()"
    ]
   },
   {
    "cell_type": "markdown",
-   "metadata": {
-    "pycharm": {
-     "name": "#%% md\n"
-    }
-   },
+   "metadata": {},
    "source": [
-    "You are now ready to move on to the [AutoML Local](01_DataPrep.ipynb) notebook."
+    "We can now move on to [Data Preperation](01_DataPrep.ipynb) notebook to train our model using Azure Machine \n",
+    "Learning."
    ]
   }
  ],
@@ -172,25 +75,13 @@
    "language": "python",
    "name": "az-ml-realtime-score"
   },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.6.2"
-  },
   "pycharm": {
    "stem_cell": {
     "cell_type": "raw",
-    "source": [],
     "metadata": {
      "collapsed": false
-    }
+    },
+    "source": []
    }
   }
  },