machine_learning_bishop.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "machine_learning_bishop.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dauparas/tensorflow_examples/blob/master/machine_learning_bishop.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "-U9karDpizhr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# This is a notebook for reading \"Pattern Recognition and Machine Learning\" by Christopher M. Bishop."
      ]
    },
    {
      "metadata": {
        "id": "N5CP5wAwjGDz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Chapter 1: Introduction\n",
        "\n",
        "**Pattern recognition is fundamental to mathematics and science.** For example given a sequence of numbers $1, 3, 5, 7,...$ one can easily spot that the next number is generated by adding $2$ to the previous number. One could also see a pattern in an oscillating simple pendulum. The period of the pendulum for small oscillations is given by $T = 2\\pi \\sqrt{L/g}$ where $L$ is the length of the pendulum and $g$ acceleration due to gravity. Spotting patterns allows us to predict the future!\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "The field of patern recognition is concerned with the automatic discovery of regularities in the data through the use of computer algorithms. Generalization is a central goal in pattern recognition.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "Supervised learning: classification, regression.\n",
        "\n",
        "Unsupervised learning: clusterring, density estimation, visualization.\n",
        "\n",
        "Reinforcement learning.\n",
        "\n",
        "# 1.1 Example: Polynomial Curve Fitting"
      ]
    },
    {
      "metadata": {
        "id": "iSx8v6Nwn_PO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Import libraries\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "plt.style.use('ggplot')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JjTYjzvFrr0-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x = np.linspace(0, 1, 11)\n",
        "N = x.shape[0]\n",
        "y = np.sin(2*np.pi*x) + 0.2*np.random.randn(N)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zZuawz0ksNj7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "18a3723e-1e16-4f64-dc4d-cf1e7a6e0798"
      },
      "cell_type": "code",
      "source": [
        "plt.scatter(x, y);"
      ],
      "execution_count": 225,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFVlJREFUeJzt3VtsFOf9//HPzJqDV2bt9ZHsj1QO\nYFxoYXEEDqmq9oKqiiCqcuFWaZNKVKyqiNKq/Tn/C6gpVMRNK2JVlkpatUhYSIGbSo2qSqlUJUVK\nLxwkLNkmNMSNY3FwfcD22MQH8rf9/C42OBjWa7NH7+P362bZfczM94uTzzx+ZjzjGGOMAADWcbNd\nAAAgPQh4ALAUAQ8AliLgAcBSBDwAWIqABwBL5WW7gPv19vYmvY1QKJSS7eQK+rXXSupVot9ktrMQ\nZvAAYCkCHgAsRcADgKUIeACwFAEPAJYi4AHAUsvqMkk8GuMN6e5wv4ybJ6eoJNvlAFhmCPgcZKYm\nNXumSerp0sDYqBQolCqr5Ebq5azNz3Z5AJYJlmhy0OyZJqn9kjQ6IpnZ6Gv7pejnAPAZAj7HGG9I\n6umKPdjTFR0HABHwuWegTxrzYo/dGZUG+zNbD4Bli4DPNeXrpUBR7LF1hVJZRWbrAbBsEfA5xikq\nkSqrYg9WVnE1DYA5BHwOciP1UrhWKgxKrht9DddGPweAz3CZZA5y1ubLd7hBxhtSqZnRbcfHzB3A\nQwj4HOYUlWhNKCRnBd1DG8DSsUQDAJZKagZ//fp1nTp1Svv379czzzwzb6yjo0MXLlyQ67qqqalR\nXV1dUoUCAB5NwjP4qakpnT17Vl/+8pdjjp89e1b19fU6efKkOjo6dPPmzYSLBAA8uoQDftWqVTpy\n5IiCweBDY/39/SooKFBpaencDL6zszOpQgEAjybhJRqfzyefzxdzzPM8BQKBufeFhYXq6+tbdJvx\nHh77KFK1nVxBv/ZaSb1K9JtqGbmKxhizpK9L1RPGeTK7vVZSvyupV4l+k9nOQtJyFU0wGJTnfX6/\nlOHhYRUXF6djVwCABaQl4MvLyzU5OamBgQHNzMyora1NO3bsSMeukAXGG5L58H3uXAkscwkv0XR3\nd+vcuXMaHByUz+dTa2urdu3apfLyctXW1ioSiai5uVmS9PTTT6+4tTUb3f+gEY150Zue8aARYNly\nzFIXyDOANfhHl8l+Z373SvRBIw8K18p3uCEjNayk7+9K6lWi32S2sxB+kxVLwoNGgNxDwGNpeNAI\nkHMIeCwNDxoBcg4BjyXhQSNA7iHgsWQ8aATILdwPHkt2/4NGNNgvlVUwcweWMQIej8wpKpEIdmDZ\nY4kGACxFwAOApQh4ALAUAQ8AliLgAcBSBDwAWIqABwBLEfAAYCkCHgAsRcADgKUIeACwFAEPAJYi\n4AHAUgQ8AFiKgAcASxHwAGApAh4ALEXAA4ClCHgAsBQBDwCWIuABwFIEPABYioBPAeMNyXz4vow3\nlO1SAGBOXrYLyGVmalKzZ5qkni5pzJMCRVJlldxIvZy1+dkuD8AKxww+CbNnmqT2S9LoiGRM9LX9\nUvRzAMgyAj5BxhuKztxj6eliuQZA1hHwiRroiy7LxHJnVBrsz2w9APAAAj5R5euja+6xrCuUyioy\nWw8APICAT5BTVCJVVsUerKyKjgNAFhHwSXAj9VK4VioMSq4bfQ3XRj8HgCxL+DLJlpYWdXV1yXEc\nHThwQJs3b54b+9GPfqSSkhK5bvT48ZOf/ETFxcXJV7vMOGvz5TvcED2hOtgvlVUwcwewbCQU8Fev\nXlVfX58aGxt18+ZN/f73v1djY+O8rzl69KjWrl2bkiKXO6eoRCLYASwzCS3RdHZ2avfu3ZKkDRs2\naHx8XBMTEyktDACQnIRm8J7naePGjXPvA4GAPM+T3++f++yPf/yjBgcH9cUvflHf+9735DjOotsN\nhUKJlJO27eQK+rXXSupVot9US8mtCowx895/5zvf0c6dO1VQUKBTp07pvffe0549exbdTm9vb9K1\nhEKhlGwnV9CvvVZSrxL9JrOdhSS0RBMMBuV5n/+Sz8jIiILB4Nz7r3/96yosLJTP51NNTY2uX7+e\nyG4AAElIKODD4bBaW1slSd3d3QoGg8rPj95ca2JiQo2NjZqenpYUPSH7+OOPp6hcAMBSJbREU11d\nrY0bN6qhoUGO4+jgwYO6ePGi/H6/amtrVVNTo5///OdavXq1Kisrl7Q8AwBIrYTX4F944YV57ysr\nK+f+vG/fPu3bty/hogAAyeM3WQHAUgQ8AFiKgAcASxHwAGApAh4ALEXAA4ClCHgAsBQBDwCWIuAB\nwFIpuZskkAnGG9Ld4X4ZN48nZwFLQMBj2TNTk5o90yT1dGlgbFQKFEqVVXIj9XLW5me7PGDZYokG\ny97smSap/ZI0OiKZ2ehr+6Xo5wAWRMBjWTPekNTTFXuwpys6DiAmAh7L20CfNObFHrszKg32Z7Ye\nIIcQ8FjeytdLgaLYY+sKpbKKzNYD5BACHsuaU1QiVVbFHqys4moaIA4CHsueG6mXwrVSYVBy3ehr\nuDb6OYAFcZkklj1nbb58hxtkvCGVmhnddnzM3IElIOCRM5yiEq0JheT09ma7FCAnsEQDAJYi4AHA\nUgQ8AFiKgAcASxHwAGApAh4ALEXAA4ClCHgAsBQBDwCWIuABwFIEPABYioAHAEsR8ABgKe4mCQAZ\nZrwh3R3ul3Hz0nrrawIeADLETE1q9kyT1NOlgbFRKVAoVVbJjdTLWZuf8v2xRAMAGTJ7pklqvySN\njkhmNvrafin6eRoQ8ACQAcYbknq6Yg/2dEXHU4yAB4BMGOiTxrzYY3dGpcH+lO/SqjX4TJ24AIBH\nVr5eChRFl2UetK5QKqtI+S4TDviWlhZ1dXXJcRwdOHBAmzdvnhvr6OjQhQsX5LquampqVFdXl5Ji\nF5LpExdYOYw3FJ15la9n0oCkOEUlUmVVdA3+QZVVafnvK6ElmqtXr6qvr0+NjY166aWXdPbs2Xnj\nZ8+eVX19vU6ePKmOjg7dvHkzJcUuJNMnLmA/MzWpmd+9otlX/lezrx3V7Cv/q5nfvSIzNZnt0pDD\n3Ei9FK6VCoOS60Zfw7XRz9MgoRl8Z2endu/eLUnasGGDxsfHNTExIb/fr/7+fhUUFKi0tFSSVFNT\no87OTm3YsCF1Vd9nKScumHnhUc1NGu65b9LgO9yQvcKQ05y1+fIdbpDxhlRqZnTb8S2/6+A9z9PG\njRvn3gcCAXmeJ7/fL8/zFAgE5sYKCwvV19e3pO2GQqFHruXucH90WSaWO6MqNTNak8B2c0ki/265\nLN39zgwNqu9Gt2ZjjLk3ulWxZpV8JWVpreEevreW+qzP/0nzblJyktUYk9DYg3p7ex99325edM19\ngRMXtx2fnAS2mytCoVBC/265KhP9mg/f1+xI7EvWZr1h9V1pl1O1La01SHxvbZeqfuMdFBNagw8G\ng/K8zy/3GRkZUTAYjDk2PDys4uLiRHazJHMnLmJJ04kLWO7e1Q6xpOlqByAdEgr4cDis1tZWSVJ3\nd7eCwaDy86NXq5SXl2tyclIDAwOamZlRW1ubduzYkbqKY8j0iQvYjUkDbJHQEk11dbU2btyohoYG\nOY6jgwcP6uLFi/L7/aqtrVUkElFzc7Mk6emnn077ulqmT1zAfm6kfu7SW90Zjc7cP7v0FsgVjnmU\nRfI0S9V6FOt49sp0v8Ybiv6GYVlFxicNfG/tlok1eKt+kxVINaeoROKnQeQo7kUDAJYi4AHAUgQ8\nAFiKgAcASxHwAGApAh4ALEXAA4ClCHgAsBQBDwCWIuABwFIEPABYioAHAEsR8ABgKQIeACxFwAOA\npQh4ALAUAQ8AliLgAcBSBDwAWIqABwBLEfAAYCkCHgAsRcADgKUIeACwFAEPAJYi4AHAUgQ8AFiK\ngAcASxHwAGApAh4ALEXAA4ClCHgAsBQBDwCWIuABwFIEPABYioAHAEsR8ABgqbxE/tL09LRef/11\nDQ4OynVdHTp0SBUVFfO+5rvf/a6qq6vn3v/iF7+Q63I8AYBMSSjg//Wvf8nv9+vkyZNqb2/X+fPn\n9bOf/Wze1/j9fp04cSIVNQIAEpDQlPrKlSuqra2VJG3fvl3Xrl1LaVEAgOQlNIP3PE+BQECS5Lqu\nHMfR9PS08vI+39ynn36q5uZm3b59W0899ZSeffbZRbcbCoUSKSdt28kV9GuvldSrRL+ptmjAv/32\n23rnnXfmfdbV1TXvvTHmob/3/e9/X1/72tckScePH9fWrVu1adOmuPvq7e1dtODFhEKhlGwnV9Cv\nvVZSrxL9JrOdhSwa8Hv37tXevXvnfXb69Gl5nicpesLVGDNv9i5J3/zmN+f+vH37dl2/fn3RgAcA\npE5Ca/DhcFitra2SpMuXL+tLX/rSvPHe3l41NzfLGKOZmRldu3ZNjz/+ePLVAgCWLKE1+K985Svq\n6OjQsWPHtGrVKh06dEiS9Oabb2rbtm3asmWLSkpKdPToUTmOo127dmnz5s0pLRwAEF9CAX/v2vcH\nPffcc3N/fvHFFxOvCgCQNH7zCAAsRcADgKUIeACwFAEPAJYi4AHAUgQ8AFiKgAcASxHwAGApAh4A\nLEXAA4ClCHgAsBQBDwCWIuABwFIEPABYioAHAEsR8ABgKQIeACxFwAOApRJ6ZB8ApJLxhnR3uF/G\nzZNTVJLtcqxBwAOYY7whaaBPKl+fkaA1U5OaPdMk9XRpYGxUChRKlVVyI/Vy1uanff+2I+ABzAta\njXlSoCgjQTt7pklqv/T5B6MjUvslzZ5pku9wQ9r2e0+mD2iZRsADyErQGm8oekCJpadLxhtKW+hm\n64CWaZxkBVa4pQRtWgz0RcM1ljuj0mB/evar+w5ooyOSMfMOaDYh4IGVLltBW74+OnOOZV2hVFaR\nlt1m7YCWBQQ8sNJlKWidohKpsir2YGVV+tbEs/iTQ6YR8MAKl7WgleRG6qVwrVQYlFw3+hqujX6e\nLlk6oGUDJ1kByI3Uf37S8c5oNOg+O+mYTs7afPkON8h4Qyo1M7rt+NJ+NcvcAe3+k8r3pPmAlmkE\nPIB5QavBfqmsIqNB5xSVaE0oJKe3NyP7y9YBLdMIeABznKISyaIZ7EKyfUDLFAIewIpl+wGNk6wA\nYCkCHgAsRcADgKUIeACwFCdZgWWI+6MjFQh4YBnh/uhIJZZogGVk/l0OZ629yyEyg4AHlomVdJdD\nZEbCAX/16lVFIhFdvnw55vi7776rI0eO6OjRo3rnnXcSLhBYMVbQXQ6RGQmtwff19elvf/ubqqur\nY45PTU3pz3/+s1599VXl5eXpyJEjqq2tVUFBQVLFAla7d5fD0ZGHxyy7yyEyI6EZfDAY1Msvvyy/\n3x9z/D//+Y82bdokv9+v1atXq7q6Wh988EFShQK2y+Zte2GnhGbwa9asiTvueZ4CgcDc+0AgIM9b\n4EfP+4RCoUTKSdt2cgX92mP22GsaOtWgT7uuatYblltUrNVV21Ty/16Rmx97QmUTm7+3saS730UD\n/u23335oDf3b3/62du7cmfJielNwq9BQKJSS7eQK+rVQ5GXJG1L5Z/dH//9FJeob8aSRxSdJuWxF\nfG/vk6p+4x0kFg34vXv3au/evY+0w2AwOG/GPjw8rKqqBX70BPCQTN8fHXZKy2WSVVVV+uijjzQ+\nPq6pqSldu3ZNW7duTceuAAALSGgNvq2tTX/9619169YtdXd366233lJDQ4PefPNNbdu2TVu2bNEL\nL7ygxsZGOY6jurq6BU/IAgDSI6GAf/LJJ/Xkk08+9Plzzz039+c9e/Zoz549iVcGAEgKv8kKAJYi\n4AHAUgQ8AFjKMcaYbBcBAEg9ZvAAYCkCHgAsRcADgKUIeACwFAEPAJYi4AHAUgQ8AFgqoXvRLBct\nLS3q6uqS4zg6cOCANm/ePDfW0dGhCxcuyHVd1dTUqK6uLouVJi9er1euXJnr9bHHHtNLL70k183t\nY3e8fu85f/68PvzwQ504cSLzBaZYvH5v376t5uZmTU9P64knntAPf/jDLFaavHi9/v3vf9e7774r\n13W1adMmHThwIHuFpsj169d16tQp7d+/X88888y8sbTnlMlR77//vnn11VeNMcbcuHHDHD16dN74\nT3/6UzM4OGhmZmbMsWPHzI0bN7JRZkos1uuPf/xjc/v2bWOMMU1NTeby5csZrzGVFuv33ucNDQ3m\n+PHjGa4u9Rbrt6mpybz33nvGGGP+9Kc/mcHBwYzXmCrxeh0fHzeHDh0y09PTxhhjTp48aa5du5aV\nOlNlcnLSnDhxwvzhD38wb7311kPj6c6pnJ3mdXZ2avfu3ZKkDRs2aHx8XBMTE5Kk/v5+FRQUqLS0\ndO7I2NnZmc1ykxKvV0n69a9/rZKS6PM6A4GAPvnkk6zUmSqL9StJ586d0/PPP5+N8lIuXr+zs7P6\n4IMPtGvXLklSJBJRaWlp1mpNVrxe8/LylJeXp6mpKc3MzOju3bsqKCjIZrlJW7VqlY4cOaJgMPjQ\nWCZyKmcDPt5zXx8cKyws1MhIjCfV54jFnnF77177IyMjam9vV01NTcZrTKXF+r148aK2bdumsrKy\nbJSXcvH6HRsbU35+vlpaWnTs2DGdP38+W2WmRLxeV69erbq6Oh0+fFiHDh1SVVVVzj+j1efzafXq\n1THHMpFTORvwDzJxbqkTbywXxepndHRUv/nNbxSJRLRu3bosVJU+9/f7ySef6J///KeeffbZLFaU\nXg9+f4eHh7Vv3z798pe/1Mcff6y2trYsVZZ69/c6MTGhv/zlL2pubtbp06fV1dWlnp6e7BWXYenI\nqZwN+Aef+zoyMjL3Y1CsZ8IWFxdnvMZUiderFP0f41e/+pWef/55hcPhbJSYUvH6vXLlisbGxnT8\n+HG99tpr+vjjj9XS0pKlSlMjXr/r1q1TaWmp1q9fL9d1tX37dt24cSNbpSYtXq+3bt1SeXm5AoGA\n8vLytHXrVnV3d2er1LTLRE7lbMCHw2G1trZKkrq7uxUMBpWfny9JKi8v1+TkpAYGBjQzM6O2tjbt\n2LEjm+UmJV6vUnQ9ev/+/dq5c2e2SkypeP3u2bNHv/3tb9XY2KiXX35ZTzzxRM5faRGvX5/Pp4qK\nCv33v/+dG8/lZYt4vZaVlenWrVv69NNPJUkfffSRHnvssazVmm6ZyKmcvl3wG2+8oX//+99yHEcH\nDx5UT0+P/H6/amtrdfXqVb3xxhuSpKeeekrf+ta3slxtchbqNRwO6wc/+IG2bNky97Vf/epX9Y1v\nfCOL1SYv3vf2noGBAb3++utWXCYZr9++vj6dPn1axhh94QtfUCQSyenLYOP1+o9//EMXL16U67qq\nrq7Wiy++mO1yk9Ld3a1z585pcHBQPp9PxcXF2rVrl8rLyzOSUzkd8ACAheXuNAAAEBcBDwCWIuAB\nwFIEPABYioAHAEsR8ABgKQIeACz1fzf5+5gmVpp7AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "_8bjYFl64-bo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Helper function\n",
        "def batch_generator(features, labels, batch_size):\n",
        "  \"\"\"Function to create python generator to shuffle and split features and labels into batches along the first dimension.\"\"\"\n",
        "  idx = np.arange(features.shape[0])\n",
        "  np.random.shuffle(idx)\n",
        "  for start_idx in range(0, features.shape[0], batch_size):\n",
        "    end_idx = min(start_idx + batch_size, features.shape[0])\n",
        "    part = idx[start_idx:end_idx]\n",
        "    yield features[part], labels[part]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "z7wf8Rp2sPxc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Reset the graph \n",
        "tf.reset_default_graph()\n",
        "\n",
        "#Polynomial degree + 1\n",
        "#P = 2 is a linear function.\n",
        "P = 20\n",
        "\n",
        "x_poly = []\n",
        "for i in range(P):\n",
        "  x1 = np.power(x, i)\n",
        "  x_poly.append(x1)\n",
        "\n",
        "x_poly = np.asarray(x_poly).T\n",
        "  \n",
        "\n",
        "#Parameters\n",
        "learning_rate = 0.5\n",
        "batch_size = 11\n",
        "n_epochs = 1000\n",
        "\n",
        "#Create placeholders\n",
        "X = tf.placeholder(tf.float32, shape=[None, P], name='X')\n",
        "Y = tf.placeholder(tf.float32, shape=[None], name='Y')\n",
        "#Create variables\n",
        "A = tf.get_variable('A', shape=[P], dtype=tf.float32, initializer=tf.random_normal_initializer())\n",
        "\n",
        "#Create a linear model\n",
        "y_pred = tf.einsum('i, ki->k', A, X)\n",
        "\n",
        "lam = 0.0\n",
        "#Define a loss function\n",
        "loss = 0.5*tf.reduce_sum(tf.square(y_pred-Y), axis=0) + lam*tf.reduce_sum(tf.square(A))\n",
        "\n",
        "#Define an optimizer\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Z2NLZ_WUvxHq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "2fcbe0ee-bc7a-42d8-ecf1-db6925edc120"
      },
      "cell_type": "code",
      "source": [
        "#Create a TF session:\n",
        "with tf.Session() as sess:\n",
        "  sess.run(tf.global_variables_initializer())\n",
        "  for i in range(n_epochs):\n",
        "    gen = batch_generator(x_poly, y, batch_size) #create batch generator\n",
        "    total_loss = 0\n",
        "    for j in range(np.int(x.shape[0]/batch_size)):\n",
        "      x_batch, y_batch = gen.__next__()\n",
        "      _, batch_loss = sess.run([optimizer, loss], feed_dict={X: x_batch, Y: y_batch})\n",
        "      total_loss += batch_loss\n",
        "    if (i+1)% 50 == 0 or i == 0:\n",
        "      print('Epoch: {0}, Loss: {1}'.format((i+1), total_loss))\n",
        "    a = sess.run([A])\n",
        "    a = np.squeeze(np.asarray(a))"
      ],
      "execution_count": 299,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1, Loss: 6.9257354736328125\n",
            "Epoch: 50, Loss: 0.8402787446975708\n",
            "Epoch: 100, Loss: 0.7453343868255615\n",
            "Epoch: 150, Loss: 0.6794025897979736\n",
            "Epoch: 200, Loss: 0.6185773015022278\n",
            "Epoch: 250, Loss: 0.565704345703125\n",
            "Epoch: 300, Loss: 0.521868109703064\n",
            "Epoch: 350, Loss: 0.48641717433929443\n",
            "Epoch: 400, Loss: 0.4578586518764496\n",
            "Epoch: 450, Loss: 0.43448105454444885\n",
            "Epoch: 500, Loss: 0.4147196114063263\n",
            "Epoch: 550, Loss: 0.39732497930526733\n",
            "Epoch: 600, Loss: 0.38139456510543823\n",
            "Epoch: 650, Loss: 0.3663322925567627\n",
            "Epoch: 700, Loss: 0.3517800271511078\n",
            "Epoch: 750, Loss: 0.3375454843044281\n",
            "Epoch: 800, Loss: 0.32354456186294556\n",
            "Epoch: 850, Loss: 0.3097561001777649\n",
            "Epoch: 900, Loss: 0.29619425535202026\n",
            "Epoch: 950, Loss: 0.282889723777771\n",
            "Epoch: 1000, Loss: 0.2698800265789032\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dKlg0CVE1cby",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "cac16f17-118a-4766-dced-130858874565"
      },
      "cell_type": "code",
      "source": [
        "p = np.poly1d(a[::-1])\n",
        "x_eval = np.linspace(0,1, 100)\n",
        "plt.plot(x_eval, p(x_eval))\n",
        "plt.scatter(x, y);"
      ],
      "execution_count": 300,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XdgFGX+x/H3M5tuCAmBABGlBgSV\ngFLtigXFLhYEFSUqcljxvANBudNYDjkPf8KdioKoeN6p2BFRBLEgCh5FEEOTGhKSLIE0SOb7+2Oo\nEgJsdjNbvq9/YnY2O5+HyGeH2WeeMSIiKKWUCjuW2wGUUkoFhha8UkqFKS14pZQKU1rwSikVprTg\nlVIqTGnBK6VUmIpyO8D+Nm3a5JfXSU9P99trBbtIGitE1ngjaawQWeP151jT09MPuU2P4JVSKkxp\nwSulVJjSgldKqTClBa+UUmFKC14ppcKUFrxSSoUpLXillApTQTUPXh0d8RZQUbgFsaIwyalux1FK\nBRkt+BAk5WXYE8fC2hzyirdBUn1okYGVNQwTF+92PKVUkNBTNCHInjgWFs2HbUUgtvN10XzncaWU\n2k0LPsSItwDW5lS/cW2Os10ppdCCDz15uVDsrX7b9m2Qv6Vu8yilgpYWfKhJawJJydVvq1cfGjWu\n2zxKqaClBR9iTHIqtMiofmOLDJ1No5TaSws+BFlZwyCzG9RPActyvmZ2cx5XSqnddJpkCDJx8XiG\njkS8BTSUKrYajx65K6UOogUfwkxyKrHp6ZgIuUmCUuro1Krg161bx5gxY+jTpw+9e/c+YNvixYt5\n8803sSyLzp0707dv31oFVUopdXR8PgdfXl7OpEmTOOmkk6rdPmnSJIYNG8Zjjz3G4sWL2bBhg88h\nlVJKHT2fCz46Oprhw4eTkpJy0LYtW7aQmJhIw4YN9x7BL1mypFZBlVJKHR2fC97j8RATE1PtNq/X\nS1JS0t7v69evT1FRka+7Ukop5YM6+ZBVRI7oeTXdHfxo+fO1gl0kjRUia7yRNFaIrPHWxVgDUvAp\nKSl4vfsupy8sLKRBgwaH/blNfpoNkp6e7rfXCnaRNFaIrPFG0lghssbrz7HW9EYRkAud0tLSKCsr\nIy8vj6qqKhYuXEjHjh0DsSullFKH4PMR/OrVq5kyZQr5+fl4PB7mzZtHly5dSEtLo1u3bmRlZTFu\n3DgAevbsGVH/9Ap34i1wFj1La6IXWCkVxHwu+FatWjF69OhDbu/QoQPZ2dm+vrwKQvvfaIRir7Po\nmd5oRKmgpWvRqCN24I1GRG80olSQ04JXR0RvNKJU6NGCV0dGbzSiVMjRgldHRm80olTI0YJXR0Rv\nNKJU6NGCV0dMbzSiVGjR9eDVEdv/RiPkb4FGjfXIXakgpgWvjppJTgUtdqWCnp6iUUqpMKUFr5RS\nYUoLXimlwpQWvFJKhSkteKWUClM6iyYMiQjs2glVVfsejInFeDzuhVJK1Tkt+BAlu3bBhrXs+N+3\n2CuWI1tzIT/XWRemdAdUVh78Q3HxcEw9SG6ASWsKjZpijj0emmdAg4YYY+p+IEqpgNGCDxGyayes\nXI4s+x+yfBFsWAtVlRxwK/PYeOfq0tQ0iD8GoqN3/7BARTmU7ICS7bDmV2TVL86mPT9brz5kdMB0\n6Izp0AnTqEndDU4pFRBa8EFMKipg6QJkwTfI4h+ckgbwRMHxrTDN25CceSrbEpKgURNITDqio3Cp\nrITCfMjbjKxfg/yWA2tyYOF3yMLvnNJvehym65mYrmdgmjQL6DiVUoGhBR+EZN0q5KsZyPdzoLzM\nebBRE0xmd0yHTtD2RExsHACJ6ekUH+XNe01UFKQ1hbSmmJNO2bffvM3Isp+QpQvh55+QD6YiH0x1\nFhM7uzem61mY2Fi/jVMpFVha8EFC7CpY+B32Z+/Bml+dB1MaYs69BNPlDDiuVcDPkZu0ps65+XMu\nQcpKkUXfI/PnwtKFyKv/h/znFcyZF2IuuAKT3CCgWZRStacF7zKprES+/RyZMQ3yNoMxzgqNZ10E\nJ52CsdyZ+WLiEzA9zoUe5yIF+cjcGcjcz5DPpiGzPsScfj6m9zWYhroOvFLBSgveJWLbyA9zkfff\ncGa/REVhzroIc+FVmMbpbsc7gElthLlyANLneuS7Wcin7yBzPkW++RxzTh9Mn2sxiUlux1RK/Y4W\nvAtkxVLsf78EG9aAJwpz3qWYi/sG/WkPEx2NOesi5PTzkR++Qt6finz+PvLNTEyf6zC9LnfO7yul\ngoL+baxDUrgVeXsS8sNcAEyPczGX9wu5KYnG48H0OBc59QxkznTk47eQtycj387C6j8Y0/YktyMq\npdCCrxNi204RvjMFKsqgRQbWjXdiWrZ1O1qtmOhozPmXIz3PRaa9hnw1A3vMCMxpvTDXD8IkJLod\nUamIpgUfYLJlE/arz0HOMkg4BnPzUOcDSit8lgEyx9TDDBiCnH4+9usTkG+/QJYvwrp56AHTMJVS\ndUsLPkBExJnL/p+JsHMndO6BdePgoD/PXhumZVus4c8g099GPn4Le9xozDkXY669DROj8+eVqmta\n8AEgO4qxX30e/jcPEhIxA+/FdDkjItZ6MVFRmMtuQDK7Yb/yLDJ7OrJyOdYdD2Ga6hWxStWl8DlP\nECRk1S/Yf7nXKfd2J2M9+hxW1zMjotz3Z45vhTXiGczZvWHDWuzH78ee96XbsZSKKHoE7wfiLUC2\nbIaVPyMfvgW2jblyAObia1y7UCkYmJhY59z8CR2xpzyPvPws9m+rMH1v1aWLlaoDWvC1IOVl2BPH\nOksLFHudB6OiMEMfxurY1d1wQcR0OQPruFbY47ORzz9ANqx1TtnU04ujlAokPUVTC/bEsbBo/r5y\nB6isRL6a4V6oIGUap2MNHwOdusMvi7GfGIbkbnA7llJhTQveR+ItgFXLq9+4NsfZrg5g4hOw7hqO\n6XMdbN2C/eRDyK9L3Y6lVNjSgveRzJsDO7ZXv3H7NsjfUreBQoSxLKwrB2AG3gsVZdjPPoL9/Ry3\nYykVlrTgfWB/8wUybcqhn1CvPjTSVRZrYp3eC+ve0RAdg7z8d+wvP3Y7klJhRwv+KIgI9vS3kcnj\nIC4B2rSv/oktMjDJqXUbLgSZ9plYf3wS6tVHpr6A/dG/nRuGK6X8Qgv+CIkI8u4U5N0p0KAh1p+f\ndo5AM7s590G1LOdrZjesrGFuxw0Z5riWWA89BalpzuqU/31FS14pP/F5muTkyZPJycnBGMPAgQNp\n06bN3m1/+MMfSE1Nxdq93so999xDgwahe4m+2Dby1kRk1kfQ+FisB/6KadAIAM/Qkc4HqvlboFFj\nPXL3gWmcjvWnp7GffQSZ+T7YNlyfFXEXhynlbz4V/LJly8jNzSU7O5sNGzbwz3/+k+zs7AOeM2LE\nCOLi4vwS0k1i28hr45GvZ8KxzZ1yT0o54DkmORW02GvFpKRiPZiNPXYk8sWHTsn3u0NLXqla8OkU\nzZIlS+ja1bmQp1mzZpSUlFBaWurXYMFAbBuZ8rxT7se3xnow+6ByV/5jkpKxHsyGY5sjX36MvPmC\nnq5RqhZ8OoL3er20atVq7/dJSUl4vV4SEhL2Pvbiiy+Sn5/PCSecwI033nhER2Lp6f67VV1tX0ts\nm6Lnn6Tkm8+JbtOetOwJWIn1/JTOv/z55+a+dKrGvEz+w3ex68tPSGyQSv1b7zng/5/wGm/NImms\nEFnjrYux+mWpgt8fZV133XV06tSJxMRExowZw/fff0+PHj0O+zqbNm3yRxzS09Nr9Voigkx9AZn9\nCRzXkqqhI8kt3g7Fh5j37qLajjVYydBRMGY42995jR27qrAuuwEI3/FWJ5LGCpE1Xn+OtaY3Cp9O\n0aSkpOD17rs8v6ioiJSUfacuzj77bOrXr4/H46Fz586sW7fOl924Rt551Sn3Zi2wHngMc0xwHrmH\nM5OUjHX/Y87smg+mYn/2ntuRlAo5PhV8ZmYm8+bNA2D16tWkpKQQHx8PQGlpKdnZ2VRWVgLOB7LH\nHXecn+IGnj39HWTGu85smfv/iknUBbHcYho0xBr2OCQ3QP77CvZ3utywUkfDp1M07dq1o1WrVowc\nORJjDIMGDWL27NkkJCTQrVs3OnfuzMMPP0xMTAwtWrQ4otMzwcD+6lPk3Vedee73/xWTlOx2pIhn\nGjXBuu8v2H/7M/Lqc5Q1bwnpLdyOpVRIMBJE0xTcPAcvC7/D/tdTkJiE9dCTmCahcfehSDlvKTnL\nsJ99BGNZmGGPh/wNy49EpPxu94ik8Qb1OfhwIyuXOUv/xsRh3ftoyJR7JDEZHbDu+COyayf2/z2G\n5Oe6HUmpoBfxBS+5G7Cfz4aqSqzBf8I0b3P4H1KuMJ26kzLkT7B9G/Zzf0VKdrgdSamgFtEFL8VF\n2P8YDSXbMTcPxZx0ituR1GEkXnwN5sKrIHcD9j+fRCp3uR1JqaAVsQUvu3Zij38CCvIwl/XDOv18\ntyOpI2SuuQVO6QkrliBTxuvVrkodQkQWvIggk5+D1Ssw3c/G7L6IRoUGY1lYtz0ALTKQ72YhM3WO\nvFLVicyC/+gtZP5X0PoEzC1364JWIcjExmL9YYQzR/7tV5ElC9yOpFTQibiClwXfIh9MhdQ0rCEj\nMNExbkdSPjLJqVhDHoaoKOyXxiCb17sdSamgElEFLxvWYk/6B8TGYQ0dqRcyhQHTMgNzy91QVor9\nfDZSqjNrlNojYgpeSrZjT3gCKsqxbrsP06yF25HUURJvARVLf3JusLIfq/vZmIuuhrxN2K/8A7Ft\nlxIqFVz8sppksBO7CvvFMZCfi+lzHeaU09yOpI6ClJc5F6KtzSGveBsk1YcWGVhZwzBxzhpI5qqb\nkHWrYNF85OP/6AfnShEhR/Dy/lRY9j/I7Ia5/Ea346ijZE8cC4vmw7YiENv5umi+8/huxuPBuv2P\nzuqTH76JLP7BxcRKBYewL3hZNB/55L/QqIlzasYK+yGHFfEWwNqc6jeuzTngdI2pl4R113CIisZ+\n+e+6nIGKeGHddpKfi/3ysxAdg3XXcExCotuR1NHKy4Vib/Xbtm9zbna+H9O8NabfHVBagv3C35Bd\neqWrilxhW/Cyayf2P5+EshLMgLswx7V0O5LyRVoTONRsp3r1oVHjgx42Z1yA6Xke/LYS+c/LAQ6o\nVPAK34J/ayKsX4M580Ks03q5HUf5yCSnQouM6je2yHC2//5njMH0v8u5effsT7C/nxPglEoFp7As\n+NK5M5E5n8KxzTE33O52HFVLVtYwyOwG9VPAspyvmd2cxw/BxMZiDf4TxMYjr01AtkTGOuNK7S/s\npklK3mYKn3vcuZjpzj9hYmLdjqRqycTF4xk6EvEW0FCq2Go81R65H/RzTZphbhqCTByL/eIYrD//\nDRMdXQeJlQoOYXUEL7aN/eIYpLQE0/8uTFO9cUc4McmpxJ7Y6YjKfQ+r+9mY08+Hdauc2zEqFUHC\nquApL4XN6zjmoquwep7rdhoVJEy/O6BJM+TzD5BF892Oo1SdCauCNwmJWH9/nZS7R7gdRQURExuH\ndecfnfnxk8cdtNSBUuEqrAoenL/Muvyv+j3TrCXmuttgx3Zdr0ZFjLAreKUOxZxzCZzcBZYvQj7/\nwO04SgWcFryKGMYYrIH3QFIyMm0Ksm6125GUCigteBVRTFIy1q33QmUl9sSxyM4KtyMpFTBa8Cri\nmJNOxfS6DDavR96d4nYcpQJGC15FJHP1zc7UyS8+RJb9z+04SgWEFryKSCYmFmvQ/eDxYE9+DinR\nW/2p8KMFryKWaZGBufR6KNqKTH3B7ThK+Z0WvIpo5uJroWVbZP4cZMG3bsdRyq+04FVEMx4P1m33\nQXQM9usTkEPdXESpEKQFryKeadIMc9VNsKPYKXkRtyOpMCbeAiqW/lQnS2aE3XLBSvnC9LoM+d88\n+Gke8v0cTI9z3I6kwoyUlzk3il+bQ17xNkiqDy0ysLKGYeLiA7JPPYJXCjCWhTXwXoiNQ958AfEW\nuh1JhRl74lhYNB+2FYHYztdF853HA0QLXqndTKMmmGsGOjfs1lM1yo/EWwBrc6rfuDYnYKdrtOCV\n2o85uze0OxkWzUfmf+V2HBUu8nLhUB/gb98G+VsCslufz8FPnjyZnJwcjDEMHDiQNm3a7N22ePFi\n3nzzTSzLonPnzvTt29cvYY+EeAuoKNyCWFFHdecfpWD3qZpb7sYefTfy5otI+46YpBS3Y6lQl9YE\nkpKd0zK/V68+NGockN36dAS/bNkycnNzyc7OZvDgwUyaNOmA7ZMmTWLYsGE89thjLF68mA0bNvgl\nbE2kvIyq5x/HfvwB8v58J/bjD1D1/ONIeVnA963Ci3Oq5hYo2Y496Tnk15/1JiGqVkxyKrTIqH5j\ni4yAHYz6VPBLliyha9euADRr1oySkhJKS0sB2LJlC4mJiTRs2HDvEfySJUv8l/gQ3PgAQ4WxHufC\nMfVg6QLsMcP1gEHVmpU1DOrv/tegMc5/Z3ZzHg8Qn07ReL1eWrVqtff7pKQkvF4vCQkJeL1ekpKS\n9m6rX78+ubm5tU9agyP5AENP16ijIa88CyXb9z2w3wGDZ+hI94Kp0FVc5JyHb9aCRnc/TEEdnEb2\nyzz4mmYbHM1MhPT0dJ/2X1G4xZlXWp3t22goVcT6+NqhwNc/t1AV6PFWFeSTu3411d3Uz1q/msax\n0XhSGwU0wx76uw0fhdOmUCJCav87iDupM8fWwT59KviUlBS83n2fCBcVFZGSklLttsLCQho0aHBE\nr7tp0yZf4iBWlHPRwCE+wNhqPBgfXzvYpaen+/znForqYrzy68/YRdWfc7e9heQuXYTJ6BDQDKC/\n23Ai27dhz/wAUtMoatmeBHzvu9+r6U3Rp3PwmZmZzJs3D4DVq1eTkpJCfLxzJVZaWhplZWXk5eVR\nVVXFwoUL6dixoy+7OWJufYChwtSeGQ/VSUwK2IwHFb5k1sewayfmgisxHk+d7denI/h27drRqlUr\nRo4ciTGGQYMGMXv2bBISEujWrRtZWVmMGzcOgJ49e9bJP7usrGF7LwNm+zZn6tHuy4CVOhp7DxgW\nzT94Y0ysHjCooyIVFcjsjyEhEXPG+XW6b5/Pwffv3/+A71u0aLH3vzt06EB2drbPoXxh4uLxDB2J\neAtoKFXOaRn9i6h8dNABQ2ISVJRDQR6y6hdM6xPcjqhChHz7OezYjulzHSY2rk73HXaLjZnkVGLT\n08P2nLuqG/sfMJC/xTktk5frTJmc8jzWqGcxUdFux1RBTuwqZOb7EBWNOa9Pne9flypQqgYmORWT\n0cH52vZEZymDTeuQ6e+4HU2FgoXfQX4u5rTzXLkiWgteqaNgrr4Fkhsgn/wH2Rz4K7RV6BIR7BnT\nwBjMBVe6kkELXqmjYBKOwbpxMFRWYr/2PGJXN1teKeDXn53PcDK7Y5rUxaz3g2nBK3WUTOce0KkH\n5CxDvvnc7TgqSNkz3gXAuugq1zJowSvlA6vfHRAXj7w9CanuAjsV0WTTOljyI7Q+AdOmvWs5tOCV\n8oFp0BBz9c1QWoL8+yW346ggI59NA8DqfbWrObTglfKRObs3tGqH/Pg1smSB23FUkBBvATJvDjQ5\nFjp2czWLFrxSPjKWB+umIeDxYL/xT6Si3O1IKgjIFx9BVaWzLIHlbsVqwStVC6ZZS2cKXEEe8uG/\n3Y6jXCblpcicTyEpGdPzXLfjaMErVVvm0hugYWNk5nvI+jVux1Eukq8+g7ISzHmXYqJj3I6jBa9U\nbZnYWKz+g8G2sV8bj9hVbkdSLpDKSuTzD5wF6c652O04gBa8Un5hTjoV0/VMWPMrMmeG23GUC+SH\nuVC0FXPmhZhj6rkdB9CCV8pvzPVZEH8MMm0K4i10O46qQyKCzHgXLAtz/uVux9lLC14pPzH1U5y5\n8WWlyFsT3Y6j6tLPC2Hjb5guZ2AaBs8NYbTglfIjc9ZF++bGL9W58ZHCnuFc2GRcXJagOlrwSvmR\nsSxnbrxlYb/xL6Siwu1IKsDkt5Xwy2Jon4k5vrXbcQ6gBa+Unzlz46+ArVuQj3VufLiTGcGxLEF1\ntOCVCgBzWT9ITUM+ew/Z+JvbcVSASH4u8uM3cFxLaN/J7TgH0YJXKgBMbBxWvzuhqgr79Qm6bnyY\nks/eA7ExF12NMcbtOAfRglcqQExmVzilJ6xcruvGhyHZvs35vaamYbqc4XacamnBKxVA1g171o2f\njBR73Y6j/EhmfQy7djqLink8bseplha8UgFkUlIxVw6A0h3Ifye5HUf5iVSUI19+DIn1MGec73ac\nQ9KCVyrAzLmXQPM2yLwvkeWL3I6j/EC+ngkl2zHn9sHExrkd55C04JUKsL3rxpvdc+N37XI7kqoF\nqax0PlyNicGc28ftODXSgleqDpjmbTDn9YEtG5Hpb7sdR9WC/DAXCvMxp1+AqVff7Tg10oJXqo6Y\nK/pDcgNk+n+R3I1ux1E+ENtGPn3HWVTswivdjnNYWvBK1RETn4DV7w6orHRu8SfidiR1tJYsgE3r\nMN3OCqpFxQ5FC16putS5J3TsCr8sRubNdjuNOkr2p87pNXNR8C1LUB0teKXqkDEG68Y7ISYW+c/L\nyI5ityOpIyQ5y2DlcujYFdOshdtxjogWvFJ1zKSmYa64EXYUI2/r3PhQYe/+cNy6+BqXkxw5LXil\nXGB6XQ7HtUS++QJZsdTtOOowZN1qWPIjtOmAadPB7ThHTAteKRcYjwfrpqHO3PjXxyO7drodSdVg\nz9RWq8+1Lic5OlrwSrnEtMxw5sbnbkQ+0bnxwUpyNyALvoHjW8OJp7gd56howSvlInNlf0hpiEx/\nG9m83u04qhry6TsggnXJtUG5JHBNtOCVcpGJS8C68Q6oqsR+bbyuGx9kpCDPmc7apBl07uF2nKMW\n5csPVVZWMmHCBPLz87EsiyFDhtC48YGT/vv160e7du32fv/II49gWfp+otTvmU49nHXjF36HfP0Z\n5qzebkdSu8mMaVBVhbm4LyYE+8ungv/6669JSEjgscceY9GiRUydOpX777//gOckJCQwevRof2RU\nKuxZ/e7AXr7IWTe+Y1dMcqrbkSKeeAuRuZ85N/TodpbbcXzi01vS0qVL6datGwAnn3wyK1as8Gso\npSKNSU7FXDMQykqx33zR7TiK3Ufvlbswl1yLifLpWNh1PqX2er0kJSUBYFkWxhgqKyuJ2u8PYefO\nnYwbN46tW7fSvXt3Lr300sO+bnp6ui9xAv5awS6SxgrhO165fiD5P31HxcLvSP7tV0hPD9uxHkqw\njLeqqIDNcz/F06gxTa8ZgImO9vs+6mKshy34L774glmzZh3wWE5OzgHfV7do0k033cRZZzn/rHn0\n0Udp3749rVu3rnFfmzZtOmzgI5Genu631wp2kTRWCP/xyvW3w1/voeD5J4g7+VRyi7e7HanOBNPv\n1n57ElJRgX3BVWzOz/f76/tzrDW9URy24Hv16kWvXr0OeGz8+PF4vc79JSsrKxGRA47eAS688MK9\n/33yySezbt26wxa8UpHONG2G6XMd8v5UvK88B31vdTtSxJHtxcjs6ZDcIKhvx3ckfDoHn5mZybx5\n8wBYsGABJ5544gHbN23axLhx4xARqqqqWLFiBccdd1zt0yoVAUzva+DY5pTMmIb8stjtOBFHZr4H\nFeWYi67GRMe4HadWfCr40047Ddu2GTVqFDNmzODGG28E4L333uPXX38lPT2d1NRURowYwahRo+jc\nuTNt2rTxa3ClwpWJisa65R6wLOwpzyMVFW5HihiyvRiZ9REkJWPOvMjtOLXm04ese+a+/96VV+67\nw8mAAQN8T6VUhDMtM6h3xY1sn/Y68sEbmGtvcztSRJAZ7zhH71fdhImNdTtOrYXezH2lIkTSgMHQ\nqAky8wNktU5FDjQpLkK+/BiSUzFnhf7RO2jBKxW0rLg451SN2NiTn9MVJwNMpr8LO3di+lwb8ufe\n99CCVyqImXYnYc69BDavRz78t9txwpZ4C5A506FBI8wZF7gdx2+04JUKcubqWyA1DZnxLrI25/A/\noI6afPJf2LUTc+n1mCj/X9TkFi14pYKciYvHuuVusPecqtnldqSwIvm5yFefQaMmmJ7nuR3Hr7Tg\nlQoBpn0m5pyLYeNvyIdT3Y4TVuS9N6CqEnPlgJBdc+ZQtOCVChHmmoHOrJpPpyGrfnE7TliQdauR\n+XPg+FaYLme4HcfvtOCVChEmLh5r4L2AYL/yD70Ayg/saVMAsK6+JSTXez+c8BuRUmHMtD0Rc8EV\nkLcJefdVt+OENFmxBJYuhBM6QodObscJCC14pUKMuXIAND0OmfUR8vNPbscJSWLb2G9PBsC6+uaQ\nu9fqkdKCVyrEmOgYrEEPgMeDPWkcsqPY7UghR+bPgbU5mK5nYlq2dTtOwGjBKxWCTPPWmCv6w7ZC\n7NcmVHtPBlU9qahA3n0NoqIxV9/sdpyA0oJXKkSZi66CjA6w8Fvku1mH/wEFgMycBkVbMRdcjmnY\n2O04AaUFr1SIMpYH67b7IT4BmfoisiU47oYUzMRbgEx/B+rVx1x8rdtxAk4LXqkQZho2xvS/CyrK\nsF96BqnUq1xrItNeh50VzkVN8Qluxwk4LXilQpzV/WzMab3gt5XIu1PcjhO0ZOUy5NsvoFnLkL8V\n35HSglcqDJh+d0CTY5GZ7yNLFrgdJ+hIVRX2G/8CwOo/GGN5XE5UN7TglQoDJi4e6/Y/QlQU9ivP\nIoVb3Y50VMRbQMXSnxBvQWBef/YnsGEt5vRemDbtA7KPYBReK+soFcHM8a0w1w1Cpr6A/eLfsB58\n4qgXzxJvAeTlQloTTHJqgJLut7/yMuyJY2FtDnnF2yCpPrTIwMoahomL988+vIXI+29AwjHO0ssR\nRAteqTBizrkEcpYhP8xF3n0Vc92gI/q5/YuWYi8kJfu9aKtjTxwLi+bve2BbESyajz1xLJ6hI/2y\nD/nvK1BWiuk/GJOUvO/xOn4zc4MWvFJhxBgDN/8BWb/aOR/fpj3mlNMO+3N1UbS/J94C5w2lOmtz\nEG9BrYtXFv2AzP8KWmTsvc+qW29mbtBz8EqFGROXgDV4OMTEOksZbF5f4/OPpGgDIi/XKdjqbN8G\n+Vtq9fJSWoL9+gTwRGENvGfOYUidAAAL+klEQVTvB6t738y2FYHIAW9m4UYLXqkwZI49HnPzUCgv\nw34+GyndcegnB7hoDymtiXP0XJ169aFR7a4ylbcngbcA0+c6zLHNncfcejNziRa8UmHK6n62s5xB\n3ibsl8YidlX1Twxw0R6KSU6FFhnVb2yRUavTM7J8ETL3Mzi2Oebia/ZtcOvNzCVa8EqFMXP1zXDS\nKbB0AfLe69U/J4BFezhW1jDI7Ab1U8CynK+Z3ZzHfSSlO7AnPwfGck7N7H8TbZfezNyiH7IqFcaM\n5cHKehD7iWHI9HewGx+LdfrBV3FaWcP2ffC4fZtTdrs/eAxovrh4PENHIt4CGkoVW42ndkfuIshr\nE6AwH3NZP8zv3rj2vpnt/4HyHgF+M3ODFrxSYc4ck4h19yjsJx9CXhuPNGiEaZ954HP2K1ryt0Cj\nxnVadiY5ldj0dMym2i2YJt/OQn78Gtq0x/S5rtrnuPVm5gYteKUigGnSDOsPI7CffQT7n09h/flp\nTPrxBz8vORVC9ChW8jYhb74A8QlYgx7AeKpfjsDtN7O6pOfglYoQpu1JmFvugbIS7HF/CbnlDGoi\nOyuwX/gbVJRj+t91ROu8m+RUTEaHsC130IJXKqJYPc5x7ulamI/9j0eR7aF/uz/nvPt4WLcac+aF\nWN3PdjtS0NCCVyrCmEuuxVxwBWxejz1uNFJe6nakWpEvPkTmzYaWbTH97nQ7TlDRglcqwhhjMNfe\nhjndWUPefj4bqSh3O5ZPZMUSZ62ZpGSsu4ZjoqMP/0MRRAteqSAU6OVzjTGYm4bCKafBiiXYz/0F\nKS8LyL4CRTauw57wBBiDNfjPmJTwPZfuK51Fo1QQqYvlc/cwHg/W7Q9iT3wGFnyLPW401j2PhsSt\n7KTA+QyB0hLMrfdhMjq4HSko6RG8UkHkwIWw7IAvhGWiorBu/yOm65mwcjn2s48E/QevsqMYe9xo\nZ52ZvrdinXae25GClha8UkHCrYWwjMeDGfQApud5sOZX7KceQvI2B2RftSU7irH/MRo2r8dccAXW\nRVe5HSmo+Vzwy5YtIysriwULqr//49y5cxk+fDgjRoxg1qxZPgdUKmK4uBCW8Xgwt96LueRaZ3Gy\npx5CVq8I2P58IduKsJ95GH5biTn9fEzfW92OFPR8Kvjc3Fw++ugj2rVrV+328vJy3n77bUaNGsXo\n0aP5+OOP2bGjhuVKlVKuL4RljMG66ibMgCGwYzv2mBHYcz8L6D6PlBTkY/9tOGz8DdPrMszNQzGW\nnoA4HJ/+hFJSUnjwwQdJSKj+w5iVK1fSunVrEhISiImJoV27dvzyyy+1CqpUuHNzVcf9WWf3xrp7\nFMTEIlOex578HLKzok72XR1ZuQz7yQchbxPm4r6Y67O03I+QT7NoYmNja9zu9XpJSkra+31SUhJe\n7yH+6bmf9PR0X+IE/LWCXSSNFcJ7vPaoZygYM5KdOcuwvYVYyQ2IyehA6h8fx6rL2S3p6VRmnsLW\nJ/7Erm8+J2rjWhrcP5qYNicEeLf7frciQskn71D04jNgC8m3P0C9K28M6P7rUl38f3zYgv/iiy8O\nOod+7bXX0qlTJ7+H2VTLleT2SE9P99trBbtIGitEyHizHgRvAWm7l8/dlZxKbpEXig5/kORfBhn2\nOOatieya8ylb7r8Z07sv5tLrA3JB0f6/W9lejPz7JWT+HEhMwrrzIbaf0JHtYfK79+f/xzW9URy2\n4Hv16kWvXr2OaocpKSkHHLEXFhaSkXGIf3oqpQ7ir+Vza50jOgYzYAhySk/sKeORT/6D/Pg11tU3\nwSmnOTf59iMRQb6fg7w1EXYUQ8u2WHc+hElN8+t+IkVATmRlZGSwatUqSkpKKC8vZ8WKFbRv3z4Q\nu1JK1QHToTPW6P/DnHcpbM3F/tfT2E88iCxdiNh2rV9fRCj/aR72mOHIy3+HnRWY6wZh/elpLfda\n8Okc/MKFC/nggw/YuHEjq1evZvr06YwcOZL33nuPDh060LZtW/r37092djbGGPr27XvID2SVUqHB\nxMVj+t2BnHcp8t7ryI9fOxccpTXFnHURpud5mEPNAjoEKS1BFv+AfPEh+XuuAejYFeuG2zGNmvh/\nEBHGiIi4HWIPPQd/9CJprBBZ4w32scpvq5yVHH/8GnbtBGPguFaY9pmYNu2hURNITdu7xILYVc48\n/43rkI2/Ib8shuX/g8pKAOJPO5eK8y7DNG/j5rDqRNCcg1dKqeqY5q0xt92HXJ+FzPsS+WkerFqO\nrFuFzNjviVFRUFUF1R1LNmuJObUnpsuZNDyla1C/oYUiLXilVK2YYxIxvS6DXpc5yw7n/IysXwNb\n85CCLVBaAh4PeKLgmERMenNIPx7Too2ehgkwLXillN+Y2Dg46VTMSae6HUWhi40ppVTY0oJXSqkw\npQWvlFJhSgteKaXClBa8UkqFKS14pZQKU1rwSikVprTglVIqTAXVWjRKKaX8R4/glVIqTGnBK6VU\nmNKCV0qpMKUFr5RSYUoLXimlwpQWvFJKhSkteKWUClMhfcOPyZMnk5OTgzGGgQMH0qbNvns5Ll68\nmDfffBPLsujcuTN9+/Z1Mal/1DTepUuX7h1v06ZNGTx4MJYVuu/fNY11j6lTp/Lrr78yevToug/o\nZzWNd+vWrYwbN47KykpatmzJHXfc4WLS2qtprJ9++ilz587Fsixat27NwIED3QvqJ+vWrWPMmDH0\n6dOH3r17H7At4D0lIernn3+WJ598UkRE1q9fLyNGjDhg+3333Sf5+flSVVUlo0aNkvXr17sR028O\nN967775btm7dKiIiY8eOlQULFtR5Rn853Fj3PD5y5Eh59NFH6zid/x1uvGPHjpXvv/9eREReeukl\nyc/Pr/OM/lLTWEtKSmTIkCFSWVkpIiKPPfaYrFixwpWc/lJWViajR4+Wf/3rXzJ9+vSDtge6p0L2\nEG/JkiV07doVgGbNmlFSUkJpaSkAW7ZsITExkYYNG+59Z1yyZImbcWutpvECPPXUU6SmpgKQlJTE\njh07XMnpD4cbK8CUKVO44YYb3IjndzWN17ZtfvnlF7p06QJAVlYWDRs2dC1rbdU01qioKKKioigv\nL6eqqoqKigoSExPdjFtr0dHRDB8+nJSUlIO21UVPhWzBe71ekpKS9n6flJSE1+utdlv9+vUpKiqq\n84z+VNN4ARISEgAoKipi0aJFdO7cuc4z+svhxjp79mw6dOhAo0aN3IjndzWNt7i4mPj4eCZPnsyo\nUaOYOnWqWzH9oqaxxsTE0LdvX4YOHcqQIUPIyMggPT3drah+4fF4iImJqXZbXfRUyBb870kNS+rU\ntC1UVTembdu28fTTT5OVlUW9evVcSBUY+491x44dfPnll1x66aUuJgqs3/9uCwsLueSSS/jLX/7C\nmjVrWLhwoUvJ/G//sZaWljJt2jTGjRvH+PHjycnJYe3ate6Fq2OB6KmQLfiUlJQDjuqKior2/jPo\n99sKCwtp0KBBnWf0p5rGC85fjieeeIIbbriBzMxMNyL6TU1jXbp0KcXFxTz66KM888wzrFmzhsmT\nJ7uU1D9qGm+9evVo2LAhTZo0wbIsTj75ZNavX+9W1FqraawbN24kLS2NpKQkoqKiaN++PatXr3Yr\nasDVRU+FbMFnZmYyb948AFavXk1KSgrx8fEApKWlUVZWRl5eHlVVVSxcuJCOHTu6GbfWahovOOek\n+/TpQ6dOndyK6Dc1jbVHjx48++yzZGdn8+CDD9KyZcuQn2lR03g9Hg+NGzdm8+bNe7eH8mmLmsba\nqFEjNm7cyM6dOwFYtWoVTZs2dS1roNVFT4X0csFvvPEGy5cvxxjDoEGDWLt2LQkJCXTr1o1ly5bx\nxhtvANC9e3cuv/xyl9PW3qHGm5mZya233krbtm33PveMM87g/PPPdzFt7dT0u90jLy+PCRMmhMU0\nyZrGm5uby/jx4xERjj/+eLKyskJ6CmxNY505cyazZ8/GsizatWvHgAED3I5bK6tXr2bKlCnk5+fj\n8Xho0KABXbp0IS0trU56KqQLXiml1KGF7mGAUkqpGmnBK6VUmNKCV0qpMKUFr5RSYUoLXimlwpQW\nvFJKhSkteKWUClP/D4mxhINCfksgAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "C6FG1X3r7waz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Notice that even without regularization the model does no overfit too much. I might be because the stochastic gradient descent has not properly converged!"
      ]
    },
    {
      "metadata": {
        "id": "DOT6AV8lE9Xb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 1.2 Probability Theory\n",
        "\n",
        "\n",
        "\n",
        "*   Sum rule: $p(X) = \\sum_Y p(X, Y)$\n",
        "*   Product rule: $p(X, Y) = p(Y\\vert X)p(X)$\n",
        "\n",
        "Applying the second rule one gets the Bayes' theorem $p(Y\\vert X) = \\frac{p(X\\vert Y)p(Y)}{p(X)}$.\n",
        "\n",
        "Two random variabes $X, Y$ are independent if and only if $p(X, Y)=p(X)p(Y)$.\n",
        "\n",
        "The maximum of a probability density $p(x)$  is dependent on the choice of variable, because changing variables multiplies the probability density by a Jacobian of the transformation.\n",
        "\n",
        "\n",
        "*   Expectation \n",
        "*   Variance\n",
        "*  Covariance"
      ]
    },
    {
      "metadata": {
        "id": "mZAgabZWFAsI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Thomas Bayes** (1701 - 1761) formulated Bayes' Theory for the case of a uniform prior. **Pierre-Simon Laplace** (1749 - 1827) independently rediscovered the theory in general form. "
      ]
    },
    {
      "metadata": {
        "id": "4ccUcLgeFwXo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Suppose we have a model to describe some phenomenon parameterised by $w$. We observe data $D$ and ask a question how our initial model parameters should be altered to incorporate the data. The answer is given by Bayes' formulation\n",
        "\n",
        "$p(w\\vert D) = \\frac{p(D\\vert w)p(w)}{p(D)}.$\n",
        "\n",
        "\n",
        "\n",
        "*   $p(w)$ is called a prior distribution.\n",
        "*   $p(D\\vert w)$ is called a likelihood function.\n",
        "*   $p(D)=\\int p(D\\vert w)p(w)dw$ is called an evidence.\n",
        "*  $p(w\\vert D)$ is called a posterior distribution.\n",
        "\n",
        "From the Bayesian viewpoint the parameter $w$ is given by a distribution, whereas in a freuenist setting, $w$ is considered to be fixed whose values are determined by some estimator (e.g. maximum likelihood estimator) of the data and error bars are obtained considering the distribution of possible data sets $D$.\n",
        "\n",
        "\n",
        "\n",
        "*   Bootstrap is a process in which multiple data sets are created by sampling with replacement from the original data set $D$..\n",
        "*   Bayesian methods based on poor choices of prior can give poor results with high confidence.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "PvVRJ5k7GsDZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Distributions\n",
        "\n",
        "\n",
        "*   Gaussian\n",
        "*   MAP: maximum a posterior. Maximising the posterior distribution to find parameters.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "cyq4NpL5j6yL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The curse of dimensionality\n",
        "New"
      ]
    }
  ]
}