Part 2 Forward Propagation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align = 'center'> Neural Networks Demystified </h1>\n",
    "<h2 align = 'center'> Part 2: Forward Propagation </h2>\n",
    "\n",
    "\n",
    "<h4 align = 'center' > @stephencwelch </h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIChAMCAgOCQgIDBUMDhERExMTCA0WGBYSGBASExIBBQUFCAcIDwkJDxQPEBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFP/AABEIAWgB4AMBIgACEQEDEQH/xAAcAAEAAQUBAQAAAAAAAAAAAAAAAQIDBAUGCAf/xABVEAABAwICBAcIDQoEBQQDAAABAAIDBBEFIQYSMUETIlFUYXGUFBgyUoGRodQHCCNCYnJ0sbTB0dXwJDM1Q0SCkpPC4RVTs/ElNIOy0mNkc6IXRUb/xAAaAQEBAAMBAQAAAAAAAAAAAAAAAQIDBAUG/8QAMxEBAQACAQEHAgMIAgMBAAAAAAECEQMhBBIxQVFhkXGBBaHwExQVIjKxweFS0WLC8UL/2gAMAwEAAhEDEQA/APGSIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICL0V3nelXOMF7XVepp3nelXOMF7XVepqbHnVF6K7zvSrnGC9sqvU07zvSrnGC9rqvU02POqL0V3nelXOMF7XVepp3nelXOMF7XVeppsedUXovvO9KucYL2uq9TUd53pVzjBe11XqabHnVF6K7zvSrnGC9rqvU1Ped6Vc4wXtdV6mrsedEXorvO9KucYL2uq9TTvO9KucYL2uq9TQedUXorvO9KucYL2yq9TU953pVzjBe2VXqaDzoi9Fd53pVzjBe11Xqad53pVzjBe11XqaDzqi9Fd53pVzjBe2VXqad53pVzjBe11XqaDzqi9Fd53pVzjBe11XqanvO9KucYL2uq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2uq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvOtKucYL2yq9TTvOtKucYL2yq9TQedEXovvO9KucYL2uq9TTvO9KucYL2uq9TQedEXorvO9KucYL2uq9TTvO9KucYL2uq9TQedUXorvO9KucYL2uq9TTvO9KucYL2uq9TQedUXorvO9KucYL2uq9TTvPNKucYL2uq9TQedUXorvPNKucYL2uq9TTvPNKucYL2uq9TQedUXorvPNKucYL2uq9TUd57pVzjBe11XqiGnnZF6J7z3SrnGC9rqvVFYxD2o+lEEMs76jB9SGKSZ+rV1RdqRML3WHcmZs0ounv8AD+Nq9F/TZV3Vn9Z+5/UFdWCpul1SiIqul1TdSipul1CIibpdQiCbpdQiCbpdQiCbpdaytx6jgnFPPUxQymHujVlcI2iEyiAPMj7MF5XBoBNyb2BsVTgmORVT5odSWCop3e6U87Wtl4Muc2KojLXFs1NIGktkYSNoNnNc0UbW6XUIgm6XUIgm6m6pRBVdLqlEFV0uqVKIlFCXRUooul0RKi+dlIKpHhHqCCpERAREQEREBERARERREREERFVQiIoIREQEREUKhSoQEREEKFKhCC1el/6OxD5DV/R5FtFq9L/0diHyGs+jyIrZD85+5/UritN/Ofuf1K4VESigBSiCIiBdERAUoiAiIgLkfZP0nfhdPDM2pw+mDpH8K/EWVsjOBjic95hZRNLnSCzTZ1hq3N8l1y572RcAnxTD5aGCqZSGaSnMkklOamOSCKeOWWnfCJYy5kjWFh4wycdt0HF4Bg8+k2GR11eYKY4nRVFDXQR0kwL6JtRO+hlg7sdrUlbG55k1y2Rh4Q2Bs1w7HAtE201XHVOqJZu5sPiwyjhLIooqanaY3TODYmgPllfDCSbANETQAMydnglNUwQltZWd2Saznmc08VKGsIB1ODiOqGts7Mm9jmTa60o0wdUlzcKopsQDXavdLnNpaC4uCW1EovMAWkHg2uV0WyOrRceNIMahe/unAw+Frm+7UGIRVDiwjNzaeaOOR5B2ty6LrocEximrGF9PIHhp1ZGEOZLE/wASaJ4D4n9DgETbPRERRERAREQEREBEREFKhEEhQNp8ikKG+EfJ8yBrjWDd52BXLLXTf8zH8U/WtiqIsllKoe8CwJAJvYEgE2FzblsFdCUVqCpjkGtHIyRt9XWY9r263JdptfoV1TQIiJoEUJdNCUUXUpoERQmhNksqUBTQqIVKpPhKofj5kURSiCCoUlQgIiKKhQpKhAWr0v8A0diHyGr+jyLaLVaYfo7EPkNX9HkRWyH5z9z+pXVaH5w/E/qV1RBQiIiJHhrS52QaC4mxNgBc5DM5KilqGSsbJG4PY8BzXDYQdhVxc5itY3CnPmLXOpJy4ljBcxVRBIDG+JKRs3Oz3rTzcv7P+bL+nzvp7/T1dXZ+D9v/ACY/1+U9fWfXznr4eOm+ZUxukfEHtMkYa57AeM0PvqFw3A2KvLV6N0To43SzW7pqXcNOeRxHEiF/esbZtugraLLiyuWO8pr9dPvrx92vnwxwzuOF3J0363z17b8PYRFqRVyx4gYZH60FVDr0vFA4OaDKoiuBd2sx0cgv4sm4LY0tstPVYo/ungoTDIw0dVMwa2b6inmjjMeu0kNa0v1XZEgkcllt3OsCTsAJPUMyud0Nw6N9BQSysBl41a1wLgWS1bnzPsWkXaeFsWm4NhcFVKp0WxVs1Q9sJe+GamhrZLlz2UtRUBj+AErsjrse2QMHg6rjkHADplHzDcqKiZsbHyPIayNrnvccg1rAXOJJ2AAFQ8HL15ditZLR/wD62jIbWuDiDV1ZAe2jBbnwDGua5/KSG8q6mCJrGtYxrWMYA1rGgNa1oFg1rRkABuC5v2LnMfhNJOxwf3U19W+QWOu+plfK51xtPGA/dXTqpBabSLCDJaqpdWOvhF4ZPBEzRmaWoI/OQP2WPgkhwsQtyiKxMFxBlVTxVDAQ2VgdqnwmOBLXxut79rw5p6WlZa53Q+UCbFadpBbT4g5zQD4PdUMVU5p5PdJJDb4S6JCdRERFEREQRERREREEREEhQ3afJ8ykKG7T5EGDU/8AMxdX2rZLWVp/KIev6wtmrAsuUwpsdbUYz3SNZkNS2gDXHitpmUtLUvaRewbI+dzneM3VByAWLiOi1NW1r5GvrQWVDJaqoZiFfEx0kQZqUUEUM7YgwNYwScW1rtN3Pc5u7rdHYJn1QkaH09fE2OspnA6kzmNEbZNZrgWuMQEbhnrNYzZq5hpNHNGabunEqiKCOmo8Qjpoo4YWiFsxp2yB2INjZYRvcZGNa8AEiBjr5tW20BxGSrwyiqJjrSyQN4R4FhI9hMbpANwcWa1vhK1SaI0tLHO3D29xy1EYhdPrzTvjjub8CJ5HBjgHOItlexIday3OF0MVLBDTQMEcMEbIYmDY2ONoaxtztyAVgyUXz/T7ShlHV8CdKMEwg8Ex3cmIwQyTcbW911n10J1HWy4vvTmtbgel8k1TBG3TDRSrD5Y29zwUrW1E4c4DgoXNxh1pXZgHUdmRkdiTqlun1ErmtHdM6aurq7DooauOqw4xirbPCyNjOGaXQ6rxIRKHtBILLjlsulXwj2R5mt0roZqaeenosTaNHsbq6cBsbqga9RR0jJ73jqSNeJ0rM2CQNDg7ZPOT9e359Put8Lf17/l1+z6pgumNLVwVdTDHWGGiknimc6llGvJTOc2cU+X5UGOY4a0esCRYElYbvZIwluFR426admGSlmpUuoqzMSPEcbzEIjI1jnkNDi2xuOVdNS0EUMDaaGNkMEcQhjijaGsjjDdRrGNGTWgZWXmWpqJan2P5KCmkLRhFJVTV5Fi61DiMopqF19j3iF8jt4bC3/MBTf6/uf7/ANPRuJaRUlM2mfO6WMVkjIaf8lqnOfLILsjc1kRMTyATZ4b4J5Csd2JvmxQ0MbtRlJSw1dSQBrSGqknipoQT4LB3PM91szxBe1767RuYYnUxVutrUlDEIqWxBZNWyRNFXUgjwhE0mnaRsc6o6Fshhj4sUfWsaXx1dJDSz2teJ9JJPLTyWJ4zHCplYbZghm4m2V6X5/0xl3N/T/bH0dq6iasxmCWdzmU9RTx05DImmFstFDM4CzeMQ+Qm7r7gs7RbEzVQvLxaWCoqKSewsDLTSuic9o3MeA2QDcHhY2EYTNRz4nVPlFSK2aOobBFTmOSMxU0VM2JrnTOEtxC03Ibm47tl/RDC30sDxKQZ6ipqKyfVN2slqZXSGJrrcZsbS2MO36l96k/x+f62yvt6/l+tNuPCK0+leONoKdsmpws008dLSQa2rw9VUP1Ioy6x1WbXudY2axxsbLcN2lcJ7J0Lv8Q0XlN+Bixt4lPvQ+XDq2OnLv8AquAHSQN6nnPrP7rfC32v9m50rra2hwirqmyU8lXS0lRUufLE8QF0UT5S1sUbg7U4uqLuvbMknbewCsqqnC6SoBj7rmo6echzS2J0r4mSOYQM2McSRcXte+drLXezRUth0dxx73NaBhdcAXODRrOp5GtaCffFxAA5Stxoa0DDsPDSCBRUgBBuCBBHmCNqTrcvt/7f6S9O79//AF/2vYBikdbTx1MVw19w5jra0cjHGOWJ9vfska5p6WrPXCewqXmkxGQ63BS49jL4L7DF3dIy7fgF7ZCOtd2l8r7S/MWec97Pi6ERFFQoVSgoIWr0v/R2IfIaz6PItotXpf8Ao7EPkNZ9HkQbL9Z+5/UqyqPf/ufWqyogihSgpkka0FziGtaCXOcbAAbSSdgXJMoXYvK6ecOZQxazaOO5aZXkFpq3DaANrfJ036qqgZKx0cjQ9jhZzXbCNtj0KxDiNNriFs8HCAZRNlj1gBYW1Abi2QXL2jinJZM7O76f8r5S+08ded+nXu7Jz5cWOWXFL3/+X/GednvfDflPDx6W8AqXyQ6spvNC90Ex5Xx5B9t2s0td+8tiseCijZJLK1pEk2pwp1nWcWN1WnVJsDbK4GayFv48bMdZfr0++vH3c/PljlncsOkvXXpfOfSXw9haXS88HDHV5/kM7Kp9szwADoqnIbbQSSut8FbpUTRte1zHgOa5pa5pFw5rhZwIO0EEhZtNj5nHVz0VBXYpGX1FLirqyoLY/dDSSTOfHSVEYHhQOYItce9Njyr6NhVKIYIIQbiKGKIHl4NjW39CwXT0OFU8UILKeJoLKeBus57rknUhiF3yG5OQvtWA52IYhYN18MpDtc4A4hM3LJrc20rTnmbu2bFWE6dIzcd0jgpXCFrX1NW4DUpKdvCTG+x0lsoY9+s+2XKuT0pwSsxGOKDEJiyStcY2UFK8ilp4m3fNPPILPqZGx2GZ1NZ4sCu3wbB6ejYWU8QZrG73kl8sjjtdLK8l0jr7yVrtHX91zz1/6u7qOkuNsMMh4aUX3STNyPixNVLN9KxvYrkb/hVLThrI30LXUE8TCCIpqRxhe02G06of/wBQFdSuQqqSTC6yprqeB89FWgS10EDQZoamMEGsijv7sHssHsbxrtBF9i3eBaQUVcwPpKqGYb2teBI0+K+J1nxu6HAbFNLK2ix8SrI6eGSeZwZFEx0j3Hc1oues7gN5K1+PaTUFCPympiY8+DCHB88hOQbHC273EnLYtCMNqcae2SvikpMMjeySGgedWoq3ts5slbqniRA2tDtuM9mbRvyijQuCSmikxOQFjcTnlq6yIixibI61NNbaNWFrA7oIO5dyDfMZg5i2zNQ4AgggEEWIIyIORFuSy0VJMaGVlNIT3NM7VpJDc8G859yyOO7bqE7hbcqTo3yIgUURERREREERFQREUEhQza7rHzKVS3a7rQa+vP5RD1j/ALgtqsWWnDnh+9treTNXiHcvoCo5aj0CpaaSd9BNVUDKp8UtVDBO8xSSxTtmdMwSlxhmks5j3MI12vNxexHWK1qv8b/6tU6rvHP8LfsTZpWio1T4x8zfsUhp8Y+Zv2JsUTU0b/Djjf8AGY13zhWBhdKHB3c1OHNILXcBFrNINwWnVuDfesu3SfR9iW6T6PsQYOkFBLU00sENXNQvlaWiqp2wunivkXRcOxzA+2wlpsuW0p9juKuweHBxVPpo4H08ramGGI1AlppWzMmbr8VszpGlzn2JcXuOV129uk+j7Et0n0fYgtUEcjI2NlkE0jQA6QMEeuR77UDiGk9B8y4+j9jmkhwfEcIicWsxN2ISVE+qNd0uIPke59t+oHtaByRhdrbpPo+xLdPzJeqzp4MLAsKgoaWCjpmCOCmiZDEwbmMAAvyk7Sd5JWYpt0pZW3aSaiFIUWPL6P7qQDyjzf3U2IbtPWsHGsNhrIHwTA6r9Vwc0gSRyRvEkU0bj4MrJGte124tCzwLKSorANCyWFkVWIastA1jLDGWvcMtfg3awa49HKdit1eHWpzT0Zjo2u4utDEwcEx3hmFjbMbLbY4ggE3sbWOxSyuzTEwqghpYIqanYI4YWNjjYLnVa3lJzcTtJOZJJWWUsORQluyTQiIoooKlQUELV6X/AKOxD5DWfR5FtFq9L/0diHyGs+jyINn7/wDd+tVFU++/d+sKtRFKt1U7ImOkkcGMYC57nGwa0bSVdK5nSeM1VZR0BPuBa+rqW/5jInNbFG63vS+/m6Fq5+S4Y7k3ekn1vSfb19nT2XgnLyayupJbb7SbuvfynusRQz4s4ySPkp8NuRHCwlktWBlryuGbYjuA/utkdEsMtbuGn5LhlnnK1y8cYnput01oAAAAAFgALAAZAADYFUtOHY8PHOTPK+Ns38b8J6Rv5PxHl3rit48Z4Y42z51431t/s5vuafDQXQumqqMXL4JHGSenZ41O85yMaP1bs7DIrf0s7JWNkjcHse0Oa5puHNOwhXFxEuPx4ZiD8NEckz6xvdOH00YzLjrd0s1jxYoQWl9ycrmwKyxx/ZZST+m9Nel8fj+3TXRjlnO0YZZZf149d/8AKbk6/wDlNzr5ze+rtnvDQXOIa0C5c4gAAbSScgFzjMflrXFmFsa+IEtkxCYO7laQbObTtFnVUgzzFmDlKobgNVWv18UlYYLAtwyC/c4cDk6omNnVR2cWwbfcV00UbWtDWtDWtADWtADWgbAAMgF1PP61q8JwCCCR051p6p4s+qnOvLbxY90MfwGABbZEUVodM62RscVLA4tqa+UU8Tm+FFGReoqB0Rxaxvylq3FFTMhijhjGrHExsbG8jWgAeXJaDRdoq558TfmHOfS0WdwykieQ+Ro3OllDiTyMaF0qtSeotLi2imG1bteooqeSTP3XgwybPaeGjs+/lW6RRdbaXBNFMNon8JS0VPFLn7qIw6bPbaZ93gdF963agIqBVjEaOOoifDKLseLHlBGbXNO5wNiD0K+pKK0uBVsjXuoqpwNRE3WjkOXdUGwSgeONjhyi+9bpa/GsNbUsFnGOaM68E7fDik5elh2FuwhWcFxQyOfTztEdXCAZGC4ZKw5CeEnwozybWnIoxnTo2yKApRRERBFlKIgIiIJVLdp61UFSzaetAm8E9SuFWpvBd1FXCrAVqqqGRNL5HtY0EC7jbjOIa1o5XFxAAGZJVNTWQxFollijLzZgkkYwvPI0OI1j1LncRqhLjGFRBzX05osRrI3NIdG+eJ9DBE9rgbOLYqqYi3j3VGzbpJh5q+4BW0preNak4ePug6oDnARX1iQ0gkbQCtlBO2RoexzXtN7OaQ5psSDYjLIgjyLjsUbRVlXRRRy0zYcLxA1MjxLHrurtWWJlLENbWMhkqC6R3KA3Mudq52jU5GJ4zTt/MxPopmjc2eppy6oA5L6kTyOWRx98pOpXT3S6hFRVdFC1mOY3DSGBjxJJNVSOipqeEB0sz2Rulk1Q5wa1jY2ucXuIAyzuQCGzRaWk0lppKajqSJ4u7tUU9NNC+Ore9wJMZpzxmua0Oc4nJoaSTbNbpAugcuap8UnrKrEYqZ7I2Ye9tGNdusJa2SniqXvk3mGKKohs1pGs4uubALX4G7E21+I0Tq41kUVJSzQVM1NTxvgq5nT61LL3M1jJY+DjiktqhzWyjM3BU3B2usp1lqMAxuGqpqWe7Y3VTSGxucNbhWB3DQt8dzHMkGXiEraqipxVAddH7FREoqtERRRERBCIiAoKlEFK1el/6OxD5DWfR5FtFq9L/wBHYh8hrPo8iK2fvv3frVRKp99+79aqUQXPgWxkl3v8Nsw//HU3eOvjtK6BaXSiF7eBrIml8lI4ucxu2SneLTsA3usA4dLFo7R/T3vSy/bz/J19iu87h4d7G4/e+HzdT7t0pViiqo5o2yxOD43i7XDMEfUejcry3SyzccuWNxur0qV8o08wls+PMrHvmEFFQMp6h8Mro5KU1MzjHVQOb4MjS4F17gtGzJd9pFj0dKBGwcNVyZQ0zDd73HYXAeBHvuVb0bwQxU8oqrS1FZrPq3bnF4I4MfAa0kDyrny5O/yTDHyu8vbp0n1vjr0+ztw4f2fBlyck/qmsJ69Zbl9JJrfnb08Kwxi8+HFkeIF01KdVkeJtaDYnICuYwe5k5e6NGqd9l08bw4BzSHNcAQ5pBBB2EEbQuf0SqDqz4dUceaiIjJe24npJATTS55OBYNR3Sw32q1Lg9RQky4Zx4jd0mGyvtE4k3LqWV3/LvzPFN2noXU86V060OmtU8QNpYXatTXv7liI2sa4XqJstgZDrm/KWrLwLGoasODNaOaPKanlGpPC7kew7uRwuCtfgZ7rrqqs2xU96Cl5NZh1quVvXJqx35IikLd9G8oKVkEUcMYtHExsbB8FgAF+nJX0RRkIi0WJaY4TTEioxKhiI3PqYQc8tmtclVG9RUQyNe0PY5r2uF2uaQ5rgdhBGRCrUUREVQWsx7DDO1r4ncFUwnXgltsdvY/licLgjpWzRDxazAcWFSHse3gqmAhtTAdsbiLhwPvo3DMO5Fs1qsbw57y2opnNjq4gQxzhxJmbTBNvMZOw7Qcwr+C4k2pYSAY5YzqTwu8OGUC5aeVu8OGRGaJL5M9FClFERY0lfA06rp4Wu8V0rA7bbYTfapcpPHolyk8WSihrgQCDcHYRmD1KVVFDdp61Khu09aCmfwXdRSun4OOSQgkRse8gbSGNLrDpNkqPAd1K65VXn2bH45dFn1kj46vGNK2x0z3MLZW0MWKSupoIXuBIpqOmhe4AGwdIxxzc5xX2mhwWl7noGwcVlCyIUckfF1Y2xCHVGVnRPiyIIsbgjMNIyHYHRFhjNHSGMytnMfc0OoZmG7JizVsZQRcP2hbBWdGOmmj0Vwtk7apmG4eyqY5z2VLaKmbOx7gQXiVrA8OOsbkEE3PKsvB8NZTNk1SXyTSunqJXeHNM4NaXutkAGMYwNGTWsaBsWaiK5f2R8VipaeF0uLPwcOmsKhlPFU8JaN5MLmzQSNa332tYHiAXzseCOn9Gz/wDvsKb8roqBnn90iX2UussOqkaciAesA/OoulvR2rE1JTy91Q1gkhY8VdO0MgqQ4awmia17gI3AgizndZXCYY19dpZXTCrmMGCUVPRsYBTuY2qxIirq4/zVwBTxUYvfX4541iukxbHGUxhZwcjxI9zLxBmpCGxvfrykuGqy7Q0aoJu8ZWuRyGCxspTWywulHdtZLWVUjn8eWeRrGBoIAtGyKOONrdgawbTmeXm7Vjx5et69P17Ovh7Fny49PDp1/wCvu3WhlZ3bpFpFLJxv8LNDhVKDmIo5KSKvqnNB8F8kk8YcRtFPHyLNxOqxsVvBQT4UY5JQ5tM+kqnzw0QIa6eoqmVbWa5IfqtEYuSG56rnDVUlaM3NawEm5da7nEZXc45krZ4diTo3ucA27y0yXHhlo1RrOOYIAsDsHIsMO2TpuNnJ2DOb0yYcIqoJcXbSyMhfiLxWU1VJDw8VPVGmgpJGywCRhkA7njlaNYA67hfi52dHKHFaCKXuqekruL7jHRUM1LLPUvPGlq5qirm1iSBd/FABJN7ADpKWqbI0OacjuO0HeD0q8HLrllnT0/JxWavX1/NzMOjPBYIKB8l5oqdzxUMFjHWguqO6YbjiltQS9vQADfNZnsdY8cUwnDcQIDXVlHBO8DYHvjBfb4Ote3Qo05xJ0FDO2FvCVlRHJT0MANnTVUrHNjGWbY2k6732s1jHE7Fd0EwFuF4Zh+HMdrtoaSCm1/HMUbWOf5XAnyrKef2/z/r8mFnh9/8AGv8AP5t1LsVMYyUy7EbsSqlSihRRQpRBCIigKCpUFUQtXpf+jsQ+Q1n0eRbRavS/9HYh8hrPo8ig2Q8I/FHzlVKkeEfij53KpARSiI0lVo83XdLSzTUUjzrP4DVMUjvGfA8Fhd0ixVh+C4i7iuxZ4YRY6lJAyTyPaeKekLokXNl2XjvrPpllJ8SyO3H8Q5pOtl98sccr85S382qwLR+no9Z0bXPmfnJPKdeZ5O27zsHQLLaopW7j48ePHu4zUc3NzZ8uXfztyvrXO6WN7mkgxNrb9zXiq7Xu6ilI13WHhGN+rJbkDl0DHhwDmkEOALSMwQRcEHeLKJY2vaWuAc1wLXNOYc0ixBG8EXWg0Nc6ATYdISXUbvcXONzJRykup3X3lovEf/jHKtjTOlYnsj0LHxRPi1o8TfI2CgnicWSslk8Iuc3w4GsDnOa64sNys4BiZwqOGgxJrIWMa2OGvZrCkqXE7ZS6/c07iS4hxsSTYrYYe/uvEZphnBQNdSRG2TqqSzqp7T8FupHl8Jb6qgZKx0crGyRvGq9j2hzHA7Q5rsiESTzi40ggEG4OYIzBB2EHeFK5T/BKvDyX4ZJwtMLk4ZUO4g3kUdQ7OAncx1257lqtMNOYe4aiCNz6XE5eBo46SccHPHUVpEbCwniyhoc52uwkcVNL3mdXRf43O6EPd/hEALZzG9zP8QqNazoRIwg9zx6tnW8Iu6ARtzojhZhFOcOojCCCIzTREXGx3g31unas/BcNio6eGmhbqxwRtjaOWwzceVzjdxO8krMTZI42XQCCC78KqKnCZdbXtBI6Wme61vdaSdxY5vQLLP0cxmoEvcOJMijrQ1z4pIb9zV0TTxpINbNkjRbWiOYvcZbOjWm0xww1NK7g8qmnIqaSQDjR1EPHZqnbZ1iwje15CbNa8G5RYOA4i2rpaeqYLNniZKAdrS5t3NN94dceRZyKIiIC1GN4bIXtqqQtZVRgNId+bqYr3MMvT4rtxW3RCxhYNiTKlhc0OY9jiyWJ+UkMg2sePmOwhXcRrGQRmR97DINGbnuOQa0b3Fa7GMKkMnddG8RVTW2c0/mqpg2RTjlG5+0X82swzGWV2JMh8B1JSmaWmcRwkVQ+QR8YbSGgGzthutPPnccf5fG3U9t+f2nVp5c7jNTxt1P+/s2RoamquaiZ9PEdkFOdV9t3CzW1r2yLRlkq6XRnD4vBpIbm13ObwjzbeXvuSelbdFjj2Xj8cp3r63rf9fSaiTs3H45TvX1vW/6+2mnqMMfBeWicWEcZ1K43gm5QAfzL7bHNy5Qtjh1W2eNsrQQHDNrvCa4ZOY4bnA3CvrVYYNSqq42+A7gprbhJI2z+q+qCp3ZxZyY+F6a9Lq3c9PC7/wDqd2ceU14Xpr31vp6eHVtlDd/WVIUN39ZXS6FM4u0gco/7hf0KsvHT5j9iIqI1uvzH7Ev1+YqUum10jW/FioL+vzFCVZlkU2aW6qoAG/zH7Fz2LYmBkCb9TvsWZilVqgrlqmUk3KwzybuPHbCxao1jt+fac1FG1pYQZA7VaQDm0atyeKHZ8qsVYuT1rDqAbEZ3Xh8meuS2vp+LCfssZOjaYNcst0nP5ltKcHWuSuPgqKiPwH5dIHzrIjxSoc4BzhYeRbJ2nD0rVycOdtu47zRmstLJETk4m3xm/wBl0bZByjzrhdH32ljffa9tz5c/QV3i7ux5Xua93kduwkz36xLWMLg/VaXhpaH2GsGkglodtDSQMugLIasbUb4o8wVYib4oXbtwaXZTkqm7FQANiqBURKJdEBEUICIiAoKlQUELV6X/AKOxD5DWfR5FtFq9L/0diHyGs+jyINi3wnfFb87lWqG+E7qb/Uq0QREQSoREEoiIC4n2Vq5+HQsxWAt4eG9MWH9dHU8UNsBmY5NWUDkY5dsubhibX10sjwH0tEH0sTDYslqZGgVMhG/UYRGPjOVjHPrNRtNGqBlNSQQscHhsYJkH617+PJLffrPc53lWwXO6GSOh4fDZSS+icOBcf1lFLc0zr7y0B0Z5NQcqzsVxfg5BTQNE9W5usIQ6zYmHZLUO/VxX8p2AKLuaXMexeOkjD3hz5HnUggjF5Z5TsjjbvPKdgGZXEaZ4JwrsKrsRDZKluL4eIowS6CjY+X83GNkkhIZrSHaQLWAXZYTg/Bv7oqH90Vjm6pmIsyNp2xU7D+aiv5TvKo00wo1tDPAy3C6rZKcnY2ohc2WA33e6MaL8hKrGzfVuFK1GieOMxCmbMBwcrSYqmB3h01SzKWGQbWkHMX2gg71tlGaVDnBo1ibAZknYAMyT0WUrkNKa99bOcHo3ZuA/xOoYR+R0rhfgs/2iZt2gbgSeqpWv9jPDap+F08rMQqYWyuqJY4nRU0rWRSVMz4w3Xj1g3ULTYk7V0b8MxC3FxQ3y8Kipndeyy3FNCyNjI42hrI2tYxoyDWtAa1o6AAFcTad1ou4cUGzEYHfHoAPOWThQIsYF/dsNdyEwVLerISn51vkTZpomx4uTxpsNYPgwVLyf4pRZQaXFiR+W0TW2z1aKRxvy8aoyG1b5Q5wAJJAABJJNgAMySTsCGnOz0mJxtfI/FaZrGgvJfh7WxsYLk6zuHvYC2d9y0Gg1HWPxaqxCrFO01FKxlOY4XRSzUjHAMfIHOcWu1gHapOxw8nSMacQkbI6/cEbtaNhuO7JGkFsrxzZpHFafCOZyAWXjLHMdFVMaXGAuEjWi7nQSAcJqgbXNIa63wStHaJ/LL6WX/v8A7aObHpMvSy/r6Nmiohla9rXscHNcLtcDcEHeCq1ul26BanBCXzVk2Wq6RkbDyiJtiekXO1W8SxAzE01IQ6Q5SSjOOFux13t/WbgFtKKnbDGyNuxgA6Tyk9JOflXLMv2vJLj/AE476+t8NT6Te/f7uaZftM5rwx8/fw19vNeCN39Z+dFDfrPzrrdSUREBQShKtuKCHuWFUyK/M9aqvmsCsayjUYzUXNrrTyFZNXJdxWI9asq6cJphSnM9Z+dWQ2+3zquZ1ta/KfnVl0i8Hmv81fRcX9MXWw32DMDz/YoazdsPSoikPIrjX52sLnesIys02uF3aB0EEeRfQY3awBGwgEdRF189oXbjvXcYQ/WgjPwbfwkt+per2HLxjx/xDHwrMCraVQpC9F5dXbqbq20qq6qK7pdU3S6CtFSpuiJRRdLoBUIiKLVaYfo7EPkNZ9HkW1Wq0vP/AA7EPkNZ9HkQbJvhO6m/1KtUN8J37v1qooxSiIgIufxLSImR9NQRd11Lbhx1tWngOz3aTYSD70Z5bVRSQYy5ofLUUcTxf3FsDpIiNxMmuHAnoXLe143LWEuXvJ0+bZN+0d8/D85j3uTLHDfhMr1vv3ZLde9kjoyi08eLSROaytibDrEBtRE4vpnE5AOc4B0LidmtkeVbhb8OTHPw+POfZy8vBlx+PhfCzrL9LP1Gs0krnQQ2izqJ3tgpx/6shsHkeKxus89DFk4RQMpoI4GXLY22Ljte4kufI7lc5xc49LlpWVMctbUVUrmtpcNaYI5HkBgqHNBqpAT4rCyO/KXAKpj5sTbkJKWgdazuNHVVbegbaendy+E4cgK2NErntO8WkFRFVYeSG0pdTYlWtY2SGClmcwO1QT7vNE/VfZtw3O+9drgmGQ0serFdxfx5Jnu15Z3n9ZLIc3k+YbrK6MPgEBphExsBY6IxBoDNRwIc23SCfOtVoVO5sctDKSZsPfwFztkpy3WpZeoxEN64ylSTVdAigIozc7jujJkn7toqh1DXBuq+RrGyQVTR4LKynNuFA2BwIcNx2LDdW6RRjVNBhlQ64AlhrpoGEb3GKaEuG7IOK65ArtjpyBw/HqsWqK2kw2InNuHRvnqS3e3umps2J3wmsJW90bwKmw+HgaZhAc4vkkkcZJ5pCSTJNK7jSPzOZ2brLZoi6ERQoqURQ5wAJJAABJJNgAMySTsCCJHhoLnENa0Ekk2AAzJJOwWWk4M4jZzw5tCCCyM3a6sINw+UbRT5XDPfbTlkqYNbEH67gW0DHe5MORrHNP52QEf8sCLtb77acrBb9Vj4oAAsBkBkAMgByAcilEUZNO/CZYnufRStgDyXSQyMMkBedr2NDgYncoGRUuw2plyqKriHbHTxiIEcheSXW6Ft0Wi9l47669N3XxvTn/dsPfXpu6+NrNHSxwtDImNY3bZotc8p5XdJV1Si3ySTU6N+OMk1BQ36z86lUtPzn51VSVF0KpLkUJVp7lLnKxK9QWal60WKTLZ1cmS53EH3KxrPGMN5Vp6uFW3Baq6MWtrPCP43K1C25zWRiLbOaeUfN/useLaLLxefHWde1w57wjJYwKWxtJyPz/YqoyrzWD8Ba5G21fozs8y7bAHXgb0Fw9JP1rh4mjkseorrdFJsnxn449DXf0ru7FlrPTg7djvDbdqURes8aiqBVKIi4CpurYKqCorul1TdCUFd0VKlBKXVKIJK1Wl/6OxD5DWfR5FtFq9L/wBHYh8hq/o8iDZt8J37vzKpUM2u6x8wVRRilaPSSqkc+KhgJbLUhxklb+pp2+G7oc7wQevoW7WkwNmtWYhMRxhJHA05ZMjjBIHIC43XP2jdkwnTd19vG/Otfd29j1j3uWzfdm5Pe2SfFu/fWmywzD4aaMRQRtjYNwGZPjOO1zukrLS6LfjjMZqdI5M88s7csru3xtW54mvaWvaHNIILXAEEHaCDtC43FMfbg0VbHM+0cUPDYeZHXLi8iMUoJzeWyubYbdU9C7UFcP7JWjcGI1eEMku2WGplqIJAA8RuiY2TjRu4sjSWNyPJuutPNJLjn4WWT6y3Wvz+XV2W3OZ8XjLjb9LjLlue+pZ7y6ZGimCPnp6Z9YxzYYwJIaOTa6V3HdU1g/WSmRznBhybcXz2ditJR4tLE9sFexkUjzqxVEZPc1QeQF2cMp/y3X6CVu10VxTXkLm8dtS4hR1uxk//AA+pOVvdDr0j3dUwLL/+qukC1+keGispZ6YmxljIY7xJRxonjpbIGu8iRMmwUrU6JYmaujhmcLSgGKdp2sqIXGOZpHx2k9RC2qjJKKEQSihFRN0UKUALn5P+JSFgJ7ghdZ5Gytmac4wd9Kw7SPCcLbAb3MTqXVUr6GBzmtZq92zjLUY4X7nid/nvG0+9aeUi25pomRsbGxoYxjQ1rWiwa0CwACMfFWBbIZAZADYByAcilFCMkooUoCIiAii6ICpbs8p+dVK005eU/OiqiVQ4qSVaeVBTI5Ysz1clcsSZyDDrpMloah1ytpXv2rUPOa15NuK25UqoqAsG2MavjuwHkNvP+AsONllt547xv6Bf+HP6lrmDLoXmdrx/n29PsuX8ml2Nt1fY07ViSS6oG87gq45LjM+QfauZ0zfizGG+zby/jctlgdSY5WuOwHPqOR9BWlin1T+As2nlucllhn3btjyzeNlfQrKCtbo9W8JHqO8Ng87NgPk2eZbMhe7hnMpuPAzxuN1UIUVJWbBUgKpS6C4CpurYKkFBcuioupBVRUl1TdEFV1qtL/0diHyGr+jyLZrV6X/o7EPkNX9HkQbWPa/rH/aFJVMW1/xh/wBrVWjEWmoiIa+oiP7SxlTHfZrMHBStB3nJh8q3K12N4b3QGOZIYZ4Xa8EwGtqu2Oa5t+PG5uRatXNjdS49bLv/ABfy/N09lzxluGd1Mprfp5y/Mm/bbYqVpmYyYQG1zO5nXtw2b6V58YSj81fxZLeVV1GkdCwZ1ULuQRvErieQNiuSVP3jj11sn16X8z9y5t6mNy9LjNy/SzcrbLnKKXurE5JWZwUcRga8eC+eQgyap32bxT/dJ5KuvBjjZJRUrrB88g1amZp2tii2xAj3zs8/It5h1HFTxthiYGRsFgB6STvcdt1q3ebKa6Yy7362eGvaeO/Ppp0TGdlwy713nlNanXuy+Nt9bOmvLrvV6KqyljmjdFKxskbxZzHC4Pk5elc+ZZ8MsH8LVYfsEuclRRDklA409OPGF3N33Ga6YqF2PN0t0s7JWNkie2SN4DmPYQ5rgd4I2q6ucrMOqKN7p8OY2SJxLp8PJDGvJ2y0r9kU3wTxXdBW1wbFIauMviLgWnUkjkaWSwyDayWM5td6DtF0JWmwz8jxSppySIsRb3dT8gqYw2OsjBvtLeCkt8ZdOuc0/Y5tKKyP89h0jaxnwo47iojPQ6AyDyBdBTytkY17Ddr2te08rXAOaR5CEqT0VotVpDj1PQtaZnF0kp1YKeMa9RUP8SGIZuPTsG8rQvj0irLubNR4PFfiR8D3fVEbuFLi2JhOXFbe3Kmltdmi4ulxfFsOaHYyylqaa5D6/DmTNNON0lVSvueC5Xx31d43rsaeZkjGyRua9j2hzHscHNc05hzXDJwI3hCVWtPpDWy3ZSUptVTg+6Wu2lhGTqh4ORI2Nbvd1FZeMYi2mj13DWe4hkMQ8OaV3gxtHkzO4AlWNHsNdA18kzhJVVDuEqJBsvsZFGD4MUbbNA6CdpKF9GVhdDHTRCKMGwuXOOb5HuzfJI73z3HMlZahFFTdQilACKEQSoRFQRFF0RKssOXlPzq5dWYzkes/OoyS4qzI5VvKx5CgtSuWHO9X5XLAqnqLGBWvWuKyqtyxSsK24oUsCgBXGBYM4yaePWy5Rbz5LQTOsAOQZ9YXS0g2LmKq4fIPFe8eZxH1Li7ZPD7u3sd61akeAC4nIbT9Q5VZE5J5B4t/nPKseudfV8Vpvbld/Y+lWKZ5vbK983HO18wAN5tZeZXq4t7A8W2ehXqeoGtqm43jLMhayF9wRncecjk6lk07DfWtnsN+vLP8bVNrljI6PD6t0T2vGYBB6xvHlC7eOQOaHNN2uAI6iLhfOqOS425C/n+xdXoxV3aYidnGZ1bx58/KV6fYeXV7teR23i//AFG6KgpdQSvTeaglUko4q056oua6nXWK6RU8KoaZweqg5YDZleZIrtGVdTdWmPVd0ErV6X/o7EPkNX9HkWzutXpcf+HYh8hq/o8io20O1/x/6Wq4rcHv/jn5mq4jBClQiIOAORAIO0HMHrCtwU0bLlkcbCdpaxrb+VoV1QpZGUzsmpUoiKsUogRFFp8YwYvkFVTP4CsY3VElrxzM/wAqpYPDZfYdrd3ItuEVSzbT4RjEdXwlLPHwNU1hFRSSWJLHAtL4zsmp3Z2ePLZajBsY7gw2pbOC5+GSyUbGA8ecAt7ha3LwpGSQt67rfY5gsFWGcIHMljOtDURHUnhdtvG8br7Wm4O8L5hiEOKjSXDKKqYx9LM4Vb6qMDUqn4ZHI6KSSIu9wlHCtY4C9yGkZDKsLbK7/RPR7gC6sq7TYlUEvmldxhAHXtTU2t+ahY06thtsTvXRJdAozk0ELjat0WAymfWEWEVL7TM95QVbzZkkTfe08ziGFgya8tIycV2S1ulOGxVlFVU07GvilgkaQ4AgHVJY8X2Oa4NcDuLQUiX1YuDUr55RX1LHRv1Sylp32vTQu2ueN1RIACeQWbyrerT6FV7qrDqGoffXlpYXvJ2l/BgOPlcCfKtvdFkSihEVKJdQoJRUqpUFCXUFECUuoJUXRU3WPEcj1lXrrFhOR+MVFS8rHlKuyOWNK5BYmctbVPWXUPWrqX7VKzkYszlZVTiqVrbIkK7ErLVfiUZM+kC5jHG6lTO073646ngP+tdRSLS6e02q6GoAuHAxO62kub5SC7+Fc3asd4bdHZctZ69XPVLbjV8HkI+rpWPEC0ltjla24EnlPKs2KRjhnby7lXwTPeuNuTIjI7iQvJsetM0wM2F20eQD7Vd1RfLMjzelXIImgZXJO3credzs86mm3HLbMgk1Tby2W7wqo1HtfuBz6jkfQStNTMGR3rPiP2brehbcL3btz82Mymna8KnCrVYfPrRtzzHFPk2eiyvmRe7jlubjwcsNWxmOerEj1Z4RWqqoaxpc9waBvcbD/dXZMdq5JFaMi1jsbpybBxPSBl6VVHiMTtjvOFhOXH1bLw5zyrYtkV2OVa9sgOYIPVmrjJFnK16beGRZbXXWngkWdw7WNLnuDWtFyTsAV2x0y7rSaZVkTaCvaZGB3cVWLawv/wAvJuGxaXFsedOSyMuZFfdk5/S48nwfnXO6Ry/kNaBzSp/0Xrkz7Zq6xduHYrZvK6fWqf33x3fUrit02x3x3fOq12vNEQoiClQiKqCKApCoIiKAiKUELk9MpW02IYPWPB4Php8PkfujNbGOBc7kaZYWtv8ADC6xa/STCY66knpZcmzMIDt8bxZ0creRzHhrh8VWJWxS653RDH+HBpKpzY8SpbR1UBNi8gWFTCD+cgkHGBGy5G5dEii5/wBkPFhSYdUvA1pZWGmpox4UlRUAxRMaOW7tbqaVscaxmloma9VURQN3a7gHO6GM8J56ACuXwGlnxWsjxSrjfDRU2t/hdHK0tkc92Tq6pYfBkysxu4Z9JRjfR1OjuHiko6WlH7PTww9Zjja0nbygrOUKUZIU2UKVAUIgQERFQVKkqklAuoRRdAusOE+F8YrKJWBA7wvjKKrkcsSZyvSOWHUORYxKl61tQ5ZVU9a+R2awrZIpVN0JVKxZxWFeiKsXV6NRWxpSr+O0XdFJLGPDDeEj+OzjADrF2/vLFpitxSPS4zKaqTLu2WPlVOeo+g+dZ0R32PkOfoVWkdD3PUyR2swnhI+TUdcgeQ3b+6saB3k8tv8AdeJljZbK9zGzLWUbKM5bT1ZqBYdJ+3/dWWv2fUeVXRfMDdy/YVi24xmUrhl+Niyy4EDZ+Ny1sbcwb79ltqzY37fQPx1JFs22+FTWNr5O+cf2utg5y5+KXPLIi32hbIVAK9PsnJvHu+jye18Wsu96oxjE2U0ZkfmdjW73O5OrpXz/ABHF5ah5c92w5AbGjkaNwUaV4rw0zjrcRnEjG7La7y7fMtNTTbDyny7dq0c/P3rrydXZuCYY7vi31LcjMrY0rrWP4K0cDyR6LfjrWZBMBlflt5vnWmZt1m2/pJtXNp8m7qIW2p6kO6DyfYuZhk6PTs6FlwTH7F0cfNcXNzdnmXV08Ei5/SnGNd3AMdxIzx7HJ0g3Hobs678gWPj+PGlppHgEyABrSMwC421id1r38y4uixEOLc7km5zv6U7V2ma7uP3Ydm7NZe9l9nXUcmWataRNBoa35JU/6L1h0tTewB38qyccyoa35JU/6D1xYZdXblOj63A8NadbLjPO/e429Ck1DPGHpVxF9E+ZW+Hj8dnnCkTs8dn8QVWqORNQcg8yCnh2eOz+IfaqhIzxm/xBRwY5B5kMTfFHmCCsOHKPOFIPUrXAs8VvmCcAzxW+YILyKzwDPFHmTgGeKFReUqzwLeQ+c/agiHT/ABO+1BeuitcGOV38TvtTU6XfxFBrtIdG6HEA0VlNHMWeBJxmTM+JNGQ9g6AbLUs0Dpmu4tZizYrWNP8A4lUGG37xLx5HLp9X4TvOPrU2PjHzN/8AFNp3Y0mG6G4ZTvbKykjfMw3bNPrVEzTytknLi09Vlvyrdj4x8zfsTVPjHzN+xNkmvBWioIPjHzD7FFj4x8w+xFXEVuzvG9DU1XeOfM1QXEVrVd458zfsTUd458zPsVF1QVa1Hf5jvMz/AMVSY3f5j/8A6f8AioLypKtGI+O/zj6gqTD8OT+M/UgvFUlY5p/hy/zH/arbqYePL/Nk+1BlErV0r83daumnaM7yZcsjz6CVbIDb2RUSuWDUvV+Z6wKhyjKMSpcsN5V6dyxnFYVsiSVTfNQSousWcVAq9GVjq5GVFbGnK2tK5aanctlTOWUY1rNPKHhIBO0ceC+tymJxAd/CbHq1lxcTt6+nvAc0tcLtcCHA7CCLEHyL5nWUpp55ITnqOsCffMNiw9eqR6V5/bOPV73q9HsXJvG4+jJjdv2rOZnYXtZayF/Ll6Vl08t9/V5OlcWnbKzW59Y37rq8xwFr7OjbdYolABzuqIprkgeT8blG6TbYwvztnc7bqzj8z2U8jmC5Dc+hpycfILqGAjPL51kNOtt2EWIIyO7zLZjlZ4NPJjt8vnnuBne5PnyurtEL7f8AbJNKsPNHNqi/BP1nxE8hObL8rSbeZThDgbH0LXazng3UDMhY/j8WV12WefSPQqoNUAW/H4zSd17jfu6cv7I097qvwzbuv+y2ELrZ8oWjguD0Z+kLOp6q23qWeOWm3u78GdVxiRha4BwIsQd4IzHSvkuNRS4bXNidfueQOfC+/IeNGT4zbjrBHSvrAlB/tyLT6W4Kyup3wk6rjxopLXMcrfBd1bQRvBIWHJN9YvH/AC3V8GnwSuD3xhuw5rp8eqB3DWDf3JU/6L18p0PqJoq2WnqAWSU7dVzTmLk8VzfGYQLg7wV32LVgNFVj/wBrUf6L1OOsebF97REX0j5cUqEQShUIgm6KApVEqVSpQFKhEEooRAupUIglQpUICIiApUIgIqUQCUuipKCSVSShVJKCHFWnFVOKtOKC3K5YkjlelKxJXKKtTOWDUFZMrlgVDlKyjFlKsEquUrBr5w0W5fx+OpaOXOYY96unh4ryZTGJmqdzfP8A2WBUSPPv3eQ29AVmWvjZm57WjlP4ueoLX11ZLqueGOY3c6YiIPHK1r7OG7cRntXkcnNnn5/D6Hi7Jhxzwn1qmvqZGG7ZZGkfDdY+lYtLpXUROs/Vlbf3wANuhzfrutHWY0C4h2R67jzha2tn1swVrxzynhXRl2XDKaykfXcC0lpp7Au4KQ+8kIAJ+C/Y70HoXUwuXmplZIw7bjpXV6M6f1FNZjrSxDLUfc2HwHDNvzdC7uLtnlm8vtH4XZ14/h9xMi5fTalDuDnAzb7m/wCKTxD5HEj94KvAdKqWtAEb9SS35p5s6/wTseOrPoCysScHsew7HNLeq+/z5rq5JOXCyPO4+9xZzc05KJx3K6GEEZ/YOlWaRpzB2glp6CDYq+8HqXkT3ex5r1LKcrm4vmculZVMTyC9/QsGM2tycmy/2FXYZyLAdZz+rcpW3Fumi4zy6EiIGRKx6aQkXv575/2Vx5uPrUlZa30Yek+FNrIHQkgO8KKS35uSxAJ+Cb2PWvnmBslhdLDMwsljeWavLsIc072HcV9OZJcZG9lYq4IprEtBeMr2z6rrK495ptuM05unLhbktc8l1en2ArYy0IAy8llhTsOxZa00+bE2DI8gV2BhuFVAwb7cv46VkiO2d/MnRtxys8F2A2sCsgC4yWCeW5Nj0BZVPKBvsclh4M7utPpFgTan3SPVZVMbZj9muwEng3kbW3JIO4k8pXG1mLFkFZDICyRtPUMc12RDuCfkQvphe03uTkdoXK6e4HDU09TPbUnjpZrSC9ntbE8hsjdh6DtHoU116JZZLvweiUUXRfSPlUoiICIiAiIqBQIiCbqVSiCoIoCXQTdCoQlBKXUIglFCIJVJREBEUIBKgoqSgFW3FVOKtuKClxVl5VxxViRyC1KVhTvV+d611RKoyi3NIsGV91VNIsZzljWUUvK5jSypc1wazwnABudgCb5k7mjMnqXSuWk0nw8zRuewXkbG9oAGbgc8vhbf4iuXtPHc8NR3di5Zx8ktcccXLDqwkhwvefLhXcuo79Uz4I8t1akk1+NI9z3fCcSfStW9ro9e4sRuIseogrGMzjsy6F52PE93PlnjPlsZ4mkHIEG/JcfYtNK4MyIsLkAnot59qu8K8utt6eRYGM6xeBfYM+s/XYBZ82MxxlY8PLbnoLw7NW5DY5BRRRE9azTRkeFkOqx8xXK7LnNsaCvkjII3cp2LuNHNPHkCOpBcNgkvd467+EPSuJqaawyWrluw7T8yzw5csL0a+Tgw5pqx9upqlr3uewhzX2cCNmyx+YLLvdcT7F1TwtNITtZMWnfkWMIXbBqZZd67cf7PudPRUCLX5N/zeVWwRtN9pt5d995VwEfjaseo68uRYsozKepNw29923Ythwlxy9JXPU8h1hu6Vt+EFhbasa2zxX2DVN9vKruzMBWoHfZ+OVZZZcdCTI5MdsaQX+ta6si5FsywgZWWNNF19e9bJk57hpqGuDdu3d9ijXLrndu5cleqKckm+zzdKxZHFpsNg9PUplWzCKgDfN2wbB9avRtO63o2LBdI6+V896zqO5tf7LrXcnR3bpkRjPyKzpEy9FV/JKn/AEXrMY3kVnSCld3FWHWv+SVGX/ResMcurHOdH2RF4U78XSbmOBdlxD7wTvxdJuY4F2XEPvBfVafGvdiBeE+/F0m5jgXZcQ+8E78XSbmOBdlxD7wTQ92ovCXfi6TcxwLsuIfeCnvxdJuY4F2XEPvBND3Yi8J9+LpNzHAuzYh94J342k/McC7LiH3gmh7sReE+/G0n5jgPZcQ+8E78bSfmOBdlxD7wV0PdiLwn342k/McC7LiH3gnfjaT8xwLsuIfeCaHuxF4T78bSfmOBdlxD7wTvxtJ+Y4D2XEPvBND3Yi8J9+NpNzHAey4h94J342k/McC7LiH3gmh7sReE+/G0n5jgXZcQ+8E78bSfmOBdlxD7wTQ92IvCffjaT8xwLsuIfeCd+NpPzHAuy4h94Joe7FC8Kd+NpPzHAey4h94J34uk3McC7LiH3gmh7rULwp34uk3McC7LiH3gnfiaTcxwLs2IfeCaNvdRVJXhbvxNJuY4F2XEPX078PSbmOBdmxD19NG3uVxVDivDZ9uDpNzHAuzYh6+oPtv9JeY4H2av9fTQ9wPKx5XLxKfbeaSn9hwPs1f6+rbvbb6SH9iwTs1f6+po29nVT1rKh68fSe2u0id+xYL5Keu9eVh3tpNID+x4P2et9dU0sr148q24ryIfbQY/zPB+z1vrqpPtncf5pg/Z631xTu1l3o9dFWyvJHfN49zTCP5Fb64oPtmse5phH8is9cU7lXvx6mxnCYaoWkFnWs2RttYefJw6Cvn+NaPSU7s+MwmzZGjik+KRtY/oNwdxK+Md8zj3NMI/kVnrity+2Txx7S11Hg7muFi009YQRyEGsWGXDvq38fae708n1aaRsTduYGzeT08gWjbK6R2w3LrZ3zOd18frPZdxGVxJpqBtzezY6mw6BrVJPnVqD2WMRZsgot+ZjqN+39f0BcHN2Tlzvk9Hg7dw4Trvf0egaCnyyBy8Hdc7xc7+pZzqMOHGJA5bjZylef2ezZioAAp8OyFh7jUXA5Ae6MlP/wCbsV2dzYda9w3gqmwPac/Ktc7By+3yzv4lxet+H23EQ0DijijK+4+da1sAlBIGQ3nl5Okr41V+zDikvhQ0Q+LHPbl2GdVD2Y8TDQ0U9AABbKKo35k3NRmbrH9w5d+Xy2Y/inDJ5/D0f7F41Y6pgFjwrHX62W+pdmZCOTzkryTgns54tSa/B02HO17E68NSdl9mrUjlWwf7YrGz+y4X5IKv1tZTsPLry+WvL8S4bd9fh6pjlJN8upRM4W6d3J/uvKzfbE40BbuXCv5FXf6WoPth8a2dy4V/Iq/W1Z2Hl9vlP4hw+/w9QgHWv5StpS5kHcvJY9sRjfNcL/kVfravw+2SxxuykwnqMFZ62pew8vt8sv4lxe/w9hUwHRblWU7IdBXjrvnMe5ng/Z6311Vt9tDj4/Y8HNtgNPW2+mrD+H83t8sv4nw+/wAPYDYiNo/HKsedoBt5V5LPtptINvceDdnrfXVZk9s7jzjc0eD9nrfXFl+4cs9Plj/EeG+O/h6srIj5LLCmiyy615ed7ZvHiLGkwjs9Z64rJ9snjh/ZMJ/kVnrit7Dy+3yxx/EeGevw9OtgubnzfUs1kfIvKY9sfjnNMJ/kVnraqb7ZLHB+yYT/ACKz1xYX8P5fb5bp+K8Pv8PWrGi2e5MddegrCMvyWpvy24CTzBeTR7ZfHeaYR/IrPXFFZ7ZXHZYpYXUmEBssb4nEQVgcGyMLCW3rLXs47QUx/DuWXy+TP8U4bPP4fFERF7z5oREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREH//2Q==\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"400\"\n",
       "            height=\"300\"\n",
       "            src=\"https://www.youtube.com/embed/UJwK6jAStmg\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x19d77d615c0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo('UJwK6jAStmg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 align = 'center'> Variables </h3>\n",
    "\n",
    "|Code Symbol | Math Symbol | Definition | Dimensions\n",
    "| :-: | :-: | :-: | :-: |\n",
    "|X|$$X$$|Input Data, each row in an example| (numExamples, inputLayerSize)|\n",
    "|y |$$y$$|target data|(numExamples, outputLayerSize)|\n",
    "|W1 | $$W^{(1)}$$ | Layer 1 weights | (inputLayerSize, hiddenLayerSize) |\n",
    "|W2 | $$W^{(2)}$$ | Layer 2 weights | (hiddenLayerSize, outputLayerSize) |\n",
    "|z2 | $$z^{(2)}$$ | Layer 2 activation | (numExamples, hiddenLayerSize) |\n",
    "|a2 | $$a^{(2)}$$ | Layer 2 activity | (numExamples, hiddenLayerSize) |\n",
    "|z3 | $$z^{(3)}$$ | Layer 3 activation | (numExamples, outputLayerSize) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last time, we setup our neural network on paper. This time, we’ll implement it in the programming language python. We’ll build our network as a python class and our init method will take care of instantiating important constants and variables. We’ll make these values accessible to the whole class by placing a self dot in front of each variable name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our network has 2 inputs, 3 hidden units, and 1 output. These are examples of hyperparameters. Hyperparameters are constants that establish the structure and behavior of a neural network, but are not updated as we train the network. Our learning algorithm is not capable of, for example, deciding that it needs another hidden unit, this is something that WE must decide on before training. What a neural network does learn are parameters, specifically the weights on the synapses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll take care of moving data through our network in a method called forward. Rather than pass inputs through the network one at a time, we’re going to use matrices to pass through multiple inputs at once. Doing this allows for big computational speedups, especially when using tools like MATLAB or Numpy. Our input data matrix, X, is of dimension 3 by 2, because we have 3, 2-dimensional examples. Our corresponding output data, y, is of dimension 3 by 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "#Import code from last time\n",
    "%pylab inline\n",
    "from partOne import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, 2) (3, 1)\n"
     ]
    }
   ],
   "source": [
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Neural_Network(object):\n",
    "    def __init__(self):        \n",
    "        #Define Hyperparameters\n",
    "        self.inputLayerSize = 2\n",
    "        self.outputLayerSize = 1\n",
    "        self.hiddenLayerSize = 3\n",
    "        \n",
    "    def forward(self, X):\n",
    "        #Propagate inputs though network\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each input value, or element in matrix X, needs to be multiplied by a corresponding weight and then added together with all the other results for each neuron. This is a complex operation, but if we take the three outputs we're looking for as a single row of a matrix, and place all our individual weights into a matrix of weights, we can create the exact behavior we need by multiplying our input data matrix by our weight matrix. Using matrix multiplication allows us to pass multiple inputs through at once by simply adding rows to the matrix X. From here on out, we'll refer to these matrics as X, W one, and z two, where z two the activity of our second layer. Notice that each entry in z is a sum of weighted inputs to each hidden neuron. Z is of size 3 by 3, one row for each example, and one column for each hidden unit. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our first official formula, $z^{(2)} = XW^{(1)}$. Matrix notation is really nice here, because it allows us to express the complex underlying process in a single line!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "z^{(2)} = XW^{(1)} \\tag{1}\\\\\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the activities for our second layer, z two, we need to apply the activation function. We'll independently apply the function to each entry in matrix z using a python method for this called sigmoid, because we’re using a sigmoid as our activation function. Using numpy is really nice here, because we can pass in a scalar, vector, or matrix, Numpy will apply the activation function element-wise, and return a result of the same dimension as it was given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    #Apply sigmoid activation function to scalar, vector, or matrix\n",
    "    return 1/(1+np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXxU9b3/8dcnk42QhLAGZN8REFQWW60VXHGptta2elt7a/Xnj161q7X2Wn+17e2ubb1eb71e9Wo3qbe1ipZKrYp1Q0EFZRMiIASQsIbsk5n5/P6YkQYMZEgmnFnez8djHpkz58zk/c3yzsmZs5i7IyIimS8v6AAiIpIaKnQRkSyhQhcRyRIqdBGRLKFCFxHJEvlBfeJ+/fr5iBEjOvXchoYGevbsmdpAAdFY0lO2jCVbxgEay3teffXVne7ev715gRX6iBEjWLp0aaeeu2jRImbNmpXaQAHRWNJTtowlW8YBGst7zOydQ83TJhcRkSyhQhcRyRIqdBGRLKFCFxHJEip0EZEs0WGhm9l9ZlZjZisOMd/M7N/NrMrM3jCzE1MfU0REOpLMGvr9wJzDzD8XGJu4XQ38suuxRETkSHW4H7q7/93MRhxmkYuAX3n8PLyLzazCzAa5+7YUZRQROaqiMachHKGhJUJza4ymcJTmSJTm1igtrTGaWuP3m1tjNLdGaWqN0tIapSUSIxyN0RqNEY7EaI16fDoSf+y96ZJIC92xS30qDiwaDGxuM12deOx9hW5mVxNfi6eyspJFixZ16hPW19d3+rnpRmNJT9kylmwZBxz5WCIxpz7s1LWS+OjUheOP1bc6zRFoinjixv6PzRGnOdp94wAYXurd8n1JRaFbO4+1e9UMd78buBtg+vTp3tkjpXTEWHrSWNJPtowD/jGWWMzZXtfMttpmttc28+6++G17beKxfc3sqg9T1xLp9Ocyg9LCfEqKQvQoCFFcEKKoIERxfh7FBe89Fr8fn5dHj4IQRfkhCvPzKAwZBaG8+O2g6cL8PNauWNYt35dUFHo1MLTN9BBgawpeV0RyVCzmbNnbRNWOejbtamTjrgZeW9vMv732LJt2NxKOxDp8jVCe0bukgN4lhfTpGb/17llIn5JCKkoKKCvOp6y4gNKifEqL8ylLfCwtyqdnYT55ee2tq6ZG86ZQt7xuKgp9PnCtmc0DTgJqtf1cRJJV29TKyi21rHm3jrXb61jzbh3rttfREG5vu0c9AP1KCxnUqweV5cUM7FX0j/uJ6X6lRZQXF3RrKaejDgvdzB4EZgH9zKwa+DZQAODudwELgPOAKqARuKK7wopIZmuNxli9bR/LNu9l2aa9LKvey/odDe0u26+0iHGVpQzv25PhfUuo37aB8z48k2F9SygtCuy8gmktmb1cLutgvgPXpCyRiGSNaMxZubWWF9/exYtv72LJht00tR645l2Yn8fEQeUcO6ic8ZWljB9YzrjKUvqWFh2w3KJFm5l4TPnRjJ9x9GdORFJqb2OYZ96q4W+ranhu3Q72NR/45uTIfj05YWgFxw+r4PihFUwYWE5hvg5aTwUVuoh02fZ9zTz+xjaeXPUuSzbuIRr7x45uQ/v04ORR/Th5TF8+OKovA8qLA0ya3VToItIpdc2tLFy5nUde38ILb+/EEx2en2ecMqYvZx5byRkTKhnWtyTYoDlEhS4iSXN3llfX8quXNrLgzW00t8Z3HywM5TF7Qn/On3IMp43rT68eBcEGzVEqdBHpUHNrlPnLt/Lrl97hzS21+x+fObIPHzthMOdNHkSvEpV40FToInJIdc2t/GbxJu59fj0768MAVJQU8MnpQ/n0ScMY3jc7LtqcLVToIvI+exvD3Pv8Bu5/cSN1ib1UJh1TzhWnjOSCKYMoLuieIx2la1ToIrJfc2uU+1/cyH8+U7V/d8OTRvbhmtljOHVsP8xy68jLTKNCFxFiMeePr1XzsyfXsq22GYBTxvTlK2eOY/qIPgGnk2Sp0EVy3IottXzrkRUs27wXgGMHlfPNcydojTwDqdBFclRdcys/e3ItD7y4kZjDgLIibjx3Ah89fnDOndQqW6jQRXLQ8+t28vU/LGdbbTN5Bp8/ZSRfOWssZcXa9TCTqdBFckhTOMqP/rKaB156B4CpQyv4wccmM+mYXgEnk1RQoYvkiBVbavnig6+zfmcD+XnGl88cy9zTRpMf0omxsoUKXSTLuTsPvrKZWx5bSTgSY3xlGbd9ciqTB2utPNuo0EWyWEvE+dpDy3n49S0AfPqkYdx8wUQdGJSlVOgiWap6TyPfW9xEdX0jPQpCfP9jk7n4xCFBx5JupEIXyULLNu/lqgeWsrPeGdWvJ7/8zDTGDywLOpZ0MxW6SJZZ8OY2vvL7ZbREYhzbJ495/3KKzoSYI1ToIlnk/hc2cMtjqwC4dMZQzuy9S2WeQ7S/kkgWcHf+/al1+8v8xnMn8MOLjyNfR3zmFK2hi2Q4d+ff/ryae5/fQJ7BDy8+jk/NGBZ0LAmACl0kg7k733pkBb99eRMFIeP2S0/gvOMGBR1LAqJCF8lQ7s53HlvFb1/eRFF+Hv91+TRmjR8QdCwJkLahi2Qgd+cHC1Zz/4sbKQypzCVOhS6SgW7761r++7kNFISMX37mRJW5ACp0kYxz/wsb+I9nqgjlGXdcdiJnHFsZdCRJEyp0kQzylze38Z3H47sm/vjjU5gzeWDAiSSdqNBFMsSSjbv50u+X4Q5fP2c8l0zTeVnkQCp0kQxQVVPPVQ8sJRyJ8ZkPDONfZo0OOpKkoaQK3czmmNlbZlZlZje2M7+XmT1mZsvNbKWZXZH6qCK5qbaxlaseWEJtUytnTazkOxdO1sWbpV0dFrqZhYA7gXOBicBlZjbxoMWuAVa5+1RgFnCbmRWmOKtIzolEY1z74Gts3NXIxEHl3H7p8YR0OL8cQjJr6DOBKndf7+5hYB5w0UHLOFBm8dWGUmA3EElpUpEc9OMn1vDcup307VnI3Z+dRkmhjgWUQzN3P/wCZpcAc9z9qsT05cBJ7n5tm2XKgPnABKAM+JS7/7md17oauBqgsrJy2rx58zoVur6+ntLS0k49N91oLOkpHcbywpZW/vvNMCGDG2YUM77PkV9lKB3GkSoaS9zs2bNfdffp7c5098PegE8A97SZvhy446BlLgF+DhgwBtgAlB/udadNm+ad9cwzz3T6uelGY0lPQY9l5ZZaH3vTAh/+jcf91y9t7PTrBD2OVNJY4oClfoheTWaTSzUwtM30EGDrQctcATyc+HxViUKfkNSfGxE5QH1LhGt+9xrhSIxPTR/KZz4wPOhIkiGSKfQlwFgzG5l4o/NS4ptX2toEnAFgZpXAeGB9KoOK5AJ3518ffpMNOxuYMLCM71w0KehIkkE6fIfF3SNmdi2wEAgB97n7SjObm5h/F/A94H4ze5P4ZpdvuPvObswtkpUefGUz85dvpaQwxJ2fPpHigiPfbi65K6m3zN19AbDgoMfuanN/K3B2aqOJ5JZVW/dxy2MrAfjBx45jdP/seANQjh4dKSqSBppbo3xp3uuEIzEunTGUj54wOOhIkoFU6CJp4KcL32JdTT2j+vfk2x/RdnPpHBW6SMBerNrJvc9vIJRn/PyTx9OjUNvNpXNU6CIBqm1q5fr/XQ7AF08fy9ShFQEnkkymQhcJ0C3zV7K1tpmpQyu4ZrbOoChdo0IXCcgTK7bxp9e30KMgxM8/OZX8kH4dpWv0EyQSgNrGVm5+NL6L4jfPm8Ao7aIoKaBCFwnA9xesYkddCzNG9OYzJ+nQfkkNFbrIUfZC1U4eWlpNYSiPH148hTyd31xSRIUuchQ1hiN88+E3AfjSmWMZM0CbWiR1VOgiR9HP/rqWTbsbOXZQOVd/eFTQcSTLqNBFjpIVW2q574UN5Bn8+OPHUaC9WiTF9BMlchTEYs63HllBzOGKU0YyZYgOIJLUU6GLHAUPLd3Mss17qSwv4itnjQs6jmQpFbpIN9vTEObHT6wB4KbzJ1JapAs9S/dQoYt0s58sfIs9ja2cPLovH5kyKOg4ksVU6CLdaNnmvcxbson8POO7F03CTPucS/dRoYt0k2jMufmRFbjDlaeOZMyAsqAjSZZToYt0k3lLNvHmlloG9Srmi6ePDTqO5AAVukg32Nfcys/+uhaAm84/lp56I1SOAhW6SDe48+kqdjWEmTGiN+cfpzdC5ehQoYuk2Du7GrjvhQ0A3HzBRL0RKkeNCl0kxX64YA2tUefiEwfriFA5qlToIim0eP0unlj5Lj0KQtxwzoSg40iOUaGLpEg05nzv8VUAzD1tNAN7FQecSHKNCl0kRf74WjUrt+5jUK9inRpXAqFCF0mBxnCEWxe+BcANc8bTozAUcCLJRSp0kRR44MV3qKlrYcqQXlw0dXDQcSRHqdBFuqi2qZW7nn0bgOvPHq9rhEpgVOgiXXTvc+upbWrlpJF9OHVsv6DjSA5ToYt0wa76Fu59Pn4Q0dfPGa+DiCRQSRW6mc0xs7fMrMrMbjzEMrPMbJmZrTSzZ1MbUyQ9/XLR2zSEo8we35/pI/oEHUdyXIdnDDKzEHAncBZQDSwxs/nuvqrNMhXAfwJz3H2TmQ3orsAi6WJbbRO/WvwOAF87e3zAaUSSW0OfCVS5+3p3DwPzgIsOWuafgIfdfROAu9ekNqZI+rnj6SrCkRjnHTeQyYN7BR1HBHP3wy9gdgnxNe+rEtOXAye5+7VtlvkFUABMAsqA2939V+281tXA1QCVlZXT5s2b16nQ9fX1lJaWduq56UZjSU8djaWmMcY3n2si5vD9D/XgmNL0fDsql74nmaQrY5k9e/ar7j69vXnJnKS5vXd5Dv4rkA9MA84AegAvmdlid197wJPc7wbuBpg+fbrPmjUriU//fosWLaKzz003Gkt66mgsX/39MqK+hY+fOIR/umDq0Qt2hHLpe5JJumssyRR6NTC0zfQQYGs7y+x09wagwcz+DkwF1iKSZdZur+NPy7ZQEDK+fKauRCTpI5n/E5cAY81spJkVApcC8w9a5lHgVDPLN7MS4CRgdWqjiqSHn/11Le5w6YxhDO1TEnQckf06XEN394iZXQssBELAfe6+0szmJubf5e6rzewJ4A0gBtzj7iu6M7hIEN6o3ssTK9+lKD+Pa08fE3QckQMkdaFDd18ALDjosbsOmv4p8NPURRNJP7cmrhP6uZNHUFmu0+NKeknPt+ZF0tDL63fx97U7KC3KZ+5po4OOI/I+KnSRJLg7t/41fnrcq04dSe+ehQEnEnk/FbpIEp5du4MlG/fQu6SAKz80Mug4Iu1SoYt0oO3a+dzTRlNWXBBwIpH2qdBFOvDEindZsWUfA8qK+OwHRwQdR+SQVOgihxGNObc9Gd+z5brTx+jScpLWVOgih/HI61uoqqlnSO8efGrGsKDjiByWCl3kEMKRGL94Kr52/uUzx1GYr18XSW/6CRU5hIeWbmbz7iZG9+/Jx07QhZ8l/anQRdoRjjp3PL0OiF+8IqQLP0sGSOrQf5Fc89SmCNv3hZl0TDlzJg0MOo5IUrSGLnKQuuZW/rw+DMD154wnT2vnkiFU6CIHue/5jdS3wvThvZk1rn/QcUSSpkIXaWNPQ5h7nlsPxNfOzbR2LplDhS7Sxl1/f5u6lgiT+4b4wKi+QccROSIqdJGEmn3NPPDiRgAuHqfztUjmUaGLJNz5TBXNrTHOmVTJqF46xF8yjwpdBNi8u5HfvbIJs/h+5yKZSIUuAvz7U+tojToXTT2GcZVlQccR6RQVuuS8qpp6/vhaNfl5xpfPHBd0HJFOU6FLzvv539YSc/jE9KGM6Ncz6DginaZCl5y2cmstf35jG4X5eXzxjDFBxxHpEhW65LTb/ho/Pe7lHxjOoF49Ak4j0jUqdMlZr76zm6fX1FBSGOILs0YHHUeky1TokpPcnZ8ujF/4+coPjaRfaVHAiUS6ToUuOemFql0sXr+b8uJ8rjp1VNBxRFJChS45J752vgaAubNG06uHDvOX7KBCl5yzcOW7LK+upX9ZEVecPDLoOCIpo0KXnBKJxvZvO//i6WPoUahztkj2UKFLTnn49S28vaOBoX168KkZw4KOI5JSKnTJGS2RKLf/LX7h56+eNY7CfP34S3ZJ6ifazOaY2VtmVmVmNx5muRlmFjWzS1IXUSQ1frt4E1v2NjG+sowLpw4OOo5IynVY6GYWAu4EzgUmApeZ2cRDLPdjYGGqQ4p0VX1LhDufqQLg6+eMJ6QLP0sWSmYNfSZQ5e7r3T0MzAMuame564A/AjUpzCeSEvc9v4FdDWFOHFbBGccOCDqOSLcwdz/8AvHNJ3Pc/arE9OXASe5+bZtlBgO/A04H7gUed/c/tPNaVwNXA1RWVk6bN29ep0LX19dTWlraqeemG42l+9WFnRv+3khTBG6cWcyEPh3v2ZKuYzlS2TIO0FjeM3v27FfdfXp78/KTeH57/5se/FfgF8A33D16uKuku/vdwN0A06dP91mzZiXx6d9v0aJFdPa56UZj6X7f//MqmiIb+PC4/sy9eGZSz0nXsRypbBkHaCzJSKbQq4GhbaaHAFsPWmY6MC9R5v2A88ws4u6PpCSlSCdtq23igZfeAeCGc3RpOcluyRT6EmCsmY0EtgCXAv/UdgF333+4nZndT3yTi8pcAnfrwrWEIzHOnzKIyYN7BR1HpFt1WOjuHjGza4nvvRIC7nP3lWY2NzH/rm7OKNIpK7fW8vDr1RSEjG+cMyHoOCLdLpk1dNx9AbDgoMfaLXJ3/1zXY4l0jbvzgwWrcYfPfnAEw/qWBB1JpNvpUDnJSovW7uCFql2UF+dz3em6tJzkBhW6ZJ1INMYPF6wG4LrTx1JRUhhwIpGjQ4UuWecPr1azdns9Q3r34LMnDw86jshRo0KXrNLQEuG2J+MXfr5hzgSK8nV6XMkdKnTJKv/93Hp21LUwdWgFH5kyKOg4IkeVCl2yxrbaJv7r2fUA3HTesRzuqGWRbKRCl6zxo7+soak1ypxJA5k5sk/QcUSOOhW6ZIUlG3fz6LKtFOXncdP5xwYdRyQQKnTJeNGY8+1HVwIw97TRDO2jg4gkN6nQJePNW7KJVdv2MbiiB3NPGx10HJHAqNAlo+1tDHPrwrcAuOn8Y+lRqN0UJXep0CWj/ezJtexpbOWDo/py7uSBQccRCZQKXTLWqq37+M3idwjlGd++cKJ2U5Scp0KXjBSLOTc98iYxh8s/MJwJA8uDjiQSOBW6ZKTfvbKJ1zftZUBZEV89e1zQcUTSggpdMk5NXTM/fmINALdcOIny4oKAE4mkBxW6ZJzvPb6auuYIs8f31xuhIm2o0CWjPLt2B48t30pxQR7fvWiy3ggVaUOFLhmjKRzl5kdWAPDlM8fpiFCRg6jQJWP8ZOEaNu1uZMLAMq780Mig44ikHRW6ZIRXNuzm/hc3EsozfnrJVApC+tEVOZh+KyTtNYWj3PCH5bjDF04bzXFDegUdSSQtqdAl7f1k4Ro27mpkfGUZ150xJug4ImlLhS5pre2mlls/MVXXCBU5DBW6pK36lghfT2xq+ZdZ2tQi0hEVuqStW+av5J1d8b1arjt9bNBxRNKeCl3S0vzlW/nDq9UU5edxx2UnUJivH1WRjui3RNLO5t2N3PSnNwG4+YKJjK0sCziRSGZQoUtaiURjfOX3y6hrjnDWxEo+fdKwoCOJZAwVuqSV259ax9J39lBZXsSPPz5F52oROQIqdEkbT63ezh1PV5Fn8PNPHk+fnoVBRxLJKEkVupnNMbO3zKzKzG5sZ/6nzeyNxO1FM5ua+qiSzTbtauQrv18GwNfOHs/JY/oFnEgk83RY6GYWAu4EzgUmApeZ2cSDFtsAnObuU4DvAXenOqhkr+bWKHN/8yr7miOceWwlXzhtdNCRRDJSMmvoM4Eqd1/v7mFgHnBR2wXc/UV335OYXAwMSW1MyVbuzrceWcGqbfsY3reE2z45lbw8bTcX6Qxz98MvYHYJMMfdr0pMXw6c5O7XHmL564EJ7y1/0LyrgasBKisrp82bN69Toevr6yktLe3Uc9NNro9l4cZWHlwTpjAPbv5gD4aWpcfbOtnyfcmWcYDG8p7Zs2e/6u7T253p7oe9AZ8A7mkzfTlwxyGWnQ2sBvp29LrTpk3zznrmmWc6/dx0k8tjeXLluz7ixsd9+Dce9/nLtnRPqE7Klu9LtozDXWN5D7DUD9Gr+Un8QagGhraZHgJsPXghM5sC3AOc6+67kv1rI7lp1dZ9fHHe67jDV88ax0emHhN0JJGMl8z/t0uAsWY20swKgUuB+W0XMLNhwMPA5e6+NvUxJZvU7GvmqgeW0BiOctHxx3Dd6TolrkgqdLiG7u4RM7sWWAiEgPvcfaWZzU3Mvwv4f0Bf4D8TB4JE/FDbeCSn1Ta18tn7XmFrbTMnDKvQwUMiKZTMJhfcfQGw4KDH7mpz/yrgfW+CirTV3Brl/zywlDXv1jGqf0/u+ex0igt0fnORVEmPXQok60WiMa793Wu8snE3A8uL+fWVJ9G3tCjoWCJZRYUu3S4ac67/3+X8bXUNFSUF/PrKmQyu6BF0LJGso0KXbhWNOV97aBmPLNtKz8IQ931uhk6HK9JNktqGLtIZ0Zjz1YeW8WiizO///ExOHNY76FgiWUuFLt2iNRrjaw8tZ/7yeJk/8PmZTB/RJ+hYIllNhS4p1xiO8IXfvMaza3eozEWOIhW6pNSehjBX3L+EZZv30qdnIfdfMYMpQyqCjiWSE1TokjI1jTEuuetF3t7RwOCKHvz6ypmM6p8dJ1MSyQQqdEmJl97exXdfaqK+FcZXlvGrK2dSWV4cdCyRnKJCly77zeJ3uGX+SiIxmD2+P7dfdgLlxQVBxxLJOSp06bSmcJTvPLaSeUs2AzBnRAF3/vMMQrpAhUggVOjSKeu213HN715j7fZ6CvPz+MHHjqNfXZXKXCRAOlJUjoi789DSzVz4Hy+wdns9o/r35NFrTuGSabrqoEjQtIYuSdu+r5mb/vQmf1tdA8DFJwzmex+dTM8i/RiJpAP9JkqH3J2HX9vCdx5byb7mCGXF+Xz7I5P4+ImDdS5zkTSiQpfDqqqp5zuPreS5dTuB+F4sP7j4OAb10tkSRdKNCl3aVd8S4Y6n1nHv8xuIxJzy4nxuvmAil0wborVykTSlQpcDhCMxfr90M3c8tY6auhbM4LKZQ7n+7PG6IIVImlOhCxA/1e2jy7bw87+tZfPuJgCOH1rBdy+apHOxiGQIFXqOa4lEefT1rdz93HqqauoBGN2/J9efPZ45kwdq84pIBlGh56japlZ+9/Im/ueFDdTUtQAwuKIHXz5zLB87YTD5IR2iIJJpVOg5xN1ZXl3Lgy9vYv7yrTS1RgGYMLCMqz88io9MPYYCFblIxlKh54Cd9S38+Y1tzFuymdXb9u1//OTRffm/p43mw2P7adOKSBZQoWepvY1hnljxLo+/sY0X395JzOOP9+lZyCXThvCpGUMZrXOVi2QVFXqWcHfe3lHPM2t28PSaGpZs3E0k0eIFIWP22P589ITBnD2pkqL8UMBpRaQ7qNAzWE1dM0s27GHx+l0sWluzf3dDgDyDU8f24yNTjuGcSQPpVaLzk4tkOxV6hojGnA0763mjupYlG3fz8vrdrN/ZcMAyfXoWctq4/syeMIAPj+1HRUlhQGlFJAgq9DTUEomyYWcDq7ft443qWlZsqWXV1n00hKMHLFdSGGLa8N7MGNGHD43tx9QhFTofuUgOU6EHJBZzdtS3sG5PlB1LN1O1o563a+qpqqln0+7G/W9itjWoVzGTB/di+vDezBzZh8mDe2k3QxHZT4XeDWIxZ09jmB31Leyoi9+21TZTvaeJ6j2NbNnTRPXeJsKRWPwJL79xwPPzDEb268nYAaUcN7gXk4f04rjBveinc6mIyGEkVehmNge4HQgB97j7jw6ab4n55wGNwOfc/bUUZw1EazRGbVPrAbd9Ta3sbTzwsd0N4f3lvbO+Zf8eJofTt2ch5fkRJo8YyJj+pYwZEL+N6FeiPVFE5Ih1WOhmFgLuBM4CqoElZjbf3Ve1WexcYGzidhLwy8THlGsMR6htcbYk1nDDkRit0Rgtbe6HIzHCB39M3G+NxGhqjdIYjtIYjtAQjtIUjtLQEqGpNfExHN3/eDga61TOipIC+pcW0b8sfhtQVsTQPiUM6d2DIb1LGFzRg55F+SxatIhZs05I8VdJRHJRMmvoM4Eqd18PYGbzgIuAtoV+EfArd3dgsZlVmNkgd9+W6sDfe3w1D77SCM88neqXblcozygvzqdXj4L4raQwcf8fj1X0KKSipIAB5cX0LyuiX2mh1rBF5KhLptAHA5vbTFfz/rXv9pYZDBxQ6GZ2NXA1QGVlJYsWLTrCuFC7I0xpgVMYyiM/j/jNjPw8KHhvOq/t9D/uh8woyIPCEBSFjKL8xMcQFCc+FuX/Y7owFH/ePw6LjyRubcSAhvht7w7YC6w7gvHU19d36uuQjjSW9JMt4wCNJRnJFHp7+8EdvIE4mWVw97uBuwGmT5/us2bNSuLTH2jWLBKbKY78uelIY0lP2TKWbBkHaCzJSGaft2pgaJvpIcDWTiwjIiLdKJlCXwKMNbORZlYIXArMP2iZ+cBnLe4DQG13bD8XEZFD63CTi7tHzOxaYCHx3Rbvc/eVZjY3Mf8uYAHxXRariO+2eEX3RRYRkfYktR+6uy8gXtptH7urzX0HrkltNBERORI6blxEJEuo0EVEsoQKXUQkS6jQRUSyhMXfzwzgE5vtAN7p5NP7ATtTGCdIGkt6ypaxZMs4QGN5z3B379/ejMAKvSvMbKm7Tw86RypoLOkpW8aSLeMAjSUZ2uQiIpIlVOgiIlkiUwv97qADpJDGkp6yZSzZMg7QWDqUkdvQRUTk/TJ1DV1ERA6iQhcRyRIZXehmdp2ZvWVmK83sJ0Hn6Sozu97M3Mz6BZ2ls8zsp2a2xszeMLM/mVlF0JmOhJnNSfxMVZnZjUHn6SwzG2pmz5jZ6sTvx5eCztRVZhYys9fN7PGgs3RF4hKdf0j8nqw2sw+m6rUzttDNbDbxa5lOcfdJwK0BR+oSMxtK/ELcm4LO0kVPApPdfTJvO+kAAAKiSURBVAqwFvhmwHmS1uaC6OcCE4HLzGxisKk6LQJ8zd2PBT4AXJPBY3nPl4DVQYdIgduBJ9x9AjCVFI4pYwsd+ALwI3dvAXD3moDzdNXPgRto59J9mcTd/+ru7114dTHxq1dliv0XRHf3MPDeBdEzjrtvc/fXEvfriJfG4GBTdZ6ZDQHOB+4JOktXmFk58GHgXgB3D7v73lS9fiYX+jjgVDN72cyeNbMZQQfqLDO7ENji7suDzpJinwf+EnSII3Coi51nNDMbAZwAvBxski75BfEVnljQQbpoFLAD+J/E5qN7zKxnql48qQtcBMXM/gYMbGfWTcSz9yb+7+QM4CEzG+Vpuh9mB2P5V+Dso5uo8w43Fnd/NLHMTcT/7f/t0czWRUld7DyTmFkp8Efgy+6+L+g8nWFmFwA17v6qmc0KOk8X5QMnAte5+8tmdjtwI3Bzql48bbn7mYeaZ2ZfAB5OFPgrZhYjfsKbHUcr35E41FjM7DhgJLDczCC+ieI1M5vp7u8exYhJO9z3BcDM/hm4ADgjXf/AHkJWXezczAqIl/lv3f3hoPN0wSnAhWZ2HlAMlJvZb9z9MwHn6oxqoNrd3/tv6Q/ECz0lMnmTyyPA6QBmNg4oJAPPxObub7r7AHcf4e4jiH/DT0zXMu+Imc0BvgFc6O6NQec5QslcED0jWHzt4F5gtbv/LOg8XeHu33T3IYnfj0uBpzO0zEn8Xm82s/GJh84AVqXq9dN6Db0D9wH3mdkKIAz8c4atDWar/wCKgCcT/3Esdve5wUZKzqEuiB5wrM46BbgceNPMliUe+9fE9YElWNcBv02sNKwHrkjVC+vQfxGRLJHJm1xERKQNFbqISJZQoYuIZAkVuohIllChi4hkCRW6iEiWUKGLiGSJ/w/NaA+kjCBEYAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "testInput = np.arange(-6,6,0.01)\n",
    "plot(testInput, sigmoid(testInput), linewidth= 2)\n",
    "grid(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7310585786300049"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.26894142, 0.5       , 0.73105858])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(np.array([-1,0,1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.40914521, 0.60648112, 0.57993188],\n",
       "       [0.41607172, 0.20028402, 0.59943281],\n",
       "       [0.34800346, 0.40257832, 0.68254148]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(np.random.randn(3,3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We now have our second formula for forward propagation, using f to denote our activation function, we can write that a two, our second layer activity, is equal to f of z two. a two will be a matrix of the same size as z two, 3 by 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "a^{(2)} = f(z^{(2)}) \\tag{2}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To finish forward propagation we need to propagate a two all the way to the output, yhat. We've already done the heavy lifting in the previous layer, so all we have to do now is multiply a two by our second layer weights W2 and apply one more activation funcion. W2 will be of size 3x1, one weight for each synapse. Multiplying a2, a 3 by 3, by W2, a 3 by 1 results in a 3 by 1 matrix z three, the activity or our third layer. z3 has three activity values, one for each example. Last but not least, we'll apply our activation function to z three yielding our official estimate of your test score, yHat.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "z^{(3)} = a^{(2)}W^{(2)} \\tag{3}\\\\\n",
    "$$\n",
    "$$\n",
    "\\hat{y} = f(z^{(3)}) \\tag{4}\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to implement our forward propagation formulas in python. First we'll initialize our weight matrices in our init method. For starting values, we'll use random numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll implement forward propagation in our forward method, using numpy's built in dot method for matrix multiplication and our own sigmoid method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Neural_Network(object):\n",
    "    def __init__(self):        \n",
    "        #Define Hyperparameters\n",
    "        self.inputLayerSize = 2\n",
    "        self.outputLayerSize = 1\n",
    "        self.hiddenLayerSize = 3\n",
    "        \n",
    "        #Weights (parameters)\n",
    "        self.W1 = np.random.randn(self.inputLayerSize, self.hiddenLayerSize)\n",
    "        self.W2 = np.random.randn(self.hiddenLayerSize, self.outputLayerSize)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        #Propagate inputs though network\n",
    "        self.z2 = np.dot(X, self.W1)\n",
    "        self.a2 = self.sigmoid(self.z2)\n",
    "        self.z3 = np.dot(self.a2, self.W2)\n",
    "        yHat = self.sigmoid(self.z3) \n",
    "        return yHat\n",
    "        \n",
    "    def sigmoid(self, z):\n",
    "        #Apply sigmoid activation function to scalar, vector, or matrix\n",
    "        return 1/(1+np.exp(-z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there you have it, a python class capable of estimating your test score given how many hours you sleep and how many hours you study. We can pass in our input data and get real outputs. Now, you may be noticing that our estimates are quite terrible. That's because we have not yet trained our network, that's what we'll work on next time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}