Off_policy_Monte_Carlo_ITA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Off-policy Monte Carlo\n",
        "\n",
        "Mario Fiorino\n",
        "\n",
        "L'apprendimento off-policy utilizza due policy:\n",
        "\n",
        "- una detta target policy , appresa durante il training, è progettata per avvicinarsi alla policy ottimale. Spesso indicata con $\\pi$\n",
        "\n",
        "- una behavior policy, o \"esplorativa\", che viene utilizzata per generare dati (o comportamenti). Spesso indicata con $b$\n",
        "\n",
        "\n",
        "\n",
        "Si ricorda che i metodi on-policy (visti nel notebook precedente) aggiornano la stessa politica ad ogni iterazione al fine di ottenere una politica che converge verso la politica ottimale. Tale l’approccio on-policy è di fatto un compromesso. Non apprende la politica ottimale, ma una politica quasi-ottimale (near-optimal), che cioè continua ad esplorare.\n",
        "\n",
        "Si ricorda inoltre che i metodi Montecarlo non richiedono la conoscenza della funzione di probabilità di transizione; cioè di un modello che mette in relazione gli stati con le azioni e ci permette di formulare previsioni.\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Prerequisiti per capire questo notebook\n",
        "\n",
        "Leggere almeno l'introduzione del notebook precedente: https://github.com/MarioFiorino/Tutorial-Reinforcement-Learning-ITA-Python/blob/main/Metodi_Monte_Carlo_RL_ITA.ipynb\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Ref\n",
        "\n",
        "Reinforcement Learning: An Introduction\n",
        "\n",
        "Richard S. Sutton and Andrew G. Barto, Second Edition - MIT Press, Cambridge, 2018.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "Come introduzione, può esser d'aiuto anche il video:\n",
        "\n",
        "https://www.youtube.com/watch?v=bpUszPiWM7o\n",
        "\n",
        "\n",
        "In questo video troverete anche una versione più intuitiva dell'algoritmo Off-Policy Monte Carlo Control (min 21.55), rispetto a quella introdotta in questo notebook."
      ],
      "metadata": {
        "id": "hlD1HByqlXdF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Premessa statistica: Incremental v.s non-incremental  \n",
        "\n",
        "E' possibile utilizzare entrambi i metodi: incrementale e non-incrementale,  per il ricavare la stima del'expected value $\\mathbb{E}[X]$ di una certa variabile random $X$.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Gli algoritmi che abbiamo visto nel notebook precedente, sono basati su update non incrementale. Ovvero, raccolgono prima tutti i campioni: $\\{x_{i} \\}^{n} _{i=1}$\n",
        "\n",
        "e poi  calcolano la media:\n",
        "\n",
        "$\\mathbb{E}[X] \\approx \\frac{1}{n} \\sum_{i=1}^{n} x_{i} $\n",
        "\n",
        "Questo metodo presenta lo svantaggio che, se il numero di campioni è elevato, l'attesa per il calcolo della media può essere lunga; e lo spazio di memoria necessario molto elevato.\n",
        "\n",
        "I metodi incrementali, invece, calcolano la media in modo computazionalmente più efficiente. Come? Sfruttando la formulazione:\n",
        "\n",
        "$w_{n}  \\;  \\doteq \\; \\frac{1}{n-1}  \\sum_{i=1}^{n-1} x_{i} $\n",
        "\n",
        "\n",
        "$w_{n+1}  \\; =  \\; w_{n} + \\frac{1}{n}   \\;   (x_{n} − w_{n})  \\; = \\; \\frac{1}{n}  \\sum_{i=1}^{n} x_{i} $\n",
        "\n",
        "\n",
        "\n",
        "Ovvero la media di un certo passo $n$ è ottenuta dalla media del passo precedente $n-1$ e aggiornata con le nuove osservazioni $x_{n}$.\n",
        "\n",
        "Per un esempio pratico , sia $w_{1} = a $ , un valore del tutto arbitrario.\n",
        "\n",
        "$w_{2}  =  \\; w_{1} + \\frac{1}{1}   \\;   ( x_{1} - w_{1} ) = x_{1} $\n",
        "\n",
        "$w_{3}  =  \\; w_{2} + \\frac{1}{2}   \\;   (x_{2} - w_{2}) = \\frac{1}{2}   \\;   (x_{1} + x_{2}) $\n",
        "\n",
        "$w_{4}  =  \\; w_{3} + \\frac{1}{3}   \\;   (x_{3} - w_{3}) = \\frac{1}{3}   \\;   (x_{1} + x_{2} + x_{3}) $\n",
        "\n",
        "e così via ....  $w_{n+1}  = \\frac{1}{n}  \\sum_{i=1}^{n} x_{i}$\n",
        "\n",
        "\n",
        "\n",
        "Per le manipolazioni simbolico-matematiche che dimostrano la validità della formula, vedi il testo di Sutton e Barto, Second Edition, pag PDF 53.\n",
        "\n",
        "\n",
        "Il vantaggio di questa equazione è che consente di calcolare la media immediatamente, ogni volta che si riceve un nuovo campione; in più questa implementazione richiede memoria solo per i dati $w_{n}$ e $n$.\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Qu9g_6CwL2JV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## MC prediction problem (policy evaluation)\n",
        "\n",
        "\n",
        "In tale fase,  rigurado la target policy $\\pi$ e la behavior policy $b$, si fa un assunzione  di \"coverage\", ovvero :\n",
        "\n",
        "$∀a  \\; \\pi(a|s) > 0 \\implies b(a|s) > 0 $\n",
        "\n",
        "cioè, in un certo stato $s$, per ogni azione $a$ che ha un probablità positva di essere estratta da $\\pi$ , deve esistere una possibilità (eventualemente anche minima) che tale azione $a$ venga estratta anche da $b$.\n",
        "Nella pratica si assume che $b$ sia una soft-policy.\n",
        "\n",
        "\n",
        "I metodi Monte-Carlo cercano di calcolare expected returns $V_{\\pi}$ della target policy; tutto ciò che abbiamo però sono i return $G_{t}$ (e quindi una approssimazione di $V_{b}$) della\n",
        "behavior policy.\n",
        "\n",
        "Problema :\n",
        "\n",
        "come ricavare $V_{\\pi}$ della target policy in base al campiomentno dei return $G_{t}$ della\n",
        "behavior policy ?\n",
        "\n",
        "Qui entra in gioco la tecnica **Importance sampling** :\n",
        "\n",
        "in termini statistici, è una tecnica Monte Carlo che consente di stimare le proprietà di una distribuzione di probabilità, anche se si dispone solo di campioni generati da una distribuzione diversa. In pratica, l'importance sampling consiste nel generare campioni dalla distribuzione diversa, e poi nel pesare i campioni in base alla probabilità che avrebbero avuto se fossero stati generati dalla distribuzione di interesse. Per maggiori info : https://en.wikipedia.org/wiki/Importance_sampling\n",
        "\n",
        "\n",
        "Si consideri la probalità del verificarsi una certa traiettoria, dato lo stato di partenza $s_{t}$, seguendo una certa  policy $\\pi$\n",
        "\n",
        "$\\prod_{k=t}^{T-1} \\pi(a_{k}|s_{k}) p (s_{k+1}|s_{k},a_{k} ) = \\pi(a_{t}|s_{t}) p (s_{t+1}|s_{t},a_{t}) \\;\\cdot \\; \\pi(a_{t+1}|s_{t+1}) p (s_{t+2}|s_{t+1},a_{t+1}) \\;\\cdot \\; ... \\;\\cdot \\;\n",
        " \\pi(a_{T-1}|s_{T-1}) p (s_{T}|s_{T-1},a_{T-1}) \\;   $\n",
        "\n",
        "Dove $p$ è la probabilita di transizione (transition probability function : la probabilità di passare allo stato $s_{k+1}$ se inizi dallo stato $s_{k}$ e intraprendi l'azione $a_{k}$ ); mentre $S_{T}$ è uno stato terminale.\n",
        "\n",
        "Il rapporto tra la probabilità del verificarsi di una certa traiettoria sotto una certa policy e la probabilità del verificarsi della stessa traiettoria sotto una pocily diversa, nel nostro caso le due policy sono target e behavior,  è detto, **importance sampling ratio**:\n",
        "\n",
        "$\\rho_{t:T-1} = \\prod_{k=t}^{T-1}  \\frac{ \\pi(a_{k}|s_{k})}{ b (a_{k}|s_{k})} $\n",
        "\n",
        "\n",
        "\n",
        "Vedremo come questo rapporto è fondamentale perché corregge il \"bias\" introdotto dal campionamento da una distribuzione diversa.\n",
        "\n",
        "Per il ragionamento matematico che ha portato a tale formulazione, per maggiori chiarimenti e integrazioni teoriche, anche per i passaggi successivi, vedere i video:\n",
        "\n",
        "https://www.youtube.com/watch?v=jbnoyCLY9cI&t=10s\n",
        "\n",
        "https://www.youtube.com/watch?v=xpevq9jBi7o&t=10s\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "L'importance sampling ratio ci consente di stimare expected value della target policy, disponendo del campionamento dei return $G_{t}$ della behavior policy, in simboli:\n",
        "\n",
        "$\\mathbb{E}[\\rho_{t:T-1} \\cdot G_{t} | S_{start}] =  V_{\\pi}(S_{start}) $\n",
        "\n",
        "La formula qui sopra, di natura del tutto generale, si può concretizzare nel seguente modo:\n",
        "\n",
        "Assumedo che stiamo usando un algoritmo del tipo MC first-visit.\n",
        "\n",
        "Assumiamo inoltre che utilizziamo la suddivisione degli episodi in base all'idea : \"It is convenient here to number time steps in a way that increases across episode boundaries.\" suggerita da Sutton e Barto, Second Edition, PDF 126. Tale idea comporta le seguenti rappresentazioni:\n",
        "\n",
        "$\\Im(s)$ indica l'insime dei time step in cui un certo stato $s$ è stato visitato per la prima volta all'interno dei loro rispettivi episodi. Esempio, un certo stato $s'$ , se è stato visitato al time step t = 3 nel primo episodio, poi la time step t = 20 nel secondo, poi nel terzo t=45, ed infine al t=1023 nel decino episodio. Allora, $\\Im(s') = \\{3,20,45,1023\\} $\n",
        "\n",
        "\n",
        "$T(t)$ indica il primo time-step terminale, incontrato nell'episodio con il time-spep $t$ . Esempio se l'episodio contente il time-step $t = 3$ termina al time-step $10$, allora $T(3) = 10$.\n",
        "\n",
        "$G_{t}$ indica il return ottenuto partendo dal time step $t$ fino a quello terminale $T(t)$.\n",
        "\n",
        "\n",
        "$\\{G_{t}\\}_{t \\in \\Im(s)}$ indica l'insime di tutti i return che partono dalla state $s$. Nel nostro esempio, considerando lo stato $s'$,allora abbiamo: $\\{G_{t}\\}_{t \\in \\Im(s')} = \\{G_{t=3}, G_{t=20}, G_{t=45}, G_{t=1023} \\}$\n",
        "\n",
        "\n",
        "\n",
        "$\\{\\rho_{t:T(t)-1} \\}_{t \\in \\Im(s)}$ denota l'insime dei relativi importance-sampling ratio della policy target e behavior. Ovvero, sempre considernado il nostro esempio, partendo dallo stato $s'$, entrambe le policy target e behavior producono una certa probabilità di estrarre una data traiettoria (uguale per entrambe, e che quindi si conclude nel rispettivo stato terminale $T(t)$). In simboli:  $\\{\\rho_{t:T(t)-1} \\}_{t \\in \\Im(s')} = { \\{ \\rho_{3:9}}, \\rho_{20:T(20)-1}, \\rho_{45:T(45)-1}, \\rho_{1023:T(1023)-1}  \\}$\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Infine , per stimare expected value della target policy, disponendo del campionamento dei return $G_{t}$ della behavior policy, usiamo la formula:\n",
        "\n",
        "$V_{\\pi}(s) = \\frac{\\sum_{t \\in \\Im(s)} \\rho_{t:T(t)-1} \\cdot G_{t} } {\\lvert \\Im(s) \\rvert} $\n",
        "\n",
        "Dove $\\lvert \\Im(s) \\rvert$, denota la cardinalità di un insieme, ovvero il numero di elementi contenuto nell'insieme; in pratica, il numero di volte che un certo stato $s$ è stato visitato la prima volta durante il training.\n",
        "\n",
        "Considerando il nostro esempio, per lo stato $s'$, la formula diventa:\n",
        "\n",
        "$V_{\\pi}(s') ≈ \\frac{ \\rho_{3:9} \\cdot G_{3} } {4}  + \\frac{ \\rho_{20:T(20)-1} \\cdot G_{20} } {4}  +  \\frac{ \\rho_{45:T(45)-1} \\cdot G_{45} } {4} + \\frac{ \\rho_{1023:T(1023)-1} \\cdot G_{1023} } {4}$\n",
        "\n",
        "\n",
        "Quando $V_{\\pi}(s)$ viene stimato in questo modo, diciamo che stiamo usando lo stimatore: **ordinary importance sampling**.\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Capire il tutto con un esempio pratico.\n",
        "\n",
        "\n",
        "La cosa si puo capire facilmente con un esempio molto semplice di un ambinete, con funzione di transizione di stato $p=1$, con soli due stati, o meglio celle contigue, una iniziale e finale: $s_{0}$ e $s_{T}$, ed due sole azioni $d$ e $alt$, ovvero \"vai a destra\" e \"resta fermo\". Reward = $+1$ ottenuto nella cella $s_{T}$, altrimenti $0$. Il fattore gamma = 1.\n",
        "\n",
        "La target policy ha la seguente distribuzione di probabilità:\n",
        "\n",
        "$\\pi(d |s_{0}) = 0.6$\n",
        "\n",
        "$\\pi(alt |s_{0}) = 0.4 $\n",
        "\n",
        "mentre la behavior:\n",
        "\n",
        "$b(d |s_{0}) = 0.2$\n",
        "\n",
        "$b(alt |s_{0}) = 0.8$\n",
        "\n",
        "\n",
        "Campioniamo 10 traiettorie dalla behavior:\n",
        "\n",
        "1. $s_{0},a_{d},s_{T}$\n",
        "\n",
        "2. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "3. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "4. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "5. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "6. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "7. $s_{0},a_{alt},a_{d},s_{T}$\n",
        "\n",
        "8. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "9. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "10. $s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{alt},s_{0},a_{d},s_{T}$\n",
        "\n",
        "Abbiamo 10 elementi $G_{t}$ con valore $1$\n",
        "\n",
        "Ricaviamo i rispettivi importance sampling ratio delle due distribuzioni, per ogni  traiettoria che parte $s_{0}$ , first-visit:\n",
        "\n",
        "1. $\\rho^{1} = \\frac{0.6}{0.2} = 3 $\n",
        "\n",
        "2. $\\rho^{2} = \\frac{0.4 * 0.4 * 0.6}{0.8 * 0.8 *0.2} = 0.75 $\n",
        "\n",
        "3. $\\rho^{3} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4* 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.0239$\n",
        "\n",
        "4. $\\rho^{4} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.187 $\n",
        "\n",
        "5. $\\rho^{5} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4* 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.0239$\n",
        "\n",
        "6. $\\rho^{6} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4* 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.0239$\n",
        "\n",
        "7. $\\rho^{7} = \\frac{0.4 * 0.6}{0.8 *0.2} = 1.5 $\n",
        "\n",
        "8. $\\rho^{8} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.187 $\n",
        "\n",
        "9. $\\rho^{9} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4 * 0.4* 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.0239$\n",
        "\n",
        "10. $\\rho^{10} = \\frac{0.4 * 0.4 * 0.4 * 0.4 * 0.6}{0.8 * 0.8 * 0.8 * 0.8 * 0.2} =  0.187 $\n",
        "\n",
        "\n",
        "Una stima dell'expected return della policy target:\n",
        "\n",
        "$ V_{\\pi}(s_{0}) \\approx  \\frac{3 + 0.75 + 0.0239 + 0.187 + 0.0239 + 0.0239 + 1.5 +  0.187  + 0.0239 + 0.187 }{ 10 } = 0.59 $\n",
        "\n",
        "Un valore molto vicino al valore vero $ V_{\\pi}(s_{0}) = 0.6$\n",
        "\n",
        "Se invece, avessimo avuto campionamento di traiettorie differenti,in $10$ traiettorie campionate da $b$ :\n",
        "\n",
        "-   $ V_{\\pi}(s_{0}) \\approx 0.0239 ⋅ 10 / 10 = 0.23 $.\n",
        "\n",
        "-   $ V_{\\pi}(s_{0}) \\approx (3⋅3) + 1.5 + (0.0239 ⋅ 6) / 10 = 1.07 $.\n",
        "\n",
        "\n",
        "L'ordinary importance sampling,  è uno stimatore \"unbiased\" ma ha una varianza \"unbounded\", ovvero non limitata: cioè puo essere molto alta, come si intuisce dal semplice esempio, legata alla varianza della distribuzione dei ratio (da 3 a 0.0239), che produce dei valori $ V_{\\pi}(s_{0})$ molto diversi.\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "\n",
        "Introduciamo uno stimatore alternativo **weighted importance sampling**:\n",
        "\n",
        "\n",
        "$V_{\\pi}(s) = \\frac{\\sum_{t \\in \\Im(s)} \\rho_{t:T(t)-1} \\cdot G_{t} } {\\sum_{t \\in \\Im(s)} \\rho_{t:T(t)-1}} $\n",
        "\n",
        "\n",
        "Nell'esempio di partenza avremo:\n",
        "\n",
        "$V_{\\pi}(s') ≈ \\frac{1} {\\rho_{3:9} +  \\rho_{20:T(20)-1} + \\rho_{45:T(45)-1} + \\rho_{1023:T(1023)-1} }  \\cdot ( \\; (\\rho_{3:9} \\cdot G_{3}) +  (\\rho_{20:T(20)-1} \\cdot G_{20})  +  (\\rho_{45:T(45)-1} \\cdot G_{45}) + (\\rho_{1023:T(1023)-1} \\cdot G_{1023})  \\;)$\n",
        "\n",
        "\n",
        "Considenrando adesso l'esempio pratico, e le $10$ traiettorie campionate da $b$ in precedenza, avremo:\n",
        "\n",
        "$ V_{\\pi}(s_{0}) \\approx \\frac{5.9}{5.9}  =  G_{t} = 1  $  che è un risultato che ha un sè un errore sistematico, ma è indipendente dalla varianza dei ratio.\n",
        "\n",
        "Se nello stesso ambinete avremo avuto due reward; $R(s_{T}) = +1$ e $R(s_{0}) = - 0.1$.\n",
        "\n",
        "\n",
        "Allora: $ V_{\\pi}(s_{0}) \\approx \\frac{3 + (0.75 \\cdot 0.8 )+ (0.0239 \\cdot 0.3)  + ( 0.187 \\cdot 0.6) +(0.0239 \\cdot 0.3)  + (0.0239 \\cdot 0.3)  + (1.5 \\cdot 0.9) +  ( 0.187 \\cdot 0.6)  + (0.0239 \\cdot 0.3)  + ( 0.187 \\cdot 0.6) }{5.9} = 0.9 $\n",
        "\n",
        "\n",
        "Oppure, campionando traiettorie diverse:\n",
        "\n",
        " - $ V_{\\pi}(s_{0}) \\approx \\frac{(1.5 \\cdot 0.9) + ((0.0239 \\cdot 0.3) ⋅ 9)}{ 1.71}  = 0.82 $.\n",
        "\n",
        " - $ V_{\\pi}(s_{0}) \\approx \\frac{(3⋅3) + (1.5 \\cdot 0.9) + ((0.0239 \\cdot 0.3) ⋅ 6)}{ 10.65}  = 0.97 $.\n",
        "\n",
        "Si puo notare che lo stimatore in senso statistico è \"biased\"; ma ha una varianza piccola.\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "*\n",
        "\n",
        "Weighted importance sampling v.s. Ordinary importance sampling\n",
        "\n",
        "Nella pratica il weighted importance sampling è preferito all’ordinary importance sampling  nell’off-policy learning, perché può produrre stime con una varianza molto inferiore.\n",
        "\n",
        "Nel testo di Sutton e Barto, Second Edition, PDF 127, si legge:\n",
        "\n",
        "\n",
        "\" Ordinary importance sampling is unbiased whereas weighted importance sampling is biased, though the bias converges asymptotically to zero\" ( significa che la distorsione dello stimatore weighted importance si avvicina sempre di più allo zero all’aumentare del numero di campioni (tale stimatore è consistente).\n",
        "\n",
        "\n",
        "\" In practice, the weighted estimator usually has dramatically lower variance and is strongly preferred \"\n",
        "\n",
        "ed a pag PDF 128:\n",
        "\n",
        "\" The estimates of ordinary importance sampling will\n",
        "typically have infinite variance, and thus unsatisfactory convergence properties, every time that\n",
        "the scaled returns (cioè il \" ratio$\\cdot G_{t}$ \") have infinite variance—and this can easily happen in off-policy learning when trajectories contain loops \"\n",
        "\n",
        "Gli esempi porposti nel testo: Example 5.4- Example 5.5,\n",
        "sostengono tale linea."
      ],
      "metadata": {
        "id": "wmAOjqs25ace"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Algoritmo: Off-policy MC prediction (per la stima di $q_{\\pi}$)\n",
        "\n",
        "\n",
        "Premesse:\n",
        "\n",
        "\n",
        "$I$.\n",
        "\n",
        "Nel testo di Sutton e Barto, Second Edition, PDF 127, si legge\n",
        "\n",
        " \" The every-visit methods for ordinary and weighed importance sampling are both biased, though, again, the bias falls asymptotically to zero as the number of samples increases. In practice, every-visit methods are often preferred because they remove the need to keep track of which states have been visited and because they are much easier to extend to approximations.\"\n",
        "\n",
        " Per tale ragione, in questa e nella sezione successiva svilupperò \"Every-visit MC algorithm\"\n",
        "\n",
        "$\\;$\n",
        "\n",
        "$II$.\n",
        "\n",
        "Si consideri: una sequenza di traittorie campionate che partono tutte dallo stesso stato, i rispettivi return ottenuti $\\{G_{1}, G_{2}, ..., G_{n-1} \\}$ , ed i rispettivi importance sampling ratio della policy target e behavior, indicati in questo caso con $W_{i}= \\rho_{t_{i}:T(t_{i})-1} $ e qui chiamati \"weight\".\n",
        "\n",
        "La stima di una funzione costo per lo stato $s$ di partenza delle traittorie campionate che sfrutta lo stimatore \"weighted importance sampling\", puo essere implementata in modo incrementale nel seguente modo:\n",
        "\n",
        " $V_{n}(s) \\doteq \\frac {W_{1} \\cdot G_{1} + W_{2} \\cdot G_{2} + ... + W_{n-1}\\cdot G_{n-1}}{W_{1} + W_{2} + ... + W_{n-1}}$   , $\\;$ con $ n  \\ge 2 $\n",
        "\n",
        "\n",
        "  $V_{n+1}(s) = V_{n}(s) + \\frac{W_{n}}{C_{n}} [G_{n} - V_{n}(s) ]$\n",
        "\n",
        "  dove, $V_{1}$ è un valore arbitrario; mentre $C_{n}$ è la somma cumulativa dei weights relativi ai primi $n$ return:\n",
        "\n",
        "  $C_{n+1} = C_{n} + W_{n+1} $ ,    $\\;$ con $C_{0}=0$.\n",
        "  Ovvero:\n",
        "\n",
        "  $C_{1}  = 0 +  W_{1} $\n",
        "\n",
        "  $C_{2}  = C_{1} +  W_{2}  = W_{1} +  W_{2} $\n",
        "\n",
        "  $C_{3}  = C_{2} +  W_{3}  = W_{1} +  W_{2} +  W_{3} $\n",
        "  \n",
        "  $C_{n}  =  W_{1} + W_{2} + ... + W_{n}$\n",
        "  \n",
        "\n",
        "$\\;$\n",
        "\n",
        "  Nota a margine, risulta che:\n",
        "\n",
        "  $  \\frac{\\sum_{k=1} ^{n} W_{k} \\cdot G_{k} } {\\sum_{k=1} ^{n} W_{k}} \\; = \\;  \\frac {W_{1} \\cdot G_{1} + W_{2} \\cdot G_{2} + ... + W_{n-1}\\cdot G_{n-1} + W_{n}\\cdot G_{n}}{W_{1} + W_{2} + ... + W_{n-1} + W_{n}} \\;  =  \\;   V_{n}(s) + \\frac{W_{n}}{C_{n}} [G_{n} - V_{n}(s) ]  $\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "\n",
        "Per tanto, per ottenere la funzione costo senza aspettare che siano state campionate tutte le traiettorie e poi ricaviti tutti i rispettivi $G_{t}$ e ratio, possiamo sfruttare la strategia incrementale, che ci cosente un update continuo, episode-by-episode.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "**Algoritmo**:\n",
        "\n",
        "Si ricorda lo scopo dell'algoritmo è stimare la funzione $q_{\\pi}(s,a)$ della la policy target $\\pi$, avendo a disposizione gli episodi generati da un altra policy detta behavior $b$, differente $b \\ne \\pi$. Entrambe le policy sono considerate fisse e date all'inizio.\n",
        "\n",
        "0.\n",
        "\n",
        "Input: inserisci una policy target $\\pi$ arbitraria\n",
        "\n",
        "Inizializzare la tabella $q_{\\pi}(s,a)$, e la cumulative sum dei weights $C(s,a)$\n",
        "\n",
        "Definisci una policy behavior $b$ facendo l'assunsione di coverage su $\\pi$\n",
        "\n",
        "1.\n",
        "\n",
        "Loop per numero di episodi che si vuole generare.\n",
        "\n",
        "1.A)\n",
        "Genera un episodio utilizzando la politica $b$\n",
        "\n",
        "1.B)\n",
        "Inizializza il return $G$ a zero; e il weight, ovvero il ratio dell'importance sampling, $W$ ad uno.\n",
        "\n",
        "\n",
        "2.\n",
        "\n",
        "Loop per ogni step $t$ nell'episodio (nota ad ogni step corrisponde un stato ed un azione: $(s_{t},a_{t})$):\n",
        "\n",
        "2.A) Calcolare return $G$ ottenuto fino allo step $t$.\n",
        "\n",
        "2.B) Aggiorna la cumulative sum dei weights  $C(s_{t},a_{t})$\n",
        "\n",
        "2.C) Aggiorna la tabella $q_{\\pi}$ nelle coordinate $(s_{t},a_{t})$ tramite l'implementazione incrementale sfruttando lo  stimatore  weighted importance sampling:\n",
        "\n",
        "$ q_{\\pi}(s_{t},a_{t}) ← q_{\\pi}(s_{t},a_{t}) + \\frac{W}{C(s_{t},a_{t})} \\cdot [ G - q_{\\pi}(s_{t},a_{t}) ] $\n",
        "\n",
        "2.D) Aggiorna i weights in base a\n",
        "$W ← W \\cdot \\frac{ \\pi(a_{t}|s_{t})}{ b (a_{t}|s_{t})} $\n",
        "\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "NOTA:\n",
        "\n",
        "L'algoritmo si applica anche in caso di on-policy, semplicemente facendo in modo che $\\pi = b$ ; e $W$ sia per tutto il training sempre uguale ad $1$\n"
      ],
      "metadata": {
        "id": "cjUGiG3_sN5k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##  MC control problem (optimization policy)\n",
        "\n",
        "\n",
        "Scopo:\n",
        "\n",
        "avere una stima delle policy e della q-function ottimali.\n",
        "\n",
        "Come:\n",
        "\n",
        "Generare episodi (in pratica azioni) con behavior policy, e migliorare la target policy. Questa tecnica richiede la condizione di coverage sopra esposta.\n",
        "\n",
        "Un vantaggio di questa separazione è che la target policy può essere deterministica (ad esempio, greedy), mentre la behavior policy può continuare a campionare qualunque azione disponibile.\n",
        "\n",
        "Nel codice proposto la target policy $\\pi$ è greedy rispetto $q_{\\pi}$\n",
        "\n",
        "Per garantire che $\\pi$ converga alla politica ottima, deve essere possibile raccogliere un numero infinito di return per ciascuna coppia di stato e azione. Questo può essere fatto scegliendo $b$ come \"soft policy.\"\n",
        "\n",
        "Un aspetto interessante che sottolinea il testo di Sutton e Barto, Second Edition, PDF 132:\n",
        "\n",
        "\"The target policy converges to optimal at all encountered states even though actions are selected according to a different soft policy b, which may change between or even within episodes.\"\n",
        "\n",
        "Questo è sottolineato anche nel box contenente la formulazione dello pseudocodice di pag PDF 133, dove \"b: any soft policy\" viene messa prima della generazione di ogni episodio.\n",
        "\n",
        "\n",
        "$\\;$\n",
        "\n",
        "Potenziali problemi.\n",
        "\n",
        "Nel testo di Sutton e Barto, Second Edition, PDF 133,\n",
        "\n",
        "\" A potential problem is that this method learns only from the tails of episodes, when all of the remaining actions in the episode are greedy. If nongreedy actions are common, then learning will be slow, particularly for states appearing in the early portions of long episodes. Potentially, this could greatly slow learning. There has been insucient experience with off-policy Monte Carlo methods to assess how serious this problem is.\""
      ],
      "metadata": {
        "id": "PcHEymtnKu6w"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Algoritmo Off-Policy Monte Carlo Control\n",
        "\n"
      ],
      "metadata": {
        "id": "hADVYRpeKbAp"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rWBvA35Y8nzm",
        "outputId": "24cc9eeb-53e7-4f2e-cef2-cdb3e0d611ad"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Action Space: Discrete(2)\n",
            "Dimention Action Space: 2\n",
            "Observation Space: Tuple(Discrete(32), Discrete(11), Discrete(2))\n",
            "Max Episode Steps: None\n",
            "Reward Range: (-inf, inf)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/usr/local/lib/python3.10/dist-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n"
          ]
        }
      ],
      "source": [
        "import gym\n",
        "import pandas as pd\n",
        "from collections import defaultdict\n",
        "import numpy as np\n",
        "import random\n",
        "\n",
        "import matplotlib\n",
        "from matplotlib import pyplot as plt\n",
        "from mpl_toolkits.mplot3d import Axes3D\n",
        "from mpl_toolkits.axes_grid1 import make_axes_locatable\n",
        "%matplotlib inline\n",
        "from IPython import display\n",
        "import pygame\n",
        "\n",
        "\n",
        "env = gym.make('Blackjack-v1')\n",
        "\n",
        "def query_environment(name):\n",
        "    env = gym.make(name)\n",
        "    spec = gym.spec(name)\n",
        "    print(f\"Action Space: {env.action_space}\")\n",
        "    print(f\"Dimention Action Space: {env.action_space.n}\")\n",
        "    print(f\"Observation Space: {env.observation_space}\")\n",
        "    print(f\"Max Episode Steps: {spec.max_episode_steps}\")\n",
        "    print(f\"Reward Range: {env.reward_range}\")\n",
        "\n",
        "\n",
        "def plot_policy(policy):\n",
        "\n",
        "    def get_Z(player_hand, dealer_showing, usable_ace):\n",
        "        if (player_hand, dealer_showing, usable_ace) in policy:\n",
        "            return policy[player_hand, dealer_showing, usable_ace]\n",
        "        else:\n",
        "            return 1\n",
        "\n",
        "    def get_figure(usable_ace, ax):\n",
        "        x_range = np.arange(1, 11)\n",
        "        y_range = np.arange(11, 22)\n",
        "        X, Y = np.meshgrid(x_range, y_range)\n",
        "        Z = np.array([[get_Z(player_hand, dealer_showing, usable_ace) for dealer_showing in x_range] for player_hand in range(21, 10, -1)])\n",
        "        surf = ax.imshow(Z, cmap=plt.get_cmap('Pastel2', 2), vmin=0, vmax=1, extent=[1, 11, 11, 22])\n",
        "        plt.xticks(x_range, ('A', '2', '3', '4', '5', '6', '7', '8', '9', '10'))\n",
        "        plt.yticks(y_range)\n",
        "        ax.set_xlabel('Dealer Showing')\n",
        "        ax.set_ylabel('Player Hand')\n",
        "        ax.grid(color='black', linestyle='-', linewidth=1)\n",
        "        divider = make_axes_locatable(ax)\n",
        "        cax = divider.append_axes(\"right\", size=\"5%\", pad=0.1)\n",
        "        cbar = plt.colorbar(surf, ticks=[0, 1], cax=cax)\n",
        "        cbar.ax.set_yticklabels(['0 (STICK)','1 (HIT)'])\n",
        "        cbar.ax.invert_yaxis()\n",
        "\n",
        "    fig = plt.figure(figsize=(8, 8))\n",
        "    ax = fig.add_subplot(121)\n",
        "    ax.set_title('Usable Ace', fontsize=12)\n",
        "    get_figure(True,ax)\n",
        "    ax = fig.add_subplot(122)\n",
        "    ax.set_title('No Usable Ace', fontsize=12)\n",
        "    get_figure(False, ax)\n",
        "    plt.show()\n",
        "\n",
        "query_environment('Blackjack-v1')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Codici sottostanti sono ispirati dal codice :\n",
        "# https://github.com/aditya1702/Machine-Learning-and-Data-Science/blob/master/Implementation%20of%20Reinforcement%20Learning%20Algorithms/Tensorflow%20Implementations/Monte%20Carlo%20Methods/Off-Policy%20Monte%20Carlo%20Control%20with%20Importance%20Sampling.ipynb\n",
        "\n",
        "\n",
        "def create_behaviour_policy(nA):\n",
        "\n",
        "    def policy_fn(obs):\n",
        "        # Una funzione che ha obs come input, e restituisce un vettore di probabilità realtivo alle azioni,\n",
        "        # in questo caso rappresentate dagli indici posizionali del vettore.\n",
        "        A = np.ones(nA, dtype=float) / nA\n",
        "        return A\n",
        "\n",
        "    return policy_fn\n",
        "# Per capire\n",
        "#s = create_behaviour_policy(3)(\"Di tutto\")\n",
        "#print(s) # [0.33333333 0.33333333 0.33333333]\n",
        "\n",
        "\n",
        "\n",
        "def create_target_policy(Q):\n",
        "\n",
        "    def policy_fn(state):\n",
        "        A = np.zeros_like(Q[state], dtype=float)\n",
        "        #print(A)\n",
        "        best_action = np.argmax(Q[state])\n",
        "        A[best_action] = 1.0\n",
        "        return A\n",
        "\n",
        "    return policy_fn\n",
        "# Per capire.\n",
        "# terminato il training, ed ottenuta optimal_Q\n",
        "#Q = optimal_Q\n",
        "#s = create_target_policy1(Q) # definisce la funzione che qui ho chiamato s\n",
        "#print( s( (14, 3, True) ) )\n",
        "#\n",
        "#oppure direttamente:\n",
        "#s = create_target_policy(Q)( (14, 3, True) )\n",
        "#\n",
        "##A = [0. 0.]\n",
        "##s( (14, 3, True) ) = [0. 1.]"
      ],
      "metadata": {
        "id": "JO3s_fem82NQ"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mc_control_off_policy_importance_sampling(behaviour_policy, env, num_episodes, discount=1.0):\n",
        "\n",
        "    # Fase di inizializzazione\n",
        "    Q = defaultdict(lambda: np.zeros(env.action_space.n))\n",
        "    C = defaultdict(lambda: np.zeros(env.action_space.n)) # Somma cumulativa dei weights dell'importance sampling formula\n",
        "    target_policy = create_target_policy(Q)\n",
        "\n",
        "    for i_episode in range(1, num_episodes+1):\n",
        "\n",
        "\n",
        "        if i_episode % 100_000 == 0:\n",
        "              print(\"\\rEpisode {}/{}.\".format(i_episode, num_episodes))\n",
        "\n",
        "\n",
        "        state = env.reset()\n",
        "        episode = []\n",
        "        while(True):\n",
        "            probs = behaviour_policy(state)  #  probs = [0.5 0.5]\n",
        "            action = np.random.choice(len(probs), p=probs)\n",
        "            next_state, reward, done, _ = env.step(action)\n",
        "            episode.append((state, action, reward))\n",
        "            if done:\n",
        "                break\n",
        "            state = next_state\n",
        "\n",
        "        G = 0.0\n",
        "        W = 1.0\n",
        "\n",
        "        for t in range(len(episode))[::-1]:\n",
        "            state, action, reward = episode[t]\n",
        "            G = discount*G + reward\n",
        "            C[state][action] = C[state][action] + W\n",
        "            Q[state][action] = Q[state][action]  + (W/C[state][action]) * (G - Q[state][action])\n",
        "\n",
        "            # Condizione critica:\n",
        "            # Se in un certo state, l’azione intrapresa dalla behaviour_policy non coincide con l'azione intrapresa\n",
        "            # dalla target (che si ricorda è deterministica); allora la probabilità sarà 0, quindi passa al prossimo episodio\n",
        "            # Vedi  nella condizione W = W * (target_policy(state)[action]/behaviour_policy(state)[action])\n",
        "            # l'action deve essere la stessa !\n",
        "            if action != np.argmax(target_policy(state)):\n",
        "                #print(\"\\t in break\") # Nota ci finisce spesso ... non sempre le azioni campionate (50% possibilità)\n",
        "                # dalla behaviour_policy sono greedy\n",
        "                break\n",
        "\n",
        "            W = W * (target_policy(state)[action]/behaviour_policy(state)[action])\n",
        "            # W di fatto ha valori del tipo  2.0 , 4.0 , 8.0 , 16.00, 32.00\n",
        "            # Il 32.00 si verifica quanto tutte le azioni campionate in precedenza in un singolo episodio\n",
        "            # dalla behaviour_policy sono greedy (e nel caso del BJ sono tutte azioni di \"hit\" = 1)\n",
        "            # Episodi del tipo:\n",
        "            #((14, 3, True), 1, 0.0), ((16, 3, True), 1, 0.0), ((19, 3, True), 1, 0.0), ((12, 3, False), 1, 0.0), ((18, 3, False), 0, 1.0)\n",
        "            #\n",
        "            # Formula update alternativo:\n",
        "            # essendo la target_policy deterministica : W = W * 1./behavior_policy(state)[action]\n",
        "\n",
        "\n",
        "    return Q,C\n",
        "\n",
        "\n",
        "behaviour_policy = create_behaviour_policy(env.action_space.n)\n",
        "#print(behaviour_policy(\"di tutto\")) # [0.5 0.5]\n",
        "optimal_Q, Cw = mc_control_off_policy_importance_sampling( behaviour_policy, env, num_episodes=300_000)"
      ],
      "metadata": {
        "id": "Wbxb4uv_NjY6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f4d5c707-405f-4cb7-8e95-f15776bc5307"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 100000/300000.\n",
            "Episode 200000/300000.\n",
            "Episode 300000/300000.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dpq = pd.DataFrame(Cw.items(),columns=['state', 'action'])\n",
        "dpq.head(9)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "_FlZMupr-VNS",
        "outputId": "20ed3b2a-4aa3-4efe-bb19-028a4af68f68"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             state            action\n",
              "0   (18, 6, False)    [689.0, 705.0]\n",
              "1   (19, 4, False)    [684.0, 644.0]\n",
              "2   (14, 4, False)    [794.0, 885.0]\n",
              "3   (17, 9, False)    [737.0, 653.0]\n",
              "4  (16, 10, False)  [3088.0, 2896.0]\n",
              "5   (19, 1, False)    [680.0, 649.0]\n",
              "6  (12, 10, False)  [3145.0, 3259.0]\n",
              "7   (12, 7, False)    [832.0, 810.0]\n",
              "8   (15, 1, False)    [763.0, 771.0]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5e682183-beb3-4a30-bfdb-9f0b0f549003\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>state</th>\n",
              "      <th>action</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(18, 6, False)</td>\n",
              "      <td>[689.0, 705.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(19, 4, False)</td>\n",
              "      <td>[684.0, 644.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(14, 4, False)</td>\n",
              "      <td>[794.0, 885.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(17, 9, False)</td>\n",
              "      <td>[737.0, 653.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(16, 10, False)</td>\n",
              "      <td>[3088.0, 2896.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(19, 1, False)</td>\n",
              "      <td>[680.0, 649.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(12, 10, False)</td>\n",
              "      <td>[3145.0, 3259.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(12, 7, False)</td>\n",
              "      <td>[832.0, 810.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(15, 1, False)</td>\n",
              "      <td>[763.0, 771.0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5e682183-beb3-4a30-bfdb-9f0b0f549003')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5e682183-beb3-4a30-bfdb-9f0b0f549003 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5e682183-beb3-4a30-bfdb-9f0b0f549003');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ec2ea459-678c-4662-822f-43ca960f74e6\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ec2ea459-678c-4662-822f-43ca960f74e6')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ec2ea459-678c-4662-822f-43ca960f74e6 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Costruisci optimal_policy e poi fai il grafico\n",
        "optimal_policy = {}\n",
        "for i in optimal_Q:\n",
        "  optimal_policy[i] = np.argmax(optimal_Q[i])\n",
        "\n",
        "plot_policy(optimal_policy)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 387
        },
        "id": "2EP6q2oYPdzc",
        "outputId": "7f6d2e38-3a37-4955-bc6f-b9fd10c7f143"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x800 with 4 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvgAAAFyCAYAAACA4a7AAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYIElEQVR4nO3de1wU9f4/8NeCutwEBEVAEVdRIVQkL3kpASUV71ZqZoqaeSxMjNIkjgmHY2anvJu3Sq3U7GSoWVoeE0hTExU18wK24v2WwrqsXITP7w9/7Nfluos77DK+no/HPnA+M/N5v4e297wZZgeFEEKAiIiIiIhkwcbSCRARERERkfmwwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwadaZe3atVAoFEhLS6ty29DQUISGhkqfFBHRYyI+Ph4KhQK3bt2qctvmzZtj3Lhx0idFRGWwwSeTVVXg27ZtK/vGesSIEVAoFHjnnXcsnQoRWVjJhQc7Oztcvny5zPrQ0FC0bdvWbPHGjRsHJyenCtc7OTnJvrHu0qULFAoFli9fbulUiKwSG3wiE2k0Gnz//fdo3rw5Nm7cCCGEpVMiIiuQn5+PDz74wNJpyF5GRgYOHTqE5s2bY/369ZZOh8gqscEnMtHmzZtRVFSEzz//HBcvXkRqaqqlUyIiK9ChQwesXr0aV65csXQqsvbVV1/Bw8MDH3/8MX777TecP3/e0ikRWR02+FQjlixZgsDAQDg4OKBBgwbo1KkTNmzYoF+flZWF119/HW3atIG9vT3c3d0xfPjwCgu3TqfDP/7xD7i7u8PZ2Rljx47FnTt3qswjPz8fs2fPhp+fH5RKJXx8fDBjxgzk5+cbfSzr16/Hs88+i7CwMAQEBFR4Ben06dMYMWIEGjVqBHt7e7Rp0wZxcXEG21y+fBkTJkxA48aNoVQqERgYiM8//9zoXIjIerz77rsoKioy6ir+/fv3kZiYiJYtW0KpVKJ58+Z49913TapFxiosLERCQgJatWoFOzs7uLu74+mnn8auXbv02xw/fhzjxo1DixYtYGdnB09PT0yYMAF///13uXPeunULI0aMgLOzM9zd3REdHY28vLwqc8nOzsa0adPg4+MDpVIJPz8/zJs3D8XFxUYfz4YNG/DCCy9g4MCBcHFxMTiXPOzgwYPo378/GjRoAEdHR7Rv3x6LFi0y2Ob06dN44YUX4ObmBjs7O3Tq1Anbtm0zOhcia1XH0gmQ/K1evRpTp07FCy+8oD8JHD9+HAcPHsRLL70EADh06BB+++03vPjii2jatCnOnz+P5cuXIzQ0FH/++SccHBwM5pwyZQpcXV0RHx+PM2fOYPny5cjKykJycjIUCkW5eRQXF2Pw4MHYu3cvJk2ahICAAJw4cQILFizA2bNnsWXLliqP5cqVK9izZw/WrVsHABg1ahQWLFiApUuXol69evrtjh8/jmeeeQZ169bFpEmT0Lx5c5w7dw7ff/895syZAwC4fv06unbtCoVCgSlTpqBRo0bYsWMHXnnlFWg0GkybNq0a320ishSVSoWxY8di9erVmDlzJry9vSvcduLEiVi3bh1eeOEFvPXWWzh48CDmzp2LU6dOISkpyax5xcfHY+7cuZg4cSK6dOkCjUaDtLQ0HDlyBM8++ywAYNeuXfjrr78wfvx4eHp64uTJk1i1ahVOnjyJAwcOlKmrI0aMQPPmzTF37lwcOHAAixcvxp07d/DFF19UmIdOp0NISAguX76Mf/zjH2jWrBl+++03xMbG4urVq1i4cGGVx3Lw4EFkZmZizZo1qFevHp577jmsX78e7777rsF2u3btwsCBA+Hl5YXo6Gh4enri1KlT2L59O6KjowEAJ0+eRI8ePdCkSRPMnDkTjo6O+OabbzB06FBs3rwZw4YNM/E7TWRFBJGJZs+eLQCImzdvlrs+MDBQhISE6JeHDBkiAgMDK51Tp9OVGdu/f78AIL744gv92Jo1awQA0bFjR1FQUKAf//DDDwUAsXXrVv1YSEiIQR5ffvmlsLGxEb/++qtBnBUrVggAYt++fZXmKIQQH330kbC3txcajUYIIcTZs2cFAJGUlGSwXc+ePUX9+vVFVlaWwXhxcbH+36+88orw8vISt27dMtjmxRdfFC4uLuV+T4jI+pTUpUOHDolz586JOnXqiKlTp+rXh4SEGNTA9PR0AUBMnDjRYJ63335bABC//PJLpfEiIyOFo6NjhesdHR1FZGSkfjkoKEgMGDCg0jnLqzcbN24UAERqaqp+rKT+Dx482GDb119/XQAQx44d04/5+voa5JGYmCgcHR3F2bNnDfadOXOmsLW1FRcuXKg0RyGEmDJlivDx8dHX0p9//lkAEEePHtVvc//+faFSqYSvr6+4c+eOwf4P1+DevXuLdu3aiby8PIP13bt3F61ataoyFyJrxlt0SHKurq64dOkSDh06VOE29vb2+n8XFhbi77//hp+fH1xdXXHkyJEy20+aNAl169bVL7/22muoU6cOfvzxxwpj/Pe//0VAQAD8/f1x69Yt/atXr14AgD179lR5LOvXr8eAAQNQv359AECrVq3QsWNHg9t0bt68idTUVEyYMAHNmjUz2L/kKpgQAps3b8agQYMghDDIp2/fvsjJySn3uInIurVo0QJjxozBqlWrcPXq1XK3KalTMTExBuNvvfUWAOCHH34wa06urq44efIkMjIyKtzm4Rqcl5eHW7duoWvXrgBQbi2KiooyWH7jjTcAoMoa/Mwzz6BBgwYGNS88PBxFRUVVfp7p/v372LRpE0aOHKmvpb169YKHh4dBDT569CjUajWmTZsGV1dXgzlK9rt9+zZ++eUXjBgxAnfv3tXn8vfff6Nv377IyMgo94lIRLUFG3ySxMO/zn3nnXfg5OSELl26oFWrVoiKisK+ffsMtr937x7ee+89/X2ZDRs2RKNGjZCdnY2cnJwy87dq1cpg2cnJCV5eXpV+2CojIwMnT55Eo0aNDF6tW7cGANy4caPSYzp16hSOHj2KHj16IDMzU/8KDQ3F9u3bodFoAAB//fUXAFT6WLybN28iOzsbq1atKpPP+PHjjcqHiKzTP//5T9y/f7/Ce/GzsrJgY2MDPz8/g3FPT0+4uroiKyvrkXN4uAb/61//QnZ2Nlq3bo127dph+vTpOH78uMH2t2/fRnR0NBo3bgx7e3s0atQIKpUKAIyqwS1btoSNjU2VNXjnzp1lal54eDiAqmvezz//jJs3b6JLly76+qtWqxEWFoaNGzfq7+M/d+4cgMprcGZmJoQQmDVrVpl8Zs+ebVQ+RNaM9+CTyezs7AA8aMrLo9Pp9NsAQEBAAM6cOYPt27dj586d2Lx5Mz755BO89957SEhIAPDg6s+aNWswbdo0dOvWDS4uLlAoFHjxxRdN+vBVZYqLi9GuXTvMnz+/3PU+Pj6V7v/VV18BAN588028+eabZdZv3rxZ35wbkwsAvPzyy4iMjCx3m/bt2xs1FxFZlxYtWuDll1/GqlWrMHPmzAq3q+jzQlWxs7NDfn4+hBBl5hBCIC8vz6AG9+zZE+fOncPWrVvx888/49NPP8WCBQuwYsUKTJw4EcCDe+p/++03TJ8+HR06dICTkxOKi4vRr18/o2qwMcdSXFyMZ599FjNmzCh3fcnFloqUXKUfMWJEuetTUlIQFhZWZR4luQDA22+/jb59+5a7TekfwIhqEzb4ZDJfX18AwJkzZ8o0xTqdDhcvXkSfPn0Mxh0dHTFy5EiMHDkSBQUFeO655zBnzhzExsbCzs4O3377LSIjI/Hxxx/r98nLy0N2dna5OWRkZBgUcq1Wi6tXr6J///4V5t2yZUscO3YMvXv3NvnEKoTAhg0bEBYWhtdff73M+sTERKxfvx7jx49HixYtAAB//PFHhfM1atQI9evXR1FRkf7qFRHJxz//+U989dVXmDdvXpl1vr6+KC4uRkZGBgICAvTj169fR3Z2tr7GVsTX1xf379/HuXPnyjShmZmZKCoqKjOHm5sbxo8fj/Hjx0Or1aJnz56Ij4/HxIkTcefOHezevRsJCQl477339PtUdktPRkaG/gp/Sdzi4mI0b968wn1atmwJrVZbrZqXm5uLrVu3YuTIkXjhhRfKrJ86dSrWr1+PsLAwtGzZEsCDGlxRrJI6XbduXdZgkiXeokMm6927N+rVq4fly5eXubKzatUq3L9/HxEREfqx0o9Zq1evHp544gkIIVBYWAgAsLW1LfMHo5YsWYKioqJyc1i1apV+XwBYvnx5mbiljRgxApcvX8bq1avLrLt37x5yc3Mr3Hffvn04f/48xo8fjxdeeKHMa+TIkdizZw+uXLmCRo0aoWfPnvj8889x4cIFg3lKjtHW1hbPP/88Nm/eXO4PAjdv3qwwFyKyfi1btsTLL7+MlStX4tq1awbrSi5ElH5qTMlvFwcMGFDp3CV1bunSpWXWLVu2zGAboGwNdnJygp+fn/6RnLa2tgBQpgZX9lSbkjgllixZUiZuaSNGjMD+/fvx008/lVmXnZ2N+/fvV7hvUlIScnNzERUVVW4NHjhwIDZv3oz8/Hw8+eSTUKlUWLhwYZmLRCXH6OHhgdDQUKxcubLcz0qwBlNtxyv4ZDIPDw+89957+Oc//4mePXti8ODBcHBwwG+//YaNGzeiT58+GDRokH77Pn36wNPTEz169EDjxo1x6tQpLF261ODDqgMHDsSXX34JFxcXPPHEE9i/fz/+97//wd3dvdwcCgoK0Lt3b4wYMQJnzpzBJ598gqeffhqDBw+uMO8xY8bgm2++weTJk7Fnzx706NEDRUVFOH36NL755hv89NNP6NSpU7n7rl+/Hra2thWeeAcPHoy4uDh8/fXXiImJweLFi/H000/jySefxKRJk6BSqXD+/Hn88MMPSE9PBwB88MEH2LNnD5566im8+uqreOKJJ3D79m0cOXIE//vf/3D79m1j/nMQkZWKi4vDl19+iTNnziAwMFA/HhQUhMjISKxatQrZ2dkICQnB77//jnXr1mHo0KFV3mbSoUMHTJw4EYsWLUJGRobBoy5//PFHTJw4EUFBQfrtn3jiCYSGhqJjx45wc3NDWloavv32W0yZMgUA4OzsjJ49e+LDDz9EYWEhmjRpgp9//hlqtbrCHNRqNQYPHox+/fph//79+Oqrr/DSSy8ZxC1t+vTp2LZtGwYOHIhx48ahY8eOyM3NxYkTJ/Dtt9/i/PnzaNiwYbn7rl+/Hu7u7ujevXu56wcPHozVq1fjhx9+wHPPPYfly5dj0KBB6NChA8aPHw8vLy+cPn0aJ0+e1P+AsWzZMjz99NNo164dXn31VbRo0QLXr1/H/v37cenSJRw7dqzS/w5EVs1CT+8hGfjqq69E165dhaOjo1AqlcLf318kJCQYPHJMCCFWrlwpevbsKdzd3YVSqRQtW7YU06dPFzk5Ofpt7ty5I8aPHy8aNmwonJycRN++fcXp06fLPGat5HF0KSkpYtKkSaJBgwbCyclJjB49Wvz9998GcUs/JlMIIQoKCsS8efNEYGCgUCqVokGDBqJjx44iISHBIJ/S+7i7u4tnnnmm0u+HSqUSwcHB+uU//vhDDBs2TLi6ugo7OzvRpk0bMWvWLIN9rl+/LqKiooSPj4+oW7eu8PT0FL179xarVq2qNBYRWY+HH5NZWmRkpABQ5lHBhYWFIiEhQahUKlG3bl3h4+MjYmNjy9TPihQVFYlFixaJoKAgYWdnJ+zs7ERQUJBYvHixKCoqMtj23//+t+jSpYtwdXUV9vb2wt/fX8yZM8fgUcOXLl3S1ysXFxcxfPhwceXKFQFAzJ49W79dyWMy//zzT/HCCy+I+vXriwYNGogpU6aIe/fuGcQtXb+FEOLu3bsiNjZW+Pn5iXr16omGDRuK7t27i48++sggn4ddv35d1KlTR4wZM6bC74dOpxMODg5i2LBh+rG9e/eKZ599VtSvX184OjqK9u3biyVLlhjsd+7cOTF27Fjh6ekp6tatK5o0aSIGDhwovv322wpjEdUGCiFK/U6OiIiIiIhqLd6DT0REREQkI2zwiYiIiIhkhA0+EVEtUVBQAD8/P/z222+WTsUot27dgoeHBy5dumTpVIiIADw+dZQNPhGRBSxbtgzNmzeHnZ0dnnrqKfz+++9V7rNixQqoVCqDJ4mkpKSgV69ecHNzg4ODA1q1aoXIyEgUFBRg3LhxUCgUFb5KnlkeGhqKadOmGcTKzMzE+PHj0bRpUyiVSqhUKowaNQppaWn6bRQKBbZs2aJfLiwsxKhRo9CkSRP88ccfaNiwIcaOHav/y6BERObEOloxNvhERDVs06ZNiImJwezZs3HkyBEEBQWhb9++uHHjRoX7CCGwdOlSvPLKK/qxP//8E/369UOnTp2QmpqKEydOYMmSJahXrx6KioqwaNEiXL16Vf8CgDVr1uiXDx06VG6stLQ0dOzYEWfPnsXKlSvx559/IikpCf7+/njrrbfK3Uen02Hw4ME4dOgQ9u7di7Zt2wIAxo8fj/Xr1/Oxr0RkVqyjVbDsQ3yIiB4/Xbp0EVFRUfrloqIi4e3tLebOnVvhPocOHRI2NjZCo9HoxxYsWCCaN29udFwAIikpqcx4SEiIiI6OFkIIUVxcLAIDA0XHjh3LPG5RiAePtC093507d0T37t1F+/btxdWrV8vso1KpxKeffmp0nkREVWEdrZzs/9BVcXExrly5gvr160OhUFg6HSKyECEE7t69C29vb9jYVP7Ly7y8PBQUFJg8f+kao1QqoVQqDcYKCgpw+PBhxMbG6sdsbGwQHh6O/fv3Vzj/r7/+itatW+v/OBwAeHp64urVq0hNTUXPnj1Nyrci6enpOHnyJDZs2KD/Pj1cR21sbKDRaPTbq9Vq/POf/4SjoyO+//57ODg4GKwHgODgYOzevRvDhw83S45EZBmso8Ypr44+zNXV1WD52rVrCAkJgZOTE1JSUsqsB4AuXbrg119/NfjtQ2Vk3+BfuXIFPj4+lk6DiKzExYsX0bRp0wrX5+XlwbtZU9y5+bdJ8zo5OUGr1RqMzZ49G/Hx8QZjt27dQlFRERo3bmww3rhxY5w+fbrC+bOysuDt7W0wNnz4cPz0008ICQmBp6cnunbtit69e2Ps2LFwdnY2Kf8SGRkZAAB/f3/9WGV1NCYmRv9vX1/fSufeuHFjtXIiIuvCOlq58upoZaKjo9GiRQvs2rULDg4O5W7j7e2No0ePGp2D7Bv8kp/SZq9bhCc6dZAszl9/nkHs8EmY+99VaPFEG8axojhyOhbGqT6dNhfjn+pvcOWmPAUFBbhz82+sOfgjHJwcTZr74sWLBieE0ledHsW9e/dgZ2dnMGZra4s1a9bg3//+N3755RccPHgQ77//PubNm4fff/8dXl5eJscR5fztQ9ZRxpFbHDkdS03GYR01Tnl1tDIDBw7Eli1bsHLlSrz55pvlbmNvbw+dTmf0nLJv8Et+1WPnYA+H+k6SxbH7/z9x2Tk4MI6VxZHTsTDOozP2Vj0HJ0eT83F2dq7yik/Dhg1ha2uL69evG4xfv34dnp6ele534sSJctc1adIEY8aMwZgxY5CYmIjWrVtjxYoVSEhIMCl/AGjdujUA4PTp0wgODgbAOso48osjp2OpyTglWEcrV14drcyYMWMwePBgTJgwAUIIg9+Mlrh9+zYaNWpkdA58ig4RUQ2qV68eOnbsiN27d+vHiouLsXv3bnTr1q3C/YKDg3H69Okqrww1aNAAXl5eyM3NrVZ+HTp0wBNPPIGPP/4YxcXF1ZqDiEhKtb2OZmdnlxmLjIzE2rVrMWPGDHz00Udl1v/xxx9G/bBQQvZX8ImIrE1MTAwiIyPRqVMndOnSBQsXLkRubi7Gjx9f4T5hYWHQarU4efKk/tFpK1euRHp6OoYNG4aWLVsiLy8PX3zxBU6ePIklS5ZUKzeFQoE1a9YgPDwczzzzDOLi4tCkSZNqzUVEJJXaVkf9/f2h1Wrx/fff4+eff0ZKSkqZ/caMGQMbGxtERkZCCIHp06cDePD4zMOHD+P99983Ogc2+ERENWzkyJG4efMm3nvvPVy7dg0dOnTAzp07y3xg7GHu7u4YNmwY1q9fj7lz5wJ48FSFvXv3YvLkybhy5QqcnJwQGBiILVu2ICQkpNr5denSBWlpaZgzZw5effVV3Lx5s9pzERFJobbV0Vu3bsHLywvdu3fHwoULK9xv9OjRsLGxwZgxY1BcXIx33nkHW7duRbNmzfDMM88YHZ8NPhGRBUyZMgVTpkwxaZ+4uDg8++yziIuLg5OTE4KDg/Hll18avX9Fv5ZOTk4uM9a6dWusW7cOAKDRaODi4mJSrkREUqtNddSU+UaNGoVRo0bplxctWoT33nvP6BwB3oNPRFRrtG/fHvPmzYNarbZ0KkREtVJtq6O3bt3Cc889Z9DwG4NX8ImIapFx48ZZOgUiolqtNtXRhg0bYsaMGSbvxyv4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCMWbfDnzp2Lzp07o379+vDw8MDQoUNx5swZg21WrVqF0NBQODs7Q6FQIDs72zLJEhFZIdZRIiIqzaINfkpKCqKionDgwAHs2rULhYWF6NOnD3Jzc/Xb6HQ69OvXD++++64FMyUisk6so0REVFodSwbfuXOnwfLatWvh4eGBw4cPo2fPngCAadOmAQCSk5NrODsiIuvHOkpERKVZtMEvLScnBwDg5uZW7Tny8/ORn5+vX9ZoNACAS39lwc7B4dESrMTFTLXBV8axnjhyOhbGqb48nU7S+a0F6yjjMI51xJBjnMeljsqBQgghLJ0EABQXF2Pw4MHIzs7G3r17y6xPTk5GWFgY7ty5A1dX1wrniY+PR0JCgoSZElFtlpOTA2dn5wrXazQauLi4YNPJFDjUdzJqTt1dLUYGhlQ5t9RYR4moJsi5jsqF1VzBj4qKwh9//FHuSckUsbGxiImJ0S9rNBr4+Pggal4c/NoGPGqaFbqYqcb86FmIWZQIHz8V41hRHDkdC+NUX55Oh9jhkySb3xqwjjIO41hPDDnGeRzqqFxYRYM/ZcoUbN++HampqWjatOkjzaVUKqFUKsuMN23hC7920p2YSvj4qRjHSuPI6VgYx3S6u1rJ5rYGrKOMwzjWGUNOceReR+XEog2+EAJvvPEGkpKSkJycDJVKup86iYjkiHWUiIhKs2iDHxUVhQ0bNmDr1q2oX78+rl27BgBwcXGBvb09AODatWu4du0aMjMzAQAnTpxA/fr10axZs0f6EBkRkRywjhIRUWkWfQ7+8uXLkZOTg9DQUHh5eelfmzZt0m+zYsUKBAcH49VXXwUA9OzZE8HBwdi2bZul0iYishqso0REVJrFb9GpSnx8POLj46VPhoioFmIdJSKi0ix6BZ+IiIiIiMyLDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZsWiDP3fuXHTu3Bn169eHh4cHhg4dijNnzhhsk5eXh6ioKLi7u8PJyQnPP/88rl+/bqGMiYisC+soERGVZtEGPyUlBVFRUThw4AB27dqFwsJC9OnTB7m5ufpt3nzzTXz//ff473//i5SUFFy5cgXPPfecBbMmIrIerKNERFRaHUsG37lzp8Hy2rVr4eHhgcOHD6Nnz57IycnBZ599hg0bNqBXr14AgDVr1iAgIAAHDhxA165dLZE2EZHVYB0lIqLSLNrgl5aTkwMAcHNzAwAcPnwYhYWFCA8P12/j7++PZs2aYf/+/eWemPLz85Gfn69f1mg0AIBLf2XBzsFBstwvZqoNvjKO9cSR07EwTvXl6XSSzm8tWEcZh3GsI4Yc4zwudVQOFEIIYekkAKC4uBiDBw9GdnY29u7dCwDYsGEDxo8fb3CiAYAuXbogLCwM8+bNKzNPfHw8EhISaiRnIqp9cnJy4OzsXOF6jUYDFxcXbDqZAof6TkbNqburxcjAkCrnlhrrKBHVBDnXUbmwmiv4UVFR+OOPP/QnpeqKjY1FTEyMflmj0cDHxwdR8+Lg1zbgUdOs0MVMNeZHz0LMokT4+KkYx4riyOlYGKf68nQ6xA6fJNn81oB1lHEYx3piyDHO41BH5cIqGvwpU6Zg+/btSE1NRdOmTfXjnp6eKCgoQHZ2NlxdXfXj169fh6enZ7lzKZVKKJXKMuNNW/jCr510J6YSPn4qxrHSOHI6FsYxne6uVrK5rQHrKOMwjnXGkFMcuddRObHoU3SEEJgyZQqSkpLwyy+/QKUy/KmzY8eOqFu3Lnbv3q0fO3PmDC5cuIBu3brVdLpERFaHdZSIiEqz6BX8qKgobNiwAVu3bkX9+vVx7do1AICLiwvs7e3h4uKCV155BTExMXBzc4OzszPeeOMNdOvWjU9+ICIC6ygREZVl0QZ/+fLlAIDQ0FCD8TVr1mDcuHEAgAULFsDGxgbPP/888vPz0bdvX3zyySc1nCkRkXViHSUiotIs2uAb8wAfOzs7LFu2DMuWLauBjIiIahfWUSIiKs2i9+ATEREREZF5scEnIiIiIpIRNvhERERERDLCBp+IiIiISEbY4BMRERERyQgbfCIiIiIiGWGDT0REREQkI2zwiYiIiIhkhA0+EREREZGMsMEnIiIiIpIRNvhERERERDLCBp+IiIiISEbY4BMRERERyQgbfCIiIiIiGbFog5+amopBgwbB29sbCoUCW7ZsMVh//fp1jBs3Dt7e3nBwcEC/fv2QkZFhmWSJiKwQ6ygREZVm0QY/NzcXQUFBWLZsWZl1QggMHToUf/31F7Zu3YqjR4/C19cX4eHhyM3NtUC2RETWh3WUiIhKq2PJ4BEREYiIiCh3XUZGBg4cOIA//vgDgYGBAIDly5fD09MTGzduxMSJE2syVSIiq8Q6SkREpVntPfj5+fkAADs7O/2YjY0NlEol9u7da6m0iIhqDdZRIqLHk0Wv4FfG398fzZo1Q2xsLFauXAlHR0csWLAAly5dwtWrVyvcLz8/X39SAwCNRgMAuPRXFuwcHCTL92Km2uAr41hPHDkdC+NUX55OJ+n81oh1lHEYx3Ix5BjncayjtZVCCCEsnQQAKBQKJCUlYejQofqxw4cP45VXXsGxY8dga2uL8PBw2NjYQAiBHTt2lDtPfHw8EhISaihrIqptcnJy4OzsXOF6jUYDFxcXbDqZAof6TkbNqburxcjAkCrnlhrrKBHVBDnXUbmw2iv4ANCxY0ekp6cjJycHBQUFaNSoEZ566il06tSpwn1iY2MRExOjX9ZoNPDx8UFcXBwCAgIky1WtVmPWrFlITEyESqViHCuKUxIjZlEifPykO5aLmWrMj2ac6saR+r2m0+kwadIkyea3Vuaso1Hz4uDXVro6Ktf3NuNYXxw5HUtNxsnT6RA7/PGro7WRVTf4JVxcXAA8+MBYWloaEhMTK9xWqVRCqVSWGff19YW/v79kOZZQqVSMY6VxfPxU8GsnXXPCOI9G6veAVquVbO7awBx1tGkLX1m95xiHceR0LDURR3f38a6jtYlFG3ytVovMzEz9slqtRnp6Otzc3NCsWTP897//RaNGjdCsWTOcOHEC0dHRGDp0KPr06WPBrImIrAfrKBERlWbRBj8tLQ1hYWH65ZJfCUdGRmLt2rW4evUqYmJicP36dXh5eWHs2LGYNWuWpdIlIrI6rKNERFSaRRv80NBQVPYZ36lTp2Lq1Kk1mBERUe3COkpERKVZ7XPwiYiIiIjIdGzwiYiIiIhkhA0+EREREZGMsMEnIiIiIpIRNvhERERERDLCBp+IiIiISEbY4BMRERERyQgbfCIiIiIiGWGDT0REREQkI2zwiYiIiIhkhA0+EREREZGMsMEnIiIiIpIRNvhERERERDLCBp+IiIiISEbY4BMRERERyYhFG/zU1FQMGjQI3t7eUCgU2LJli8F6rVaLKVOmoGnTprC3t8cTTzyBFStWWCZZIiIrxDpKRESlWbTBz83NRVBQEJYtW1bu+piYGOzcuRNfffUVTp06hWnTpmHKlCnYtm1bDWdKRGSdWEeJiKi0OpYMHhERgYiIiArX//bbb4iMjERoaCgAYNKkSVi5ciV+//13DB48uIayJCKyXqyjRERUmlFX8DUajdEvc+revTu2bduGy5cvQwiBPXv24OzZs+jTp49Z4xARSY11lIiIaopRV/BdXV2hUCiMmrCoqOiREnrYkiVLMGnSJDRt2hR16tSBjY0NVq9ejZ49e1a4T35+PvLz8/XLJSfLrKwsODg4mC230tRqtcFXxrGeOCVzX8yU9lhK5mec6sWR+r2m0+kknb8qcqijl/7Kgp2EdVSu723Gsb44cjqWmoyTZ+E6SsZTCCFEVRulpKTo/33+/HnMnDkT48aNQ7du3QAA+/fvx7p16zB37lxERkZWLxGFAklJSRg6dKh+7KOPPsLq1avx0UcfwdfXF6mpqYiNjUVSUhLCw8PLnSc+Ph4JCQnVyoGI5C8nJwfOzs4VrtdoNHBxccGmkylwqO9k1Jy6u1qMDAypdG7WUSKSC0vVUTKeUQ3+w3r37o2JEydi1KhRBuMbNmzAqlWrkJycXL1ESp2Y7t27BxcXFyQlJWHAgAH67SZOnIhLly5h586d5c5T3pUnHx8fxMXFISAgoFq5GUOtVmPWrFlITEyESqViHCuKI6djYZzq0+l0mDRpklWcmGprHY2aFwe/ttLV0YuZasyPnoWYRYnw8ZPuvcA4jCOnY6nJOHk6HWKHW0cdpcqZ/CHb/fv3l/uItU6dOmHixIlmSQoACgsLUVhYCBsbw48J2Nraori4uML9lEollEplmXFfX1/4+/ubLb+KqFQqxrHSOHI6FsYxnVarlWxuU9XWOtq0hS/82knX4Jfw8VMxDuPUSBw5HUtNxNHdtZ46SpUz+TGZPj4+WL16dZnxTz/9FD4+PibNpdVqkZ6ejvT0dAAPruSlp6fjwoULcHZ2RkhICKZPn47k5GSo1WqsXbsWX3zxBYYNG2Zq2kREVoN1lIiIpGTyFfwFCxbg+eefx44dO/DUU08BAH7//XdkZGRg8+bNJs2VlpaGsLAw/XJMTAwAIDIyEmvXrsXXX3+N2NhYjB49Grdv34avry/mzJmDyZMnm5o2EZHVYB0lIiIpmdzg9+/fHxkZGfjkk09w+vRpAMCgQYMwefJkk688hYaGorKPAHh6emLNmjWmpkhEZNVYR4mISErV+kNXTZs2xfvvv2/uXIiIHhuso0REJJVqNfjZ2dn4/fffcePGjTIf1Bo7dqxZEiMikjPWUSIikorJDf7333+P0aNHQ6vVwtnZ2eAPtygUCp6YiIiqwDpKRERSMvkpOm+99RYmTJgArVaL7Oxs3LlzR/+6ffu2FDkSEckK6ygREUnJ5Ab/8uXLmDp1Khwk/HPlRERyxjpKRERSMrnB79u3L9LS0qTIhYjoscA6SkREUjL5HvwBAwZg+vTp+PPPP9GuXTvUrVvXYP3gwYPNlhwRkRyxjhIRkZRMbvBfffVVAMC//vWvMusUCgWKiooePSsiIhljHSUiIimZ3OCXfpwbERGZhnWUiIikZPI9+EREREREZL2q9YeucnNzkZKSggsXLqCgoMBg3dSpU82SGBGRnLGOEhGRVExu8I8ePYr+/ftDp9MhNzcXbm5uuHXrFhwcHODh4cETExFRFVhHiYhISibfovPmm29i0KBBuHPnDuzt7XHgwAFkZWWhY8eO+Oijj6TIkYhIVlhHiYhISiY3+Onp6XjrrbdgY2MDW1tb5Ofnw8fHBx9++CHeffddKXIkIpIV1lEiIpKSyQ1+3bp1YWPzYDcPDw9cuHABAODi4oKLFy+aNzsiIhliHSUiIimZ3OAHBwfj0KFDAICQkBC89957WL9+PaZNm4a2bduaNFdqaioGDRoEb29vKBQKbNmyxWC9QqEo9/Wf//zH1LSJiKwG6ygREUnJ5Ab//fffh5eXFwBgzpw5aNCgAV577TXcvHkTq1atMmmu3NxcBAUFYdmyZeWuv3r1qsHr888/h0KhwPPPP29q2kREVoN1lIiIpGTyU3Q6deqk/7eHhwd27txZ7eARERGIiIiocL2np6fB8tatWxEWFoYWLVpUOyYRkaWxjhIRkZSq9Rx8S7h+/Tp++OEHrFu3rtLt8vPzkZ+fr1/WaDQAgKysLDg4OEiWn1qtNvjKONYTR07HwjjVp9PpJJ2/NnjUOnrpryzYSVhHL2aqDb4yDuPU5hhyjJPHOlprKIQQwpgNg4ODoVAoqtzuyJEj1UtEoUBSUhKGDh1a7voPP/wQH3zwAa5cuQI7O7sK54mPj0dCQkK1ciAi+cvJyYGzs3OF6zUaDVxcXLDpZAoc6jsZNafurhYjA0OqnJt1lIjkwJJ1lIxj9BX8h08YQgjMnTsXkydPhpubmxR5lfH5559j9OjRlZ6UACA2NhYxMTH6ZY1GAx8fH8TFxSEgIECy/NRqNWbNmoXExESoVCrGsaI4cjoWxqk+nU6HSZMmSTa/MVhHK1fyXohZlAgfP+neCxcz1ZgfzTjWGkd7pObOCzX1PaupOir18eTpdIgdbtk6SsYxusGfPXu2wfLHH3+M6OjoGrmP89dff8WZM2ewadOmKrdVKpVQKpVlxn19feHv7y9FegZUKhXjWGkcOR0L45hOq9VKNrexWEeN4+Ongl876X6QYBzrjpNz58HXmqg9NfU9q6k6KvXx6O5avo6ScUx+io4lfPbZZ+jYsSOCgoIsnQoRUa3EOkpE9Piw6IdstVotMjMz9ctqtRrp6elwc3NDs2bNADz41fB///tffPzxx5ZKk4jIarGOEhFRaRZt8NPS0hAWFqZfLrnnMzIyEmvXrgUAfP311xBCYNSoUZZIkYjIqrGOEhFRaUY3+IsXLzZYvn//PtauXYuGDRsajE+dOtXo4KGhoajqIT6TJk2y+AfjiIjMgXWUiIhqgtEN/oIFCwyWPT098eWXXxqMKRQKk05MRESPE9ZRIiKqCUY3+FL/ERoiIrljHSUioppQK56iQ0RERERExmGDT0REREQkI2zwiYiIiIhkhA0+EREREZGMmPQc/Pv372PDhg3o27cvGjduLFVOREQW1/imgNO9yh8/WUKrNW47gHWUiB4fUtVRqppJV/Dr1KmDyZMnIy8vT6p8iIhkjXWUiIikZvItOl26dEF6eroEqRARPR5YR4mISEom3aIDAK+//jpiYmJw8eJFdOzYEY6Ojgbr27dvb7bkiIjkiHWUiIikZHKD/+KLLwIw/FPqCoUCQggoFAoUFRWZLzsiIhliHSUiIimZ3ODzLzESET0a1lEiIpKSyQ2+r6+vFHkQET02WEeJiEhK1XoO/pdffokePXrA29sbWVlZAICFCxdi69atZk2OiEiuWEeJiEgqJjf4y5cvR0xMDPr374/s7Gz9vaKurq5YuHChufMjIpId1lEiIpKSyQ3+kiVLsHr1asTFxcHW1lY/3qlTJ5w4ccKkuVJTUzFo0CB4e3tDoVBgy5YtZbY5deoUBg8eDBcXFzg6OqJz5864cOGCqWkTEVkN1lEiIpKSyQ2+Wq1GcHBwmXGlUonc3FyT5srNzUVQUBCWLVtW7vpz587h6aefhr+/P5KTk3H8+HHMmjULdnZ2pqZNRGQ1WEeJiEhKJn/IVqVSIT09vcyHxHbu3ImAgACT5oqIiEBERESF6+Pi4tC/f398+OGH+rGWLVualjARkZVhHSUiIimZ3ODHxMQgKioKeXl5EELg999/x8aNGzF37lx8+umnZkusuLgYP/zwA2bMmIG+ffvi6NGjUKlUiI2NxdChQyvcLz8/H/n5+fpljUYDAMjKyoKDg4PZ8iut5LF3Uj/+jnGsMwbjWH8cnU4n6fymYB0tX8l74GKmtO+FkvkZxzrjaGvwvFBT37OaqqNSH0+eFdVRqpxCCCFM3Wn9+vWIj4/HuXPnAADe3t5ISEjAK6+8Uv1EFAokJSXpTzrXrl2Dl5cXHBwc8O9//xthYWHYuXMn3n33XezZswchISHlzhMfH4+EhIRq50FE8paTkwNnZ+cK12s0Gri4uCA5ORlOTk5GzanVahEaGlrl3A9jHSWi2spa6ihVrFoNfgmdTgetVgsPD49HT6TUienKlSto0qQJRo0ahQ0bNui3Gzx4MBwdHbFx48Zy5ynvypOPjw/i4uJM/tW3KdRqNWbNmoXExESoVCrGsaI4cjoWxqk+nU6HSZMmWd2JiXX0/5S8F2IWJcLHT7r3wsVMNeZHM461xtEe4XnBWuNYax2lsky+RWf27NmYMGECfH194eDgINmvaxs2bIg6dergiSeeMBgPCAjA3r17K9xPqVRCqVSWGff19YW/v7/Z8yxNpVIxjpXGkdOxMI7ptFqtZHObinW0cj5+Kvi1k+4HCcax7jg5dx585XnB+uJYUx2lypn8FJ2tW7eiZcuW6N27NzZs2GBwlcec6tWrh86dO+PMmTMG42fPnuVfgSSiWo11lIiIpGRyg5+eno5Dhw4hMDAQ0dHR8PT0xGuvvYZDhw6ZHFyr1SI9PR3p6ekAHvyKKT09Xf985unTp2PTpk1YvXo1MjMzsXTpUnz//fd4/fXXTY5FRGQtWEeJiEhKJjf4ABAcHIzFixfjypUr+Oyzz3Dp0iX06NED7du3x6JFi5CTk2PUPGlpaQgODtY/DzomJgbBwcF47733AADDhg3DihUr8OGHH6Jdu3b49NNPsXnzZjz99NPVSZuIyGqwjhIRkVSq1eCXEEKgsLAQBQUFEEKgQYMGWLp0KXx8fLBp06Yq9w8NDYUQosxr7dq1+m0mTJiAjIwM3Lt3D+np6RgyZMijpExEZFVYR4mIyNyq1eAfPnwYU6ZMgZeXF958800EBwfj1KlTSElJQUZGBubMmYOpU6eaO1ciItlgHSUiIqmY3OC3a9cOXbt2hVqtxmeffYaLFy/igw8+gJ+fn36bUaNG4ebNm2ZNlIhILlhHiYhISiY/JnPEiBGYMGECmjRpUuE2DRs2RHFx8SMlRkQkV6yjREQkJZMb/FmzZkmRBxHRY4N1lIiIpGRygw8Aly5dwrZt23DhwgUUFBQYrJs/f75ZEiMikjPWUSIikorJDf7u3bsxePBgtGjRAqdPn0bbtm1x/vx5CCHw5JNPSpEjEZGssI4SEZGUTP6QbWxsLN5++22cOHECdnZ22Lx5My5evIiQkBAMHz5cihyJiGSFdZSIiKRkcoN/6tQpjB07FgBQp04d3Lt3D05OTvjXv/6FefPmmT1BIiK5YR0lIiIpmdzgOzo66u8X9fLywrlz5/Trbt26Zb7MiIhkinWUiIikZPI9+F27dsXevXsREBCA/v3746233sKJEyfw3XffoWvXrlLkSEQkK6yjREQkJZMb/Pnz50Or1QIAEhISoNVqsWnTJrRq1YpPfiAiMgLrKBERScnkBr9Fixb6fzs6OmLFihVmTYiISO5YR4mISEom34NPRERERETWy6gr+A0aNIBCoTBqwtu3bz9SQkREcsQ6SkRENcWoBn/hwoUSp0FEJG+so0REVFOMavAjIyMlCZ6amor//Oc/OHz4MK5evYqkpCQMHTpUv37cuHFYt26dwT59+/bFzp07JcmHiEgqrKNERFRTjL4Hv7i4GPPmzUOPHj3QuXNnzJw5E/fu3Xuk4Lm5uQgKCsKyZcsq3KZfv364evWq/rVx48ZHiklEZCmso0REVBOMforOnDlzEB8fj/DwcNjb22PRokW4ceMGPv/882oHj4iIQERERKXbKJVKeHp6VjsGEZG1YB0lIqKaYHSD/8UXX+CTTz7BP/7xDwDA//73PwwYMACffvopbGykexhPcnIyPDw80KBBA/Tq1Qv//ve/4e7uXuH2+fn5yM/P1y9rNBoAQFZWFhwcHCTLU61WG3xlHOuJI6djYZzq0+l0ks5vDNbRypW8By5mSvteKJmfcawzjpbnBauNYw11lIyjEEIIYzZUKpXIzMyEj4+PfszOzg6ZmZlo2rTpoyeiUJS5d/Trr7+Gg4MDVCoVzp07h3fffRdOTk7Yv38/bG1ty50nPj4eCQkJj5wPEclTTk4OnJ2dK1yv0Wjg4uKC5ORkODk5GTWnVqtFaGholXOzjhKRHFiyjpJxjL6Cf//+fdjZ2RmM1a1bF4WFhWZPqsSLL76o/3e7du3Qvn17tGzZEsnJyejdu3e5+8TGxiImJka/rNFo4OPjg7i4OAQEBEiWq1qtxqxZs5CYmAiVSsU4VhRHTsfCONWn0+kwadIkyeY3Buto5UreCzGLEuHjJ9174WKmGvOjGcda42iP8LxgrXGsoY6ScYxu8IUQGDduHJRKpX4sLy8PkydPhqOjo37su+++M2+GD2nRogUaNmyIzMzMCk9MSqXSIMcSvr6+8Pf3lyy3EiqVinGsNI6cjoVxTKfVaiWb21iso8bx8VPBr510P0gwjnXHybnz4CvPC9YXxxrqKBnH6Aa/vEe8vfzyy2ZNpiqXLl3C33//DS8vrxqNS0RkDqyjRERUE4xu8NesWWP24FqtFpmZmfpltVqN9PR0uLm5wc3NDQkJCXj++efh6emJc+fOYcaMGfDz80Pfvn3NngsRkdRYR4mIqCYY3eBLIS0tDWFhYfrlkns+IyMjsXz5chw/fhzr1q1DdnY2vL290adPHyQmJpb7q2MioscR6ygREZVm0QY/NDQUlT3E56effqrBbIiIah/WUSIiKk26By8TEREREVGNY4NPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRiza4KempmLQoEHw9vaGQqHAli1bKtx28uTJUCgUWLhwYY3lR0Rk7VhHiYioNIs2+Lm5uQgKCsKyZcsq3S4pKQkHDhyAt7d3DWVGRFQ7sI4SEVFpdSwZPCIiAhEREZVuc/nyZbzxxhv46aefMGDAgBrKjIiodmAdJSKi0iza4FeluLgYY8aMwfTp0xEYGGjUPvn5+cjPz9cvazQaAEBWVhYcHBwkyRMA1Gq1wVfGsZ44cjoWxqk+nU4n6fzWqjbW0YuZ0r4XSuZnHOuMo+V5wWrjPK51tDZSCCGEpZMAAIVCgaSkJAwdOlQ/NnfuXOzZswc//fQTFAoFmjdvjmnTpmHatGkVzhMfH4+EhATpEyaiWiknJwfOzs4VrtdoNHBxcUFycjKcnJyMmlOr1SI0NLTKuaXGOkpENUHOdVQurPYK/uHDh7Fo0SIcOXIECoXC6P1iY2MRExOjX9ZoNPDx8UFcXBwCAgKkSBXAg5+aZ82ahcTERKhUKsaxojhyOhbGqT6dTodJkyZJNr81qq11NGZRInz8pHsvXMxUY350zb23a+p45BJHe4TnBWuN8zjW0drKahv8X3/9FTdu3ECzZs30Y0VFRXjrrbewcOFCnD9/vtz9lEollEplmXFfX1/4+/tLla6eSqViHCuNI6djYRzTabVayea2VrW1jvr4qeDXTrofJErU1Hu7po5HLnFy7jz4yvOC9cV5HOtobWW1Df6YMWMQHh5uMNa3b1+MGTMG48ePt1BWRES1B+soEdHjyaINvlarRWZmpn5ZrVYjPT0dbm5uaNasGdzd3Q22r1u3Ljw9PdGmTZuaTpWIyCqxjhIRUWkWbfDT0tIQFhamXy655zMyMhJr1661UFZERLUH6ygREZVm0QY/NDQUpjzEp6L7RYmIHleso0REVJpF/5ItERERERGZFxt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMmLRBj81NRWDBg2Ct7c3FAoFtmzZYrA+Pj4e/v7+cHR0RIMGDRAeHo6DBw9aJlkiIivEOkpERKVZtMHPzc1FUFAQli1bVu761q1bY+nSpThx4gT27t2L5s2bo0+fPrh582YNZ0pEZJ1YR4mIqLQ6lgweERGBiIiICte/9NJLBsvz58/HZ599huPHj6N3795Sp0dEZPVYR4mIqDSLNvimKCgowKpVq+Di4oKgoKAKt8vPz0d+fr5+WaPRAACysrLg4OAgWX5qtdrgK+NYTxw5HQvjVJ9Op5N0/tqgttTRi5nSvhdK5q+p93ZNHY9c4mh5XrDaOKyjtYdCCCEsnQQAKBQKJCUlYejQoQbj27dvx4svvgidTgcvLy9s2bIFnTt3rnCe+Ph4JCQkSJwtEdVWOTk5cHZ2rnC9RqOBi4sLkpOT4eTkZNScWq0WoaGhVc4tNdZRIqoJcq6jcmH1V/DDwsKQnp6OW7duYfXq1RgxYgQOHjwIDw+PcrePjY1FTEyMflmj0cDHxwdxcXEICAiQLE+1Wo1Zs2YhMTERKpWKcawojpyOhXGqT6fTYdKkSZLNb81qWx2NWZQIHz/p3gsXM9WYH804ptIekU9NkFt9Yx2l0qy+wXd0dISfnx/8/PzQtWtXtGrVCp999hliY2PL3V6pVEKpVJYZ9/X1hb+/v9TpQqVSMY6VxpHTsTCO6bRarWRzW7vaVkd9/FTwayfdDxKMUz05dx58lUtNqKkYcorzONfR2qbWPQe/uLjY4N5QIiIyDesoEZG8WfQKvlarRWZmpn5ZrVYjPT0dbm5ucHd3x5w5czB48GB4eXnh1q1bWLZsGS5fvozhw4dbMGsiIuvBOkpERKVZtMFPS0tDWFiYfrnkns/IyEisWLECp0+fxrp163Dr1i24u7ujc+fO+PXXXxEYGGiplImIrArrKBERlWbRBj80NBSVPcTnu+++q8FsiIhqH9ZRIiIqrdbdg09ERERERBVjg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhGLNrgp6amYtCgQfD29oZCocCWLVv06woLC/HOO++gXbt2cHR0hLe3N8aOHYsrV65YLmEiIivDOkpERKVZtMHPzc1FUFAQli1bVmadTqfDkSNHMGvWLBw5cgTfffcdzpw5g8GDB1sgUyIi68Q6SkREpdWxZPCIiAhERESUu87FxQW7du0yGFu6dCm6dOmCCxcuoFmzZjWRIhGRVWMdJSKi0iza4JsqJycHCoUCrq6uFW6Tn5+P/Px8/bJGowEAZGVlwcHBQbLc1Gq1wVfGsZ44cjoWxqk+nU4n6fy1RW2ooxczpX0vlMzPOKbRyqgmyK2+sY5SaQohhLB0EgCgUCiQlJSEoUOHlrs+Ly8PPXr0gL+/P9avX1/hPPHx8UhISJAoSyKq7XJycuDs7Fzheo1GAxcXFyQnJ8PJycmoObVaLUJDQ6ucW2qso0RUE+RcR+WiVlzBLywsxIgRIyCEwPLlyyvdNjY2FjExMfpljUYDHx8fxMXFISAgQLIc1Wo1Zs2ahcTERKhUKsaxojhyOhbGqT6dTodJkyZJNr+1Yx1lHMap2RhyjPO419HaxOob/JKTUlZWFn755Zcqf6pTKpVQKpVlxn19feHv7y9VmnoqlYpxrDSOnI6FcUyn1Wolm9vasY4yDuNYLoac4jzOdbS2seoGv+SklJGRgT179sDd3d3SKRER1Sqso0REjx+LNvharRaZmZn6ZbVajfT0dLi5ucHLywsvvPACjhw5gu3bt6OoqAjXrl0DALi5uaFevXqWSpuIyGqwjhIRUWkWbfDT0tIQFhamXy655zMyMhLx8fHYtm0bAKBDhw4G++3ZswehoaE1lSYRkdViHSUiotIs2uCHhoaisof4WMkDfoiIrBbrKBERlWbRv2RLRERERETmxQafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhFRLdKzZ09s2LBB8ji3bt2Ch4cHLl26JHksIqKa9DjUUYs2+KmpqRg0aBC8vb2hUCiwZcsWg/Xfffcd+vTpA3d3dygUCqSnp1skTyIic6qq9lVk27ZtuH79Ol588UX9mKenJwIDA8vMFR8fjw4dOujrqIODg76ONm/eHAqFosLXuHHj0LBhQ4wdOxazZ8+W4DtARPRozFlHmzdvjoULF5bZtqSOlrds7XXUog1+bm4ugoKCsGzZsgrXP/3005g3b14NZ0ZEJJ2qal9FFi9ejPHjx8PG5v9KtxACTZo0qbKOhoeH68cOHTqEq1ev4urVq9i8eTMA4MyZM/qxRYsWAQDGjx+P9evX4/bt26YeIhGRpMxZR6vD2utonRqNVkpERAQiIiIqXD9mzBgAwPnz52soIyIi6VVV+8pz8+ZN/PLLL/qTRgl7e3v0798fw4YNK3e/kjo6bdo0/VijRo30/3ZzcwMAeHh4wNXV1WDfkt8MbN++3aRciYikZs46Wh2m1tGkpCS88sorjxzXWBZt8KWQn5+P/Px8/XJOTg4A4OzZs5LGzcrKAgCcOnUKOp2OcawojpyOhXGq7969ewAeXPE2Rm5urtFzl2yr0WgMxpVKJZRKpdHzVGbv3r1wcHBAQECAWearzMN1NDg4GMnJyQBYRxlHPnHkdCw1GYd1tHq6dOmCX3/9tUYbfAgrAUAkJSWVu06tVgsA4ujRo1XOM3v2bAGAL7744qvc18WLFyutIffu3ROenp4mz+vk5FRmbPbs2Y9U+x62YMEC0aJFizLjvr6+ol69esLR0VEAEEqlUjg6Ooq6deuKoKAg/XbR0dECKFtH9+zZIwCIO3fu6MdYR/nii6/KXnKuow+/StfR2bNnGyyXKK+OPuzNN98UoaGhVeZnTrK7gh8bG4uYmBj9cnZ2Nnx9fXHhwgW4uLhIFlej0cDHxwcXL16Es7Mz41hRHDkdC+NUnxACd+/ehbe3d6Xb2dnZQa1Wo6CgwOT5FQqFwZi5rjoBD66c2dnZlbtu+vTpGDduHFq1aoUFCxbg2WefxeLFi5GamlqtWA/X0X/961/YtWsXjh8/zjrKOLKJI6djqck4j0Mdfdij1NGH2dvbS/qblfLIrsGv6Fc5Li4ukr7pSzg7OzOOlcaR07EwTvUY25za2dlVeBKwlIYNG+LOnTsVrvPz8wMAeHl5wc/PT39PaHU8XEdzc3Ph6emJ48ePs44yjuziyOlYairO41BHSzxKHX3Y7du3De7Zrwl8Dj4RUS0QHByMa9euVXhyksoff/yBoKCgGo1JRCQFS9bR4ODgGo1p0QZfq9UiPT1d/3x7tVqN9PR0XLhwAcCDn3jS09Px559/Anjw+KH09HRcu3bNUikTET2yqmpfeYKDg9GwYUPs27fPYLy4uBiXL18uM1fJAwZK6ujNmzcBmFZHdTodDh8+jF69elXjKImIpGPOOiqlkjrap0+fGosJABb9kG3JhxJKvyIjI4UQQqxZs6bc9cZ84KJEXl6emD17tsjLy5PmIBjH6uPI6VgYRx6qqn0VmTFjhnjxxRcNxho3blzuXEFBQSIoKKjKOlrZh8M2bNgg2rRpI7v3AuMwjpyOpSbjWBNz1lFfX1+xYMGCMtuW/lBtdT5kW1JHa5pCCCOfdURERBZ17do1BAYG4siRI/D19ZU8XteuXTF16lS89NJLksciIqoJj0sd5T34RES1hKenJz777LNKfwVtLrdu3cJzzz2HUaNGSR6LiKimPC51lFfwiYiIiIhkhFfwiYiIiIhkhA0+EREREZGMyLrB379/P2xtbTFgwADJYsydOxedO3dG/fr14eHhgaFDh+LMmTNmjbF8+XK0b99e/wcsunXrhh07dpg1Rnk++OADKBQKTJs2zazzxsfHQ6FQGLz8/f3NGqPE5cuX8fLLL8Pd3R329vZo164d0tLSzBqjefPmZY5HoVAgKirKrHGKioowa9YsqFQq2Nvbo2XLlkhMTIQUd9ndvXsX06ZNg6+vL+zt7dG9e3ccOnTokeZMTU3FoEGD4O3tDYVCgS1bthisF0Lgvffeg5eXF+zt7REeHo6MjIxHiknmIXUtrYk6ClimlrKOGod11Diso2QsWTf4n332Gd544w2kpqbiypUrksRISUlBVFQUDhw4gF27dqGwsBB9+vRBbm6u2WI0bdoUH3zwAQ4fPoy0tDT06tULQ4YMwcmTJ80Wo7RDhw5h5cqVaN++vSTzBwYG4urVq/rX3r17zR7jzp076NGjB+rWrYsdO3bgzz//xMcff4wGDRqYNc6hQ4cMjmXXrl0AgOHDh5s1zrx587B8+XIsXboUp06dwrx58/Dhhx9iyZIlZo0DABMnTsSuXbvw5Zdf4sSJE+jTpw/Cw8Nx+fLlas+Zm5uLoKAgLFu2rNz1H374IRYvXowVK1bg4MGDcHR0RN++fZGXl1ftmGQeUtfSmqijQM3XUtZR47GOGod1lIxW4w/mrCF3794VTk5O4vTp02LkyJFizpw5NRL3xo0bAoBISUmRNE6DBg3Ep59+Ksncd+/eFa1atRK7du0SISEhIjo62qzzV/QcWXN75513xNNPPy15nNKio6NFy5YtRXFxsVnnHTBggJgwYYLB2HPPPSdGjx5t1jg6nU7Y2tqK7du3G4w/+eSTIi4uziwxAIikpCT9cnFxsfD09BT/+c9/9GPZ2dlCqVSKjRs3miUmVY8lamlN1VEhpKulrKOPhnW0aqyjVBnZXsH/5ptv4O/vjzZt2uDll1/G559/Lsmv4Eor+euRbm5uksxfVFSEr7/+Grm5uejWrZskMaKiojBgwACEh4dLMj8AZGRkwNvbGy1atMDo0aMleVzVtm3b0KlTJwwfPhweHh4IDg7G6tWrzR7nYQUFBfjqq68wYcIEKBQKs87dvXt37N69G2fPngUAHDt2DHv37kVERIRZ49y/fx9FRUWws7MzGLe3t5fkCiHw4C8QXrt2zeA95+Ligqeeegr79++XJCYZxxK1VOo6CkhfS1lHq491tHpYR8mApX/CkEr37t3FwoULhRBCFBYWioYNG4o9e/ZIGrOoqEgMGDBA9OjRw+xzHz9+XDg6OgpbW1vh4uIifvjhB7PHEEKIjRs3irZt24p79+4JIYQkV55+/PFH8c0334hjx46JnTt3im7duolmzZoJjUZj1jhKpVIolUoRGxsrjhw5IlauXCns7OzE2rVrzRrnYZs2bRK2trbi8uXLZp+7qKhIvPPOO0KhUIg6deoIhUIh3n//fbPHEUKIbt26iZCQEHH58mVx//598eWXXwobGxvRunVrs8yPUlee9u3bJwCIK1euGGw3fPhwMWLECLPEpOqp6VoqZR0VomZqKevoo2EdNQ7rKFVGlg3+6dOnRZ06dcT169f1Y1FRUeLll1+WNO7kyZOFr6+vuHjxotnnzs/PFxkZGSItLU3MnDlTNGzYUJw8edKsMS5cuCA8PDzEsWPH9GNSnJhKu3PnjnB2djb7r8nr1q0runXrZjD2xhtviK5du5o1zsP69OkjBg4cKMncGzduFE2bNhUbN24Ux48fF1988YVwc3OT5ESbmZkpevbsKQAIW1tb0blzZzF69Gjh7+9vlvl5YqodLFFLpayjQkhfS1lHHx3rqHFYR6kysmzwp0+frv8fquRlY2Mj7O3tRXZ2tiQxo6KiRNOmTcVff/0lyfyl9e7dW0yaNMmscyYlJZX5vgEQCoVC2Nraivv375s13sM6deokZs6cadY5mzVrJl555RWDsU8++UR4e3ubNU6J8+fPCxsbG7FlyxZJ5m/atKlYunSpwVhiYqJo06aNJPGEEEKr1epPFiNGjBD9+/c3y7ylT0znzp0TAMTRo0cNtuvZs6eYOnWqWWKS6Wq6ltZ0HRXC/LWUdfTRsI4aj3WUKiO7e/Dv37+PL774Ah9//DHS09P1r2PHjsHb2xsbN240azwhBKZMmYKkpCT88ssvUKlUZp2/IsXFxcjPzzfrnL1798aJEycMvm+dOnXC6NGjkZ6eDltbW7PGK6HVanHu3Dl4eXmZdd4ePXqUedTe2bNn4evra9Y4JdasWQMPDw/JHiWo0+lgY2P4v6ytrS2Ki4sliQcAjo6O8PLywp07d/DTTz9hyJAhksRRqVTw9PTE7t279WMajQYHDx6U7LMmVLmarKWWqqOA+Wsp6+ijYR2tPtZRMmDpnzDMLSkpSdSrV6/cq0szZswQnTp1Mmu81157Tbi4uIjk5GRx9epV/Uun05ktxsyZM0VKSopQq9Xi+PHjYubMmUKhUIiff/7ZbDEqIsWvlt966y2RnJws1Gq12LdvnwgPDxcNGzYUN27cMGuc33//XdSpU0fMmTNHZGRkiPXr1wsHBwfx1VdfmTWOEA/u62zWrJl45513zD53icjISNGkSROxfft2oVarxXfffScaNmwoZsyYYfZYO3fuFDt27BB//fWX+Pnnn0VQUJB46qmnREFBQbXnvHv3rjh69Kg4evSoACDmz58vjh49KrKysoQQQnzwwQfC1dVVbN26VRw/flwMGTJEqFQq/X3MVLNqspbWRB0VwnK1lHXUOKyjVWMdJWPJrsEfOHBghb/+OnjwoABgcG/kowJQ7mvNmjVmizFhwgTh6+sr6tWrJxo1aiR69+5dI829ENKcmEaOHCm8vLxEvXr1RJMmTcTIkSNFZmamWWOU+P7770Xbtm2FUqkU/v7+YtWqVZLE+emnnwQAcebMGUnmF0IIjUYjoqOjRbNmzYSdnZ1o0aKFiIuLE/n5+WaPtWnTJtGiRQtRr1494enpKaKioh75low9e/aU+/9KZGSkEOLBI95mzZolGjduLJRKpejdu7ek30+qXE3W0pqoo0JYrpayjhqHdbRqrKNkLIUQNfDsSCIiIiIiqhGyuwefiIiIiOhxxgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJ4tr3rw5Fi5caJHY48aNw9ChQ2s0pkKhwJYtW2o0JhHJG+soET2MDT6VMW7cOCgUCigUCtStWxeNGzfGs88+i88//xzFxcWWTs8kq1evRlBQEJycnODq6org4GDMnTvXojldvXoVERERFs2BiKTFOiot1lGiyrHBp3L169cPV69exfnz57Fjxw6EhYUhOjoaAwcOxP379y2dnoGCgoJyxz///HNMmzYNU6dORXp6Ovbt24cZM2ZAq9XWcIaGPD09oVQqLZoDEUmPdVQ6rKNElWODT+VSKpXw9PREkyZN8OSTT+Ldd9/F1q1bsWPHDqxdu1a/XXZ2NiZOnIhGjRrB2dkZvXr1wrFjx/Trz507hyFDhqBx48ZwcnJC586d8b///a/S2FXNGR8fjw4dOuDTTz+FSqWCnZ1dufNs27YNI0aMwCuvvAI/Pz8EBgZi1KhRmDNnTpltP/roI3h5ecHd3R1RUVEoLCzUr7tz5w7Gjh2LBg0awMHBAREREcjIyAAACCHQqFEjfPvtt/rtO3ToAC8vL/3y3r17oVQqodPpABj+avn8+fNQKBT47rvvEBYWBgcHBwQFBWH//v0G+a1evRo+Pj5wcHDAsGHDMH/+fLi6ulb6fSQiy2IdZR0lshQ2+GS0Xr16ISgoCN99951+bPjw4bhx4wZ27NiBw4cP48knn0Tv3r1x+/ZtAIBWq0X//v2xe/duHD16FP369cOgQYNw4cKFCuNUNScAZGZmYvPmzfjuu++Qnp5e7jyenp44cOAAsrKyKj2uPXv24Ny5c9izZw/WrVuHtWvXGpx8x40bh7S0NGzbtg379++HEAL9+/dHYWEhFAoFevbsieTkZAAPTmKnTp3CvXv3cPr0aQBASkoKOnfuDAcHhwpziIuLw9tvv4309HS0bt0ao0aN0l/h27dvHyZPnozo6Gikp6fj2WefLffkSkTWj3WUdZSoRgiiUiIjI8WQIUPKXTdy5EgREBAghBDi119/Fc7OziIvL89gm5YtW4qVK1dWOH9gYKBYsmSJftnX11csWLDA6Dlnz54t6tatK27cuFHpcVy5ckV07dpVABCtW7cWkZGRYtOmTaKoqMjgWH19fcX9+/f1Y8OHDxcjR44UQghx9uxZAUDs27dPv/7WrVvC3t5efPPNN0IIIRYvXiwCAwOFEEJs2bJFPPXUU2LIkCFi+fLlQgghwsPDxbvvvqvfH4BISkoSQgihVqsFAPHpp5/q1588eVIAEKdOnRJCPPieDxgwwODYRo8eLVxcXCo9fiKyHNZR1lEiS+IVfDKJEAIKhQIAcOzYMWi1Wri7u8PJyUn/UqvVOHfuHIAHV57efvttBAQEwNXVFU5OTjh16lSFV56MmRMAfH190ahRo0pz9fLywv79+3HixAlER0fj/v37iIyMRL9+/Qw+5BYYGAhbW1uD/W7cuAEAOHXqFOrUqYOnnnpKv97d3R1t2rTBqVOnAAAhISH4888/cfPmTaSkpCA0NBShoaFITk5GYWEhfvvtN4SGhlaaa/v27Q3iA9DncObMGXTp0sVg+9LLRFR7sI6yjhJJrY6lE6Da5dSpU1CpVAAenHS8vLz0v1Z9WMl9jW+//TZ27dqFjz76CH5+frC3t8cLL7xQ4Qe6jJkTABwdHY3OuW3btmjbti1ef/11TJ48Gc888wxSUlIQFhYGAKhbt67B9gqFwqSnXLRr1w5ubm5ISUlBSkoK5syZA09PT8ybNw+HDh1CYWEhunfvXukcD+dQcuKvbU/aICLjsI6WxTpKZF5s8Mlov/zyC06cOIE333wTAPDkk0/i2rVrqFOnDpo3b17uPvv27cO4ceMwbNgwAA9OPOfPn68whjFzPoonnngCAJCbm2vU9gEBAbh//z4OHjyoP7n8/fffOHPmjH4uhUKBZ555Blu3bsXJkyfx9NNPw8HBAfn5+Vi5ciU6depk0om0tDZt2uDQoUMGY6WXiah2YB1lHSWqCbxFh8qVn5+Pa9eu4fLlyzhy5Ajef/99DBkyBAMHDsTYsWMBAOHh4ejWrRuGDh2Kn3/+GefPn8dvv/2GuLg4pKWlAQBatWql/wDXsWPH8NJLL1V6RcWYOY312muvITExEfv27UNWVhYOHDiAsWPHolGjRujWrZtRc7Rq1QpDhgzBq6++ir179+LYsWN4+eWX0aRJEwwZMkS/XWhoKDZu3IgOHTrAyckJNjY26NmzJ9avX4+QkBCT8i7tjTfewI8//oj58+cjIyMDK1euxI4dO/RXqIjIOrGOQp8/6yhRzWKDT+XauXMnvLy80Lx5c/Tr1w979uzB4sWLsXXrVv19lgqFAj/++CN69uyJ8ePHo3Xr1njxxReRlZWFxo0bAwDmz5+PBg0aoHv37hg0aBD69u2LJ598ssK4xsxprPDwcBw4cADDhw9H69at8fzzz8POzg67d++Gu7u70fOsWbMGHTt2xMCBA9GtWzcIIfDjjz8a/Do4JCQERUVFBveIhoaGlhmrjh49emDFihWYP38+goKCsHPnTrz55psVPtaOiKwD6+j/YR0lqlkKIYSwdBJEZJpXX30Vp0+fxq+//mrpVIiIaiXWUZIz3oNPVAt89NFHePbZZ+Ho6IgdO3Zg3bp1+OSTTyydFhFRrcE6So8TXsEnqgVGjBiB5ORk3L17Fy1atMAbb7yByZMnWzotIqJag3WUHids8ImIiIiIZIQfsiUiIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikpH/B2jlzVb4wrz7AAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dpq = pd.DataFrame(optimal_Q.items(),columns=['state', 'action'])\n",
        "dpq.head(9)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "PJ_-7JcM9qWc",
        "outputId": "f8a39cf3-04c3-484f-8334-4683c0e4c86c"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             state                                       action\n",
              "0  (19, 10, False)  [-0.04360753221010919, -0.7359232849766415]\n",
              "1  (12, 10, False)   [-0.5711954304365574, -0.4255017230893983]\n",
              "2   (9, 10, False)   [-0.5736213757816947, -0.2556306306306308]\n",
              "3   (18, 7, False)    [0.3970873786407768, -0.6151658767772509]\n",
              "4   (17, 10, True)  [-0.4755927475592748, -0.20949720670391067]\n",
              "5   (16, 1, False)   [-0.7099911582670209, -0.6032028469750887]\n",
              "6  (21, 10, False)                   [0.8986543313708998, -1.0]\n",
              "7  (20, 10, False)   [0.44092128095914174, -0.8612155388471164]\n",
              "8   (20, 2, False)     [0.624680306905371, -0.8822815533980582]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a525b8c0-70f8-426a-a646-54bf8c94fcf9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>state</th>\n",
              "      <th>action</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(19, 10, False)</td>\n",
              "      <td>[-0.04360753221010919, -0.7359232849766415]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(12, 10, False)</td>\n",
              "      <td>[-0.5711954304365574, -0.4255017230893983]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(9, 10, False)</td>\n",
              "      <td>[-0.5736213757816947, -0.2556306306306308]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(18, 7, False)</td>\n",
              "      <td>[0.3970873786407768, -0.6151658767772509]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(17, 10, True)</td>\n",
              "      <td>[-0.4755927475592748, -0.20949720670391067]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(16, 1, False)</td>\n",
              "      <td>[-0.7099911582670209, -0.6032028469750887]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(21, 10, False)</td>\n",
              "      <td>[0.8986543313708998, -1.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(20, 10, False)</td>\n",
              "      <td>[0.44092128095914174, -0.8612155388471164]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(20, 2, False)</td>\n",
              "      <td>[0.624680306905371, -0.8822815533980582]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a525b8c0-70f8-426a-a646-54bf8c94fcf9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a525b8c0-70f8-426a-a646-54bf8c94fcf9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a525b8c0-70f8-426a-a646-54bf8c94fcf9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6003c83b-307a-41ea-aa95-48d483eef1a7\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6003c83b-307a-41ea-aa95-48d483eef1a7')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6003c83b-307a-41ea-aa95-48d483eef1a7 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Per capire un pò meglio l'inner loop della funzione mc_control_off_policy_importance_sampling"
      ],
      "metadata": {
        "id": "mm-gtZIXf_6i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import copy\n",
        "\n",
        "def mc_off(behaviour_policy, env, discount=1.0, Q = copy.deepcopy(optimal_Q), C = copy.deepcopy(Cw) ):\n",
        "\n",
        "    C[(9, 10, False)][1] = 0\n",
        "    target_policy = create_target_policy(Q)\n",
        "\n",
        "    for i_episode in range(1, 4):\n",
        "\n",
        "        print(f\"\\nEP:{i_episode}\")\n",
        "        #Episodi inventati per capire il codice.\n",
        "\n",
        "        if  i_episode == 1:\n",
        "            # la prima azione immessa nella sequenza non è greedy, nè apporta un return G sufficentemente grande da renderla tale,\n",
        "            # e verificare la condizione action == np.argmax(target_policy(state))\n",
        "            # Per tanto, l'algoritmo aggiorna solo Q dell'ultimo \"state\" dell'episodio (Nota usando W=1), ignorando il resto dell'episodio\n",
        "            episode = [((4, 3, True), 1, 70.0),((6, 3, True), 1, 45.0), ((14, 3, True), 1, 0.0), ((12, 10, False), 1, 0.0), ((19, 10, False), 1, 1.0)]\n",
        "\n",
        "        if i_episode == 2 :\n",
        "            # la prima azione immessa nella sequenza non è greedy, ma apporta un return G sufficentemente grande da renderla tale.\n",
        "            # La condizione action == np.argmax(target_policy(state)) è verificata. Si passa allo state successivo.\n",
        "            # L'azione selezionata non è greedy, la condizione action == np.argmax(target_policy(state)) non è verificata;\n",
        "            # algoritmo di ferma li, e passa al prossimo episodio\n",
        "            episode = [((18, 7, False), 0, 70.0), ((12, 10, False), 0, - 3000.0), ((19, 10, False), 1, 3000.0)]\n",
        "\n",
        "        if i_episode == 3 :\n",
        "            # tutte le azioni sono greedy, l'algoritmo apprende da tutti gli state dell'episodio\n",
        "            episode = [((18, 7, False), 0, 0.5), ((9, 10, False), 1, 67.0), ((12, 10, False), 1, 2.5), ((12, 10, False), 1, 0.0), ((20, 10, False), 0, 0.0)]\n",
        "\n",
        "\n",
        "        print(episode[::-1])\n",
        "\n",
        "        G = 0.0\n",
        "        W = 1.0\n",
        "\n",
        "        for t in range(len(episode))[::-1]:\n",
        "            state, action, reward = episode[t]\n",
        "\n",
        "            G = discount*G + reward\n",
        "\n",
        "            print(\" \")\n",
        "            print(\"G=\", G)\n",
        "            print(f\"  IN   Q[{state}] =\", Q[state])\n",
        "\n",
        "            print(\"IN W=\",W)\n",
        "            C[state][action] = C[state][action] + W\n",
        "            print(f\"C[{state}][{action}] + W = \", C[state][action])\n",
        "            print(f\"(W/C[{state}][{action}]) = \", W/C[state][action])\n",
        "\n",
        "\n",
        "            print(f\"Formula  Q[{state}][{action}] = Q[{state}][{action}] + W/C[{state}][{action}] * (G - Q[{state}][{action}]) \")\n",
        "            Q[state][action] = Q[state][action]  + (W/C[state][action]) * (G - Q[state][action])\n",
        "            print(f\"  OUT modify Q[{state}][{action}] = \",Q[state][action])\n",
        "\n",
        "\n",
        "            if action != np.argmax(target_policy(state)):\n",
        "                print(\"\\t  --- in break ---- \")\n",
        "                break\n",
        "\n",
        "            W = W * (target_policy(state)[action]/behaviour_policy(state)[action])\n",
        "            print(\"UPDATE W=\",W)\n",
        "\n",
        "\n",
        "\n",
        "    return print(\"\\nEnd\")\n",
        "\n",
        "\n",
        "behaviour_policy = create_behaviour_policy(env.action_space.n)\n",
        "mc_off( behaviour_policy, env)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZsJfDojHcq1u",
        "outputId": "e9655560-0ffb-4810-f47b-efce1f3bfa24"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "EP:1\n",
            "[((19, 10, False), 1, 1.0), ((12, 10, False), 1, 0.0), ((14, 3, True), 1, 0.0), ((6, 3, True), 1, 45.0), ((4, 3, True), 1, 70.0)]\n",
            " \n",
            "G= 1.0\n",
            "  IN   Q[(19, 10, False)] = [-0.04360753 -0.73592328]\n",
            "IN W= 1.0\n",
            "C[(19, 10, False)][1] + W =  4068.0\n",
            "(W/C[(19, 10, False)][1]) =  0.0002458210422812193\n",
            "Formula  Q[(19, 10, False)][1] = Q[(19, 10, False)][1] + W/C[(19, 10, False)][1] * (G - Q[(19, 10, False)][1]) \n",
            "  OUT modify Q[(19, 10, False)][1] =  -0.7354965585054083\n",
            "\t  --- in break ---- \n",
            "\n",
            "EP:2\n",
            "[((19, 10, False), 1, 3000.0), ((12, 10, False), 0, -3000.0), ((18, 7, False), 0, 70.0)]\n",
            " \n",
            "G= 3000.0\n",
            "  IN   Q[(19, 10, False)] = [-0.04360753 -0.73549656]\n",
            "IN W= 1.0\n",
            "C[(19, 10, False)][1] + W =  4069.0\n",
            "(W/C[(19, 10, False)][1]) =  0.0002457606291472106\n",
            "Formula  Q[(19, 10, False)][1] = Q[(19, 10, False)][1] + W/C[(19, 10, False)][1] * (G - Q[(19, 10, False)][1]) \n",
            "  OUT modify Q[(19, 10, False)][1] =  0.0019660850331774116\n",
            "UPDATE W= 2.0\n",
            " \n",
            "G= 0.0\n",
            "  IN   Q[(12, 10, False)] = [-0.57119543 -0.42550172]\n",
            "IN W= 2.0\n",
            "C[(12, 10, False)][0] + W =  4904.0\n",
            "(W/C[(12, 10, False)][0]) =  0.0004078303425774878\n",
            "Formula  Q[(12, 10, False)][0] = Q[(12, 10, False)][0] + W/C[(12, 10, False)][0] * (G - Q[(12, 10, False)][0]) \n",
            "  OUT modify Q[(12, 10, False)][0] =  -0.5709624796084838\n",
            "\t  --- in break ---- \n",
            "\n",
            "EP:3\n",
            "[((20, 10, False), 0, 0.0), ((12, 10, False), 1, 0.0), ((12, 10, False), 1, 2.5), ((9, 10, False), 1, 67.0), ((18, 7, False), 0, 0.5)]\n",
            " \n",
            "G= 0.0\n",
            "  IN   Q[(20, 10, False)] = [ 0.44092128 -0.86121554]\n",
            "IN W= 1.0\n",
            "C[(20, 10, False)][0] + W =  6340.0\n",
            "(W/C[(20, 10, False)][0]) =  0.0001577287066246057\n",
            "Formula  Q[(20, 10, False)][0] = Q[(20, 10, False)][0] + W/C[(20, 10, False)][0] * (G - Q[(20, 10, False)][0]) \n",
            "  OUT modify Q[(20, 10, False)][0] =  0.4408517350157728\n",
            "UPDATE W= 2.0\n",
            " \n",
            "G= 0.0\n",
            "  IN   Q[(12, 10, False)] = [-0.57096248 -0.42550172]\n",
            "IN W= 2.0\n",
            "C[(12, 10, False)][1] + W =  4935.0\n",
            "(W/C[(12, 10, False)][1]) =  0.0004052684903748733\n",
            "Formula  Q[(12, 10, False)][1] = Q[(12, 10, False)][1] + W/C[(12, 10, False)][1] * (G - Q[(12, 10, False)][1]) \n",
            "  OUT modify Q[(12, 10, False)][1] =  -0.42532928064842995\n",
            "UPDATE W= 4.0\n",
            " \n",
            "G= 2.5\n",
            "  IN   Q[(12, 10, False)] = [-0.57096248 -0.42532928]\n",
            "IN W= 4.0\n",
            "C[(12, 10, False)][1] + W =  4939.0\n",
            "(W/C[(12, 10, False)][1]) =  0.0008098805426199635\n",
            "Formula  Q[(12, 10, False)][1] = Q[(12, 10, False)][1] + W/C[(12, 10, False)][1] * (G - Q[(12, 10, False)][1]) \n",
            "  OUT modify Q[(12, 10, False)][1] =  -0.4229601133832763\n",
            "UPDATE W= 8.0\n",
            " \n",
            "G= 69.5\n",
            "  IN   Q[(9, 10, False)] = [-0.57362138 -0.25563063]\n",
            "IN W= 8.0\n",
            "C[(9, 10, False)][1] + W =  8.0\n",
            "(W/C[(9, 10, False)][1]) =  1.0\n",
            "Formula  Q[(9, 10, False)][1] = Q[(9, 10, False)][1] + W/C[(9, 10, False)][1] * (G - Q[(9, 10, False)][1]) \n",
            "  OUT modify Q[(9, 10, False)][1] =  69.5\n",
            "UPDATE W= 16.0\n",
            " \n",
            "G= 70.0\n",
            "  IN   Q[(18, 7, False)] = [ 0.39708738 -0.61516588]\n",
            "IN W= 16.0\n",
            "C[(18, 7, False)][0] + W =  1046.0\n",
            "(W/C[(18, 7, False)][0]) =  0.015296367112810707\n",
            "Formula  Q[(18, 7, False)][0] = Q[(18, 7, False)][0] + W/C[(18, 7, False)][0] * (G - Q[(18, 7, False)][0]) \n",
            "  OUT modify Q[(18, 7, False)][0] =  1.4617590822179731\n",
            "UPDATE W= 32.0\n",
            "\n",
            "End\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sempre al fine di capire meglio il codice, sposto la condizione di break dell'inner loop e l'update di W; in modo da far sembrare il tutto più rigorosamente coerente con la teoria probabilistica del weighted importance Sampling: cioè se azione $a_{T-1}$ scelta nello stato dell'episodio $s_{T-1}$ dalla behaviour_policy ha probabilità zero di essere estratta dalla target_policy (quindi $W=0$), allora passa al prossimo episodio, senza aggionarare $Q$ con $G= R_{T}$, e senza aggiornare $C$.\n",
        "\n",
        "Cosa succede? l'algoritmo non esplora come nel precedente codice ( dove $Q$ ad ogni episodio veniva \"inizializzata-aggiornata\" nello step : $T-1$) e si polarizza solo sulle azioni considerate greedy."
      ],
      "metadata": {
        "id": "y_aw0wGg6zsj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def mc_control_off_policy_importance_sampling1(behaviour_policy, env, num_episodes, discount=1.0):\n",
        "\n",
        "    # Fase di inizializzazione\n",
        "    Q = defaultdict(lambda: np.zeros(env.action_space.n))\n",
        "    C = defaultdict(lambda: np.zeros(env.action_space.n)) # Somma cumulativa dei weights dell'importance sampling formula\n",
        "    target_policy = create_target_policy(Q)\n",
        "\n",
        "    for i_episode in range(1, num_episodes+1):\n",
        "\n",
        "\n",
        "        if i_episode % 250_000 == 0:\n",
        "              print(\"\\rEpisode {}/{}.\".format(i_episode, num_episodes))\n",
        "\n",
        "\n",
        "        state = env.reset()\n",
        "        episode = []\n",
        "        while(True):\n",
        "            probs = behaviour_policy(state)  #  probs = [0.5 0.5]\n",
        "            action = np.random.choice(len(probs), p=probs)\n",
        "            next_state, reward, done, _ = env.step(action)\n",
        "            episode.append((state, action, reward))\n",
        "            if done:\n",
        "                break\n",
        "            state = next_state\n",
        "\n",
        "        G = 0.0\n",
        "        W = 1.0\n",
        "\n",
        "        for t in range(len(episode))[::-1]:\n",
        "            state, action, reward = episode[t]\n",
        "\n",
        "            #       ---\n",
        "            if action != np.argmax(target_policy(state)):\n",
        "                break\n",
        "\n",
        "            W = W*(target_policy(state)[action]/behaviour_policy(state)[action])\n",
        "            #       ---\n",
        "\n",
        "            G = discount*G + reward\n",
        "            C[state][action] = C[state][action] + W\n",
        "\n",
        "            Q[state][action] = Q[state][action]  + (W/C[state][action]) * (G - Q[state][action])\n",
        "\n",
        "\n",
        "    return Q,C\n",
        "\n",
        "\n",
        "behaviour_policy = create_behaviour_policy(env.action_space.n)\n",
        "#print(behaviour_policy(\"di tutto\")) # [0.5 0.5]\n",
        "optimal_Q1, Cw1 = mc_control_off_policy_importance_sampling1( behaviour_policy, env, num_episodes=300_000)\n",
        "#Costruisci optimal_policy e poi fai il grafico\n",
        "\n",
        "optimal_policy1 = {}\n",
        "for i in optimal_Q1:\n",
        "  optimal_policy1[i] = np.argmax(optimal_Q1[i])\n",
        "\n",
        "plot_policy(optimal_policy1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 461
        },
        "outputId": "7d0fea4f-3581-40e1-8acc-4deb7e5e76c8",
        "id": "qrBAdx1x3_wq"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\rEpisode 250000/300000.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x800 with 4 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvgAAAFyCAYAAACA4a7AAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZhElEQVR4nO3deVxU9f4/8NeAOmyCgiKgiKNoECqSSy4lkKbibqVmpqiZ18LEKE3iknC5ZnbL3dwqtVKzm6FmaXlNIE1NVNTMBWzEfUthHEYW4fP7wx/zdVhncM7McHw9H4954Nk+7/eZGd7n7ZkzB4UQQoCIiIiIiGTBztoJEBERERGR+bDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBp1plzZo1UCgUSE9Pr3bdsLAwhIWFSZ8UEdEjIiEhAQqFAjdv3qx23RYtWmDcuHHSJ0VE5bDBJ5NVV+Dbtm0r+8Z6xIgRUCgUeOedd6ydChFZWemJBwcHB1y6dKnc8rCwMLRt29Zs8caNGwcXF5dKl7u4uMi+se7SpQsUCgWWLVtm7VSIbBIbfCITaTQafP/992jRogU2bNgAIYS1UyIiG1BQUIAPPvjA2mnIXmZmJg4ePIgWLVpg3bp11k6HyCaxwScy0aZNm1BcXIzPP/8cFy5cQFpamrVTIiIb0KFDB6xatQqXL1+2diqy9tVXX8HT0xMff/wxfvvtN5w7d87aKRHZHDb4ZBGLFy9GUFAQnJyc0LBhQ3Tq1Anr16/XL8/Ozsbrr7+Oxx57DI6OjvDw8MDw4cMrLdw6nQ7/+Mc/4OHhAVdXV4wdOxa3b9+uNo+CggLMmjUL/v7+UCqV8PX1xYwZM1BQUGD0vqxbtw7PPvsswsPDERgYWOkZpFOnTmHEiBFo3LgxHB0d8dhjjyEuLs5gnUuXLmHChAlo0qQJlEolgoKC8PnnnxudCxHZjnfffRfFxcVGncW/d+8ekpKS0KpVKyiVSrRo0QLvvvuuSbXIWEVFRUhMTETr1q3h4OAADw8PPPXUU9i5c6d+nWPHjmHcuHFo2bIlHBwc4OXlhQkTJuDvv/+ucMybN29ixIgRcHV1hYeHB6Kjo5Gfn19tLjk5OZg2bRp8fX2hVCrh7++PuXPnoqSkxOj9Wb9+PV544QUMHDgQbm5uBseSBx04cAD9+/dHw4YN4ezsjPbt22PhwoUG65w6dQovvPAC3N3d4eDggE6dOmHr1q1G50Jkq+pYOwGSv1WrVmHq1Kl44YUX9AeBY8eO4cCBA3jppZcAAAcPHsRvv/2GF198Ec2aNcO5c+ewbNkyhIWF4c8//4STk5PBmFOmTEGDBg2QkJCA06dPY9myZcjOzkZKSgoUCkWFeZSUlGDw4MHYs2cPJk2ahMDAQBw/fhzz58/HmTNnsHnz5mr35fLly9i9ezfWrl0LABg1ahTmz5+PJUuWoF69evr1jh07hqeffhp169bFpEmT0KJFC5w9exbff/89Zs+eDQC4du0aunbtCoVCgSlTpqBx48bYvn07XnnlFWg0GkybNq0GzzYRWYtKpcLYsWOxatUqzJw5Ez4+PpWuO3HiRKxduxYvvPAC3nrrLRw4cABz5szByZMnkZycbNa8EhISMGfOHEycOBFdunSBRqNBeno6Dh8+jGeffRYAsHPnTvz1118YP348vLy8cOLECaxcuRInTpzA/v37y9XVESNGoEWLFpgzZw7279+PRYsW4fbt2/jiiy8qzUOn0yE0NBSXLl3CP/7xDzRv3hy//fYbYmNjceXKFSxYsKDafTlw4ACysrKwevVq1KtXD8899xzWrVuHd99912C9nTt3YuDAgfD29kZ0dDS8vLxw8uRJbNu2DdHR0QCAEydOoEePHmjatClmzpwJZ2dnfPPNNxg6dCg2bdqEYcOGmfhME9kQQWSiWbNmCQDixo0bFS4PCgoSoaGh+ukhQ4aIoKCgKsfU6XTl5u3bt08AEF988YV+3urVqwUA0bFjR1FYWKif/+GHHwoAYsuWLfp5oaGhBnl8+eWXws7OTvz6668GcZYvXy4AiL1791aZoxBCfPTRR8LR0VFoNBohhBBnzpwRAERycrLBej179hT169cX2dnZBvNLSkr0/37llVeEt7e3uHnzpsE6L774onBzc6vwOSEi21Nalw4ePCjOnj0r6tSpI6ZOnapfHhoaalADMzIyBAAxceJEg3HefvttAUD88ssvVcaLjIwUzs7OlS53dnYWkZGR+ung4GAxYMCAKsesqN5s2LBBABBpaWn6eaX1f/DgwQbrvv766wKAOHr0qH6en5+fQR5JSUnC2dlZnDlzxmDbmTNnCnt7e3H+/PkqcxRCiClTpghfX199Lf35558FAHHkyBH9Ovfu3RMqlUr4+fmJ27dvG2z/YA3u1auXaNeuncjPzzdY3r17d9G6detqcyGyZbxEhyTXoEEDXLx4EQcPHqx0HUdHR/2/i4qK8Pfff8Pf3x8NGjTA4cOHy60/adIk1K1bVz/92muvoU6dOvjxxx8rjfHf//4XgYGBCAgIwM2bN/WPZ555BgCwe/fuavdl3bp1GDBgAOrXrw8AaN26NTp27Ghwmc6NGzeQlpaGCRMmoHnz5gbbl54FE0Jg06ZNGDRoEIQQBvn07dsXubm5Fe43Edm2li1bYsyYMVi5ciWuXLlS4TqldSomJsZg/ltvvQUA+OGHH8yaU4MGDXDixAlkZmZWus6DNTg/Px83b95E165dAaDCWhQVFWUw/cYbbwBAtTX46aefRsOGDQ1qXu/evVFcXFzt95nu3buHjRs3YuTIkfpa+swzz8DT09OgBh85cgRqtRrTpk1DgwYNDMYo3e7WrVv45ZdfMGLECNy5c0efy99//42+ffsiMzOzwjsiEdUWbPBJEg9+nPvOO+/AxcUFXbp0QevWrREVFYW9e/carH/37l289957+usyGzVqhMaNGyMnJwe5ubnlxm/durXBtIuLC7y9vav8slVmZiZOnDiBxo0bGzzatGkDALh+/XqV+3Ty5EkcOXIEPXr0QFZWlv4RFhaGbdu2QaPRAAD++usvAKjytng3btxATk4OVq5cWS6f8ePHG5UPEdmmf/7zn7h3716l1+JnZ2fDzs4O/v7+BvO9vLzQoEEDZGdnP3QOD9bgf/3rX8jJyUGbNm3Qrl07TJ8+HceOHTNY/9atW4iOjkaTJk3g6OiIxo0bQ6VSAYBRNbhVq1aws7Ortgbv2LGjXM3r3bs3gOpr3s8//4wbN26gS5cu+vqrVqsRHh6ODRs26K/jP3v2LICqa3BWVhaEEIiPjy+Xz6xZs4zKh8iW8Rp8MpmDgwOA+015RXQ6nX4dAAgMDMTp06exbds27NixA5s2bcInn3yC9957D4mJiQDun/1ZvXo1pk2bhm7dusHNzQ0KhQIvvviiSV++qkpJSQnatWuHefPmVbjc19e3yu2/+uorAMCbb76JN998s9zyTZs26ZtzY3IBgJdffhmRkZEVrtO+fXujxiIi29KyZUu8/PLLWLlyJWbOnFnpepV9X6g6Dg4OKCgogBCi3BhCCOTn5xvU4J49e+Ls2bPYsmULfv75Z3z66aeYP38+li9fjokTJwK4f039b7/9hunTp6NDhw5wcXFBSUkJ+vXrZ1QNNmZfSkpK8Oyzz2LGjBkVLi892VKZ0rP0I0aMqHB5amoqwsPDq82jNBcAePvtt9G3b98K1yn7HzCi2oQNPpnMz88PAHD69OlyTbFOp8OFCxfQp08fg/nOzs4YOXIkRo4cicLCQjz33HOYPXs2YmNj4eDggG+//RaRkZH4+OOP9dvk5+cjJyenwhwyMzMNCrlWq8WVK1fQv3//SvNu1aoVjh49il69epl8YBVCYP369QgPD8frr79ebnlSUhLWrVuH8ePHo2XLlgCAP/74o9LxGjdujPr166O4uFh/9oqI5OOf//wnvvrqK8ydO7fcMj8/P5SUlCAzMxOBgYH6+deuXUNOTo6+xlbGz88P9+7dw9mzZ8s1oVlZWSguLi43hru7O8aPH4/x48dDq9WiZ8+eSEhIwMSJE3H79m3s2rULiYmJeO+99/TbVHVJT2Zmpv4Mf2nckpIStGjRotJtWrVqBa1WW6Oal5eXhy1btmDkyJF44YUXyi2fOnUq1q1bh/DwcLRq1QrA/RpcWazSOl23bl3WYJIlXqJDJuvVqxfq1auHZcuWlTuzs3LlSty7dw8RERH6eWVvs1avXj08/vjjEEKgqKgIAGBvb1/uD0YtXrwYxcXFFeawcuVK/bYAsGzZsnJxyxoxYgQuXbqEVatWlVt29+5d5OXlVbrt3r17ce7cOYwfPx4vvPBCucfIkSOxe/duXL58GY0bN0bPnj3x+eef4/z58wbjlO6jvb09nn/+eWzatKnC/wjcuHGj0lyIyPa1atUKL7/8MlasWIGrV68aLCs9EVH2rjGlny4OGDCgyrFL69ySJUvKLVu6dKnBOkD5Guzi4gJ/f3/9LTnt7e0BoFwNruquNqVxSi1evLhc3LJGjBiBffv24aeffiq3LCcnB/fu3at02+TkZOTl5SEqKqrCGjxw4EBs2rQJBQUFeOKJJ6BSqbBgwYJyJ4lK99HT0xNhYWFYsWJFhd+VYA2m2o5n8Mlknp6eeO+99/DPf/4TPXv2xODBg+Hk5ITffvsNGzZsQJ8+fTBo0CD9+n369IGXlxd69OiBJk2a4OTJk1iyZInBl1UHDhyIL7/8Em5ubnj88cexb98+/O9//4OHh0eFORQWFqJXr14YMWIETp8+jU8++QRPPfUUBg8eXGneY8aMwTfffIPJkydj9+7d6NGjB4qLi3Hq1Cl88803+Omnn9CpU6cKt123bh3s7e0rPfAOHjwYcXFx+PrrrxETE4NFixbhqaeewhNPPIFJkyZBpVLh3Llz+OGHH5CRkQEA+OCDD7B79248+eSTePXVV/H444/j1q1bOHz4MP73v//h1q1bxrwcRGSj4uLi8OWXX+L06dMICgrSzw8ODkZkZCRWrlyJnJwchIaG4vfff8fatWsxdOjQai8z6dChAyZOnIiFCxciMzPT4FaXP/74IyZOnIjg4GD9+o8//jjCwsLQsWNHuLu7Iz09Hd9++y2mTJkCAHB1dUXPnj3x4YcfoqioCE2bNsXPP/8MtVpdaQ5qtRqDBw9Gv379sG/fPnz11Vd46aWXDOKWNX36dGzduhUDBw7EuHHj0LFjR+Tl5eH48eP49ttvce7cOTRq1KjCbdetWwcPDw907969wuWDBw/GqlWr8MMPP+C5557DsmXLMGjQIHTo0AHjx4+Ht7c3Tp06hRMnTuj/g7F06VI89dRTaNeuHV599VW0bNkS165dw759+3Dx4kUcPXq0yteByKZZ6e49JANfffWV6Nq1q3B2dhZKpVIEBASIxMREg1uOCSHEihUrRM+ePYWHh4dQKpWiVatWYvr06SI3N1e/zu3bt8X48eNFo0aNhIuLi+jbt684depUudusld6OLjU1VUyaNEk0bNhQuLi4iNGjR4u///7bIG7Z22QKIURhYaGYO3euCAoKEkqlUjRs2FB07NhRJCYmGuRTdhsPDw/x9NNPV/l8qFQqERISop/+448/xLBhw0SDBg2Eg4ODeOyxx0R8fLzBNteuXRNRUVHC19dX1K1bV3h5eYlevXqJlStXVhmLiGzHg7fJLCsyMlIAKHer4KKiIpGYmChUKpWoW7eu8PX1FbGxseXqZ2WKi4vFwoULRXBwsHBwcBAODg4iODhYLFq0SBQXFxus++9//1t06dJFNGjQQDg6OoqAgAAxe/Zsg1sNX7x4UV+v3NzcxPDhw8Xly5cFADFr1iz9eqW3yfzzzz/FCy+8IOrXry8aNmwopkyZIu7evWsQt2z9FkKIO3fuiNjYWOHv7y/q1asnGjVqJLp37y4++ugjg3wedO3aNVGnTh0xZsyYSp8PnU4nnJycxLBhw/Tz9uzZI5599llRv3594ezsLNq3by8WL15ssN3Zs2fF2LFjhZeXl6hbt65o2rSpGDhwoPj2228rjUVUGyiEKPOZHBERERER1Vq8Bp+IiIiISEbY4BMRERERyQgbfCKiWqKwsBD+/v747bffrJ2KUW7evAlPT09cvHjR2qkQEQF4dOooG3wiIitYunQpWrRoAQcHBzz55JP4/fffq91m+fLlUKlUBncSSU1NxTPPPAN3d3c4OTmhdevWiIyMRGFhIcaNGweFQlHpo/Se5WFhYZg2bZpBrKysLIwfPx7NmjWDUqmESqXCqFGjkJ6erl9HoVBg8+bN+umioiKMGjUKTZs2xR9//IFGjRph7Nix+r8MSkRkTqyjlWODT0RkYRs3bkRMTAxmzZqFw4cPIzg4GH379sX169cr3UYIgSVLluCVV17Rz/vzzz/Rr18/dOrUCWlpaTh+/DgWL16MevXqobi4GAsXLsSVK1f0DwBYvXq1fvrgwYMVxkpPT0fHjh1x5swZrFixAn/++SeSk5MREBCAt956q8JtdDodBg8ejIMHD2LPnj1o27YtAGD8+PFYt24db/tKRGbFOloN697Eh4jo0dOlSxcRFRWlny4uLhY+Pj5izpw5lW5z8OBBYWdnJzQajX7e/PnzRYsWLYyOC0AkJyeXmx8aGiqio6OFEEKUlJSIoKAg0bFjx3K3WxTi/i1ty453+/Zt0b17d9G+fXtx5cqVctuoVCrx6aefGp0nEVF1WEerJvs/dFVSUoLLly+jfv36UCgU1k6HiKxECIE7d+7Ax8cHdnZVf3iZn5+PwsJCk8cvW2OUSiWUSqXBvMLCQhw6dAixsbH6eXZ2dujduzf27dtX6fi//vor2rRpo//jcADg5eWFK1euIC0tDT179jQp38pkZGTgxIkTWL9+vf55erCO2tnZQaPR6NdXq9X45z//CWdnZ3z//fdwcnIyWA4AISEh2LVrF4YPH26WHInIOlhHjVNRHX1QgwYNDKavXr2K0NBQuLi4IDU1tdxyAOjSpQt+/fVXg08fqiL7Bv/y5cvw9fW1dhpEZCMuXLiAZs2aVbo8Pz8fPs2b4faNv00a18XFBVqt1mDerFmzkJCQYDDv5s2bKC4uRpMmTQzmN2nSBKdOnap0/OzsbPj4+BjMGz58OH766SeEhobCy8sLXbt2Ra9evTB27Fi4urqalH+pzMxMAEBAQIB+XlV1NCYmRv9vPz+/KsfesGFDjXIiItvCOlq1iupoVaKjo9GyZUvs3LkTTk5OFa7j4+ODI0eOGJ2D7Bv80v+lzVq7EI936iBZnL/+PI3Y4ZMw578r0fLxxxjHhuLIaV8Yp+Z02jyMf7K/wZmbihQWFuL2jb+x+sCPcHJxNmnsCxcuGBwQyp51ehh3796Fg4ODwTx7e3usXr0a//73v/HLL7/gwIEDeP/99zF37lz8/vvv8Pb2NjmOqOBvH7KOMo7c4shpXywZh3XUOBXV0aoMHDgQmzdvxooVK/Dmm29WuI6joyN0Op3RY8q+wS/9qMfByRFO9V0ki+Pw///H5eDkxDg2FkdO+8I4D8/YS/WcXJxNzsfV1bXaMz6NGjWCvb09rl27ZjD/2rVr8PLyqnK748ePV7isadOmGDNmDMaMGYOkpCS0adMGy5cvR2Jiokn5A0CbNm0AAKdOnUJISAgA1lHGkV8cOe2LJeOUYh2tWkV1tCpjxozB4MGDMWHCBAghDD4ZLXXr1i00btzY6Bx4Fx0iIguqV68eOnbsiF27dunnlZSUYNeuXejWrVul24WEhODUqVPVnhlq2LAhvL29kZeXV6P8OnTogMcffxwff/wxSkpKajQGEZGUansdzcnJKTcvMjISa9aswYwZM/DRRx+VW/7HH38Y9Z+FUrI/g09EZGtiYmIQGRmJTp06oUuXLliwYAHy8vIwfvz4SrcJDw+HVqvFiRMn9LdOW7FiBTIyMjBs2DC0atUK+fn5+OKLL3DixAksXry4RrkpFAqsXr0avXv3xtNPP424uDg0bdq0RmMREUmlttXRgIAAaLVafP/99/j555+RmppabrsxY8bAzs4OkZGREEJg+vTpAO7fPvPQoUN4//33jc6BDT4RkYWNHDkSN27cwHvvvYerV6+iQ4cO2LFjR7kvjD3Iw8MDw4YNw7p16zBnzhwA9++qsGfPHkyePBmXL1+Gi4sLgoKCsHnzZoSGhtY4vy5duiA9PR2zZ8/Gq6++ihs3btR4LCIiKdS2Onrz5k14e3uje/fuWLBgQaXbjR49GnZ2dhgzZgxKSkrwzjvvYMuWLWjevDmefvppo+OzwScisoIpU6ZgypQpJm0TFxeHZ599FnFxcXBxcUFISAi+/PJLo7ev7GPplJSUcvPatGmDtWvXAgA0Gg3c3NxMypWISGq1qY6aMt6oUaMwatQo/fTChQvx3nvvGZ0jwGvwiYhqjfbt22Pu3LlQq9XWToWIqFaqbXX05s2beO655wwafmPwDD4RUS0ybtw4a6dARFSr1aY62qhRI8yYMcPk7XgGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGTEqg3+nDlz0LlzZ9SvXx+enp4YOnQoTp8+bbDOypUrERYWBldXVygUCuTk5FgnWSIiG8Q6SkREZVm1wU9NTUVUVBT279+PnTt3oqioCH369EFeXp5+HZ1Oh379+uHdd9+1YqZERLaJdZSIiMqqY83gO3bsMJhes2YNPD09cejQIfTs2RMAMG3aNABASkqKhbMjIrJ9rKNERFSWVRv8snJzcwEA7u7uNR6joKAABQUF+mmNRgMAuPhXNhycnB4uwSpcyFIb/GQc24kjp31hnJrL1+kkHd9WsI4yDuPYRgw5xnlU6qgcKIQQwtpJAEBJSQkGDx6MnJwc7Nmzp9zylJQUhIeH4/bt22jQoEGl4yQkJCAxMVHCTImoNsvNzYWrq2ulyzUaDdzc3LDxRCqc6rsYNabujhYjg0KrHVtqrKNEZAlyrqNyYTNn8KOiovDHH39UeFAyRWxsLGJiYvTTGo0Gvr6+iJobB/+2gQ+bZqUuZKkxLzoeMQuT4OuvYhwbiiOnfWGcmsvX6RA7fJJk49sC1lHGkTKO9rAa8fHxSEpKgkolXZzDf/O4YKtxHoU6Khc20eBPmTIF27ZtQ1paGpo1a/ZQYymVSiiVynLzm7X0g3876Q5MpXz9VYxjo3HktC+MYzrdHa1kY9sC1lHGkTpO7u37P1UqFQICAiSLc+Pa/Z88LtheHLnXUTmxaoMvhMAbb7yB5ORkpKSkSHpGgIhIjlhHiYioLKs2+FFRUVi/fj22bNmC+vXr4+rVqwAANzc3ODo6AgCuXr2Kq1evIisrCwBw/Phx1K9fH82bN3+oL5EREckB6ygREZVl1fvgL1u2DLm5uQgLC4O3t7f+sXHjRv06y5cvR0hICF599VUAQM+ePRESEoKtW7daK20iIpvBOkpERGVZ/RKd6iQkJCAhIUH6ZIiIaiHWUSIiKsuqZ/CJiIiIiMi82OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSEas2+HPmzEHnzp1Rv359eHp6YujQoTh9+rTBOvn5+YiKioKHhwdcXFzw/PPP49q1a1bKmIjItrCOEhFRWVZt8FNTUxEVFYX9+/dj586dKCoqQp8+fZCXl6df580338T333+P//73v0hNTcXly5fx3HPPWTFrIiLbwTpKRERl1bFm8B07dhhMr1mzBp6enjh06BB69uyJ3NxcfPbZZ1i/fj2eeeYZAMDq1asRGBiI/fv3o2vXrtZIm4jIZrCOEhFRWVZt8MvKzc0FALi7uwMADh06hKKiIvTu3Vu/TkBAAJo3b459+/ZVeGAqKChAQUGBflqj0QAALv6VDQcnJ8lyv5ClNvjJOLYTR077wjg1l6/TSTq+rWAdZRypaNX3x1erJd6fv3lcsNU4j0odlQOFEEJYOwkAKCkpweDBg5GTk4M9e/YAANavX4/x48cbHGgAoEuXLggPD8fcuXPLjZOQkIDExESL5ExEtU9ubi5cXV0rXa7RaODm5oaNJ1LhVN/FqDF1d7QYGRRa7dhSYx0lIkuQcx2VC5s5gx8VFYU//vhDf1CqqdjYWMTExOinNRoNfH19ERcXh8DAwIdNs1JqtRrx8fGIWZgEX3+VZHEuZKkxL1p+cZKSkqBSSROHrw3jAPfPPMUOnyTZ+LZA6joaNTcO/m2lq6Nye88xju3GkdO+WDLOo1BH5cImGvwpU6Zg27ZtSEtLQ7NmzfTzvby8UFhYiJycHDRo0EA//9q1a/Dy8qpwLKVSCaVSWW6+n58fAgICzJ57Wb7+Kvi3k+4AKNc4KpVK8tdHbs8Z45hGd0cr2di2wBJ1tFlLP1m8FxiHcSwZQ05x5F5H5cSqd9ERQmDKlClITk7GL7/8Uu4MbseOHVG3bl3s2rVLP+/06dM4f/48unXrZul0iYhsDusoERGVZdUz+FFRUVi/fj22bNmC+vXr4+rVqwAANzc3ODo6ws3NDa+88gpiYmLg7u4OV1dXvPHGG+jWrRvv/EBEBNZRIiIqz6oN/rJlywAAYWFhBvNXr16NcePGAQDmz58POzs7PP/88ygoKEDfvn3xySefWDhTIiLbxDpKRERlWbXBN+YGPg4ODli6dCmWLl1qgYyIiGoX1lEiIirLqtfgExERERGRebHBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhmxaoOflpaGQYMGwcfHBwqFAps3bzZYfu3aNYwbNw4+Pj5wcnJCv379kJmZaZ1kiYhsEOsoERGVZdUGPy8vD8HBwVi6dGm5ZUIIDB06FH/99Re2bNmCI0eOwM/PD71790ZeXp4VsiUisj2so0REVFYdawaPiIhAREREhcsyMzOxf/9+/PHHHwgKCgIALFu2DF5eXtiwYQMmTpxoyVSJiGwS6ygREZVls9fgFxQUAAAcHBz08+zs7KBUKrFnzx5rpUVEVGuwjhIRPZqsega/KgEBAWjevDliY2OxYsUKODs7Y/78+bh48SKuXLlS6XYFBQX6gxoAaDQaAEB2djacnJwky1etVgMALmSpJYvx4Phyi1P6/EmBrw3jAEC+Tifp+LbI3HX04l/ZcJCwjsrtPcc4thtHTvtiyTiPYh2trRRCCGHtJABAoVAgOTkZQ4cO1c87dOgQXnnlFRw9ehT29vbo3bs37OzsIITA9u3bKxwnISEBiYmJFsqaiGqb3NxcuLq6Vrpco9HAzc0NG0+kwqm+i1Fj6u5oMTIotNqxpcY6SkSWIOc6Khc2ewYfADp27IiMjAzk5uaisLAQjRs3xpNPPolOnTpVuk1sbCxiYmL00xqNBr6+voiaGwf/toGS5XohS4150fFISkqCSqWSLI5arUZ8fDxiFibB11+6OKX7Y6k4Uj5vpc8ZX5tHO06+TofY4ZMkG99WmbOOxsXFITBQujrK3yHGsVQcOe3Lg3GkPs7pdDpMmvTo1dHayKYb/FJubm4A7n9hLD09HUlJSZWuq1QqoVQqy81v1tIP/u2kOzCVUqlUCAgIkDyOr7/KIvtjqTiWeN742jzacXR3tJKNXRuYo476+fnxd4hxZBVHTvsCSH+c02of7Tpam1i1wddqtcjKytJPq9VqZGRkwN3dHc2bN8d///tfNG7cGM2bN8fx48cRHR2NoUOHok+fPlbMmojIdrCOEhFRWVZt8NPT0xEeHq6fLv1IODIyEmvWrMGVK1cQExODa9euwdvbG2PHjkV8fLy10iUisjmso0REVJZVG/ywsDBU9R3fqVOnYurUqRbMiIiodmEdJSKismz2PvhERERERGQ6NvhERERERDLCBp+IiIiISEbY4BMRERERyQgbfCIiIiIiGWGDT0REREQkI2zwiYiIiIhkhA0+EREREZGMsMEnIiIiIpIRNvhERERERDLCBp+IiIiISEbY4BMRERERyQgbfCIiIiIiGWGDT0REREQkI2zwiYiIiIhkxKoNflpaGgYNGgQfHx8oFAps3rzZYLlWq8WUKVPQrFkzODo64vHHH8fy5cutkywRkQ1iHSUiorKs2uDn5eUhODgYS5curXB5TEwMduzYga+++gonT57EtGnTMGXKFGzdutXCmRIR2SbWUSIiKquONYNHREQgIiKi0uW//fYbIiMjERYWBgCYNGkSVqxYgd9//x2DBw+2UJZERLaLdZSIiMoy6gy+RqMx+mFO3bt3x9atW3Hp0iUIIbB7926cOXMGffr0MWscIiKpsY4SEZGlGHUGv0GDBlAoFEYNWFxc/FAJPWjx4sWYNGkSmjVrhjp16sDOzg6rVq1Cz549K92moKAABQUF+unSg+XFv7Lh4ORkttzKupClBgCo1WrJYjw4fmk8qZSOb6k4Uj5vpWPztXm04+TrdJKOXx051NHs7Gw4SVhH+TvEOJaKI6d9eXB8qY9zOivXUTKeQgghqlspNTVV/+9z585h5syZGDduHLp16wYA2LdvH9auXYs5c+YgMjKyZokoFEhOTsbQoUP18z766COsWrUKH330Efz8/JCWlobY2FgkJyejd+/eFY6TkJCAxMTEGuVARPKXm5sLV1fXSpdrNBq4ublh44lUONV3MWpM3R0tRgaFVjk26ygRyYW16igZz6gG/0G9evXCxIkTMWrUKIP569evx8qVK5GSklKzRMocmO7evQs3NzckJydjwIAB+vUmTpyIixcvYseOHRWOU9GZJ19fX0TNjYN/28Aa5WaMC1lqzIuOR8zCJPj6qySPk5SUBJVKujhqtRrx8fKIUxrDUq+N3N4Dcnne8nU6xA6fZBMHJtbRiln6d4hxHt04ctoXS8axpTpKVTP5S7b79u2r8BZrnTp1wsSJE82SFAAUFRWhqKgIdnaGXxOwt7dHSUlJpdsplUoolcpy85u19IN/O+kOTKV8/VUWiaNSqRAQEMA4JrDUayO394BcnjfdHa1kY5uKdbRqcnnPMY7tx5HTvlgiji3VUaqaybfJ9PX1xapVq8rN//TTT+Hr62vSWFqtFhkZGcjIyABw/4xhRkYGzp8/D1dXV4SGhmL69OlISUmBWq3GmjVr8MUXX2DYsGGmpk1EZDNYR4mISEomn8GfP38+nn/+eWzfvh1PPvkkAOD3339HZmYmNm3aZNJY6enpCA8P10/HxMQAACIjI7FmzRp8/fXXiI2NxejRo3Hr1i34+flh9uzZmDx5sqlpExHZDNZRIiKSkskNfv/+/ZGZmYlPPvkEp06dAgAMGjQIkydPNvnMU1hYGKr6CoCXlxdWr15taopERDaNdZSIiKRUoz901axZM7z//vvmzoWI6JHBOkpERFKpUYOfk5OD33//HdevXy/3Ra2xY8eaJTEiIjljHSUiIqmY3OB///33GD16NLRaLVxdXQ3+cItCoeCBiYioGqyjREQkJZPvovPWW29hwoQJ0Gq1yMnJwe3bt/WPW7duSZEjEZGssI4SEZGUTG7wL126hKlTp0r658qJiOSMdZSIiKRkcoPft29fpKenS5ELEdEjgXWUiIikZPI1+AMGDMD06dPx559/ol27dqhbt67B8sGDB5stOSIiOWIdJSIiKZnc4L/66qsAgH/961/llikUChQXFz98VkREMsY6SkREUjK5wS97OzciIjIN6ygREUnJ5GvwiYiIiIjIdtXoD13l5eUhNTUV58+fR2FhocGyqVOnmiUxIiI5Yx0lIiKpmNzgHzlyBP3794dOp0NeXh7c3d1x8+ZNODk5wdPTkwcmIqJqsI4SEZGUTL5E580338SgQYNw+/ZtODo6Yv/+/cjOzkbHjh3x0UcfSZEjEZGssI4SEZGUTG7wMzIy8NZbb8HOzg729vYoKCiAr68vPvzwQ7z77rtS5EhEJCuso0REJCWTG/y6devCzu7+Zp6enjh//jwAwM3NDRcuXDBvdkREMsQ6SkREUjK5wQ8JCcHBgwcBAKGhoXjvvfewbt06TJs2DW3btjVprLS0NAwaNAg+Pj5QKBTYvHmzwXKFQlHh4z//+Y+paRMR2QzWUSIikpLJDf77778Pb29vAMDs2bPRsGFDvPbaa7hx4wZWrlxp0lh5eXkIDg7G0qVLK1x+5coVg8fnn38OhUKB559/3tS0iYhsBusoERFJyeS76HTq1En/b09PT+zYsaPGwSMiIhAREVHpci8vL4PpLVu2IDw8HC1btqxxTCIia2MdJSIiKdXoPvjWcO3aNfzwww9Yu3ZtlesVFBSgoKBAP63RaAAAF//KhoOTk2T5XchSG/yUOo5aLW2c0vHlEKd0bEu9NnJ7D8jlecvX6SQdvzZgHWUcxrFcDDnGYR2tPRRCCGHMiiEhIVAoFNWud/jw4ZololAgOTkZQ4cOrXD5hx9+iA8++ACXL1+Gg4NDpeMkJCQgMTGxRjkQkfzl5ubC1dW10uUajQZubm7YeCIVTvVdjBpTd0eLkUGh1Y7NOkpEcmDNOkrGMfoM/oMHDCEE5syZg8mTJ8Pd3V2KvMr5/PPPMXr06CoPSgAQGxuLmJgY/bRGo4Gvry/i4uIQGBgoWX5qtRrx8fGIWZgEX3+VZHEuZKkxLzoeSUlJUKmki1O6P5aKI+XzVvqcye214f6YJl+nQ+zwSZKNb4zaXkej5sbBv610ddTSv6ty+x2SOs7hvy3zvGkPS78/lq6jcoljC3WUjGN0gz9r1iyD6Y8//hjR0dEWuY7z119/xenTp7Fx48Zq11UqlVAqleXm+/n5ISAgQIr0DPj6q+DfTroDYCmVSmWR/bFUHEs8b3J7bbg/ptHd0Uo2trFqex1t1tLPIu85S7235fY7JHWcG9fu/5T6ecu9ff+nJZ43ub3XHoU6SsYx+S461vDZZ5+hY8eOCA4OtnYqRES1EusoEdGjw6pfstVqtcjKytJPq9VqZGRkwN3dHc2bNwdw/6Ph//73v/j444+tlSYRkc1iHSUiorKs2uCnp6cjPDxcP116zWdkZCTWrFkDAPj6668hhMCoUaOskSIRkU1jHSUiorKMbvAXLVpkMH3v3j2sWbMGjRo1Mpg/depUo4OHhYWhupv4TJo0CZMm8QsdRFT7sY4SEZElGN3gz58/32Day8sLX375pcE8hUJh0oGJiOhRwjpKRESWYHSDL/Uf1SEikjvWUSIisoRacRcdIiIiIiIyDht8IiIiIiIZYYNPRERERCQjbPCJiIiIiGTEpPvg37t3D+vXr0ffvn3RpEkTqXIiIrK6JjcEXO5WffvJUlqtcesBrKNE9OiQqo5S9Uw6g1+nTh1MnjwZ+fn5UuVDRCRrrKNERCQ1ky/R6dKlCzIyMiRIhYjo0cA6SkREUjLpEh0AeP311xETE4MLFy6gY8eOcHZ2Nljevn17syVHRCRHrKNERCQlkxv8F198EYDhn1JXKBQQQkChUKC4uNh82RERyRDrKBERScnkBp9/iZGI6OGwjhIRkZRMbvD9/PykyIOI6JHBOkpERFKq0X3wv/zyS/To0QM+Pj7Izs4GACxYsABbtmwxa3JERHLFOkpERFIxucFftmwZYmJi0L9/f+Tk5OivFW3QoAEWLFhg7vyIiGSHdZSIiKRkcoO/ePFirFq1CnFxcbC3t9fP79SpE44fP27SWGlpaRg0aBB8fHygUCiwefPmcuucPHkSgwcPhpubG5ydndG5c2ecP3/e1LSJiGwG6ygREUnJ5AZfrVYjJCSk3HylUom8vDyTxsrLy0NwcDCWLl1a4fKzZ8/iqaeeQkBAAFJSUnDs2DHEx8fDwcHB1LSJiGwG6ygREUnJ5C/ZqlQqZGRklPuS2I4dOxAYGGjSWBEREYiIiKh0eVxcHPr3748PP/xQP69Vq1amJUxEZGNYR4mISEomN/gxMTGIiopCfn4+hBD4/fffsWHDBsyZMweffvqp2RIrKSnBDz/8gBkzZqBv3744cuQIVCoVYmNjMXTo0Eq3KygoQEFBgX5ao9EAALKzs+Hk5GS2/Moqve3dhSxpb39XOr7Ut9krHd9ScaR83krHlttrw/0xTb5OJ+n4pqitdfTiX9lwkLCOWvp3VW6/Q1LHufC3ZZ43rQX2x9J1VC5xbKmOUtUUQghh6kbr1q1DQkICzp49CwDw8fFBYmIiXnnllZonolAgOTlZf9C5evUqvL294eTkhH//+98IDw/Hjh078O6772L37t0IDQ2tcJyEhAQkJibWOA8ikrfc3Fy4urpWulyj0cDNzQ0pKSlwcXExakytVouwsLBqx34Q6ygR1Va2UkepcjVq8EvpdDpotVp4eno+fCJlDkyXL19G06ZNMWrUKKxfv16/3uDBg+Hs7IwNGzZUOE5FZ558fX0RNTcO/m1N++jbFBey1JgXHY+YhUnw9VdJHicpKQkqlXRx1Go14uMttz9SxrH0a8M4NYsj9Xtap9Nh0qRJNndgYh39P5aub6yjj24cOe3Lg3Ee1TpK5Zl8ic6sWbMwYcIE+Pn5wcnJSbLLXho1aoQ6derg8ccfN5gfGBiIPXv2VLqdUqmEUqksN79ZSz/4t5PuwFTK119lkTgqlQoBAQGSx7HU/lgijpz2RY5xpH5Pa7VaycY2Feto1SxV31hHGUdO+wI8WnWUqmbyXXS2bNmCVq1aoVevXli/fr3BWR5zqlevHjp37ozTp08bzD9z5gz/CiQR1Wqso0REJCWTG/yMjAwcPHgQQUFBiI6OhpeXF1577TUcPHjQ5OBarRYZGRnIyMgAcP/jzIyMDP39madPn46NGzdi1apVyMrKwpIlS/D999/j9ddfNzkWEZGtYB0lIiIpmdzgA0BISAgWLVqEy5cv47PPPsPFixfRo0cPtG/fHgsXLkRubq5R46SnpyMkJER/P+iYmBiEhITgvffeAwAMGzYMy5cvx4cffoh27drh008/xaZNm/DUU0/VJG0iIpvBOkpERFKpUYNfSgiBoqIiFBYWQgiBhg0bYsmSJfD19cXGjRur3T4sLAxCiHKPNWvW6NeZMGECMjMzcffuXWRkZGDIkCEPkzIRkU1hHSUiInOrUYN/6NAhTJkyBd7e3njzzTcREhKCkydPIjU1FZmZmZg9ezamTp1q7lyJiGSDdZSIiKRicoPfrl07dO3aFWq1Gp999hkuXLiADz74AP7+/vp1Ro0ahRs3bpg1USIiuWAdJSIiKZl8m8wRI0ZgwoQJaNq0aaXrNGrUCCUlJQ+VGBGRXLGOEhGRlExu8OPj46XIg4jokcE6SkREUjK5wQeAixcvYuvWrTh//jwKCwsNls2bN88siRERyRnrKBERScXkBn/Xrl0YPHgwWrZsiVOnTqFt27Y4d+4chBB44oknpMiRiEhWWEeJiEhKJn/JNjY2Fm+//TaOHz8OBwcHbNq0CRcuXEBoaCiGDx8uRY5ERLLCOkpERFIyucE/efIkxo4dCwCoU6cO7t69CxcXF/zrX//C3LlzzZ4gEZHcsI4SEZGUTG7wnZ2d9deLent74+zZs/plN2/eNF9mREQyxTpKRERSMvka/K5du2LPnj0IDAxE//798dZbb+H48eP47rvv0LVrVylyJCKSFdZRIiKSkskN/rx586DVagEAiYmJ0Gq12LhxI1q3bs07PxARGYF1lIiIpGRyg9+yZUv9v52dnbF8+XKzJkREJHeso0REJCWTr8EnIiIiIiLbZdQZ/IYNG0KhUBg14K1btx4qISIiOWIdJSIiSzGqwV+wYIHEaRARyRvrKBERWYpRDX5kZKQkwdPS0vCf//wHhw4dwpUrV5CcnIyhQ4fql48bNw5r16412KZv377YsWOHJPkQEUmFdZSIiCzF6GvwS0pKMHfuXPTo0QOdO3fGzJkzcffu3YcKnpeXh+DgYCxdurTSdfr164crV67oHxs2bHiomERE1sI6SkRElmD0XXRmz56NhIQE9O7dG46Ojli4cCGuX7+Ozz//vMbBIyIiEBERUeU6SqUSXl5eNY5BRGQrWEeJiMgSjG7wv/jiC3zyySf4xz/+AQD43//+hwEDBuDTTz+FnZ10N+NJSUmBp6cnGjZsiGeeeQb//ve/4eHhUen6BQUFKCgo0E9rNBoAwMW/suHg5CRZnhey1AY/pY6jVksbp3R8S+2PlHEs/dowTs3iSP2e1ul0ko5vDNbRqlm6vrGOPrpx5LQvD47/KNRRMo5CCCGMWVGpVCIrKwu+vr76eQ4ODsjKykKzZs0ePhGFoty1o19//TWcnJygUqlw9uxZvPvuu3BxccG+fftgb29f4TgJCQlITEx86HyISJ5yc3Ph6upa6XKNRgM3NzekpKTAxcXFqDG1Wi3CwsKqHZt1lIjkwJp1lIxj9Bn8e/fuwcHBwWBe3bp1UVRUZPakSr344ov6f7dr1w7t27dHq1atkJKSgl69elW4TWxsLGJiYvTTGo0Gvr6+iJobB/+2gZLleiFLjXnR8UhKSoJKpZIsjlqtRnw849QkRszCJPj6S7cvln4PWGp/5BInX6dD7PBJko1vDNbRqsntPcc4thtHTvtiyTi2UEfJOEY3+EIIjBs3DkqlUj8vPz8fkydPhrOzs37ed999Z94MH9CyZUs0atQIWVlZlR6YlEqlQY6lmrX0g3876Q5MpVQqFQICAhjHBuP4+qtk9R6w1P7IJY7ujlaysY3FOmocubznGMf248hpXywRxxbqKBnH6Aa/olu8vfzyy2ZNpjoXL17E33//DW9vb4vGJSIyB9ZRIiKyBKMb/NWrV5s9uFarRVZWln5arVYjIyMD7u7ucHd3R2JiIp5//nl4eXnh7NmzmDFjBvz9/dG3b1+z50JEJDXWUSIisgSjG3wppKenIzw8XD9des1nZGQkli1bhmPHjmHt2rXIycmBj48P+vTpg6SkpAo/OiYiehSxjhIRUVlWbfDDwsJQ1U18fvrpJwtmQ0RU+7COEhFRWdLdeJmIiIiIiCyODT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZsWqDn5aWhkGDBsHHxwcKhQKbN2+udN3JkydDoVBgwYIFFsuPiMjWsY4SEVFZVm3w8/LyEBwcjKVLl1a5XnJyMvbv3w8fHx8LZUZEVDuwjhIRUVl1rBk8IiICERERVa5z6dIlvPHGG/jpp58wYMAAC2VGRFQ7sI4SEVFZVm3wq1NSUoIxY8Zg+vTpCAoKMmqbgoICFBQU6Kc1Gg0A4OJf2XBwcpIkTwC4kKUGAKjVasliPDg+45geo/Q1koql3wOW2h+5xMnX6SQd31bVxjoql/cc49huHDntiyXjPKp1tDZSCCGEtZMAAIVCgeTkZAwdOlQ/b86cOdi9ezd++uknKBQKtGjRAtOmTcO0adMqHSchIQGJiYnSJ0xEtVJubi5cXV0rXa7RaODm5oaUlBS4uLgYNaZWq0VYWFi1Y0uNdZSILEHOdVQubPYM/qFDh7Bw4UIcPnwYCoXC6O1iY2MRExOjn9ZoNPD19UVcXBwCAwOlSBXA/bOq8fHxiFmYBF9/lWRxLmSpMS+acWoSIykpCSqVdPtS+h6QWxxLvQek3h+dTodJkyZJNr4tYh2tmKXrm9x+V6WOoz0sfY2T23PGOkpl2WyD/+uvv+L69eto3ry5fl5xcTHeeustLFiwAOfOnatwO6VSCaVSWW6+n58fAgICpEpXz9dfBf920h0AGafmVCqVRd4DcotjqfeA1Puj1WolG9tWsY7aRhy5/a5KHSf39v2flnje5PKclWIdpVI22+CPGTMGvXv3NpjXt29fjBkzBuPHj7dSVkREtQfrKBHRo8mqDb5Wq0VWVpZ+Wq1WIyMjA+7u7mjevDk8PDwM1q9bty68vLzw2GOPWTpVIiKbxDpKRERlWbXBT09PR3h4uH669JrPyMhIrFmzxkpZERHVHqyjRERUllUb/LCwMJhyE5/KrhclInpUsY4SEVFZVv1LtkREREREZF5s8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREcmIVRv8tLQ0DBo0CD4+PlAoFNi8ebPB8oSEBAQEBMDZ2RkNGzZE7969ceDAAeskS0Rkg1hHiYioLKs2+Hl5eQgODsbSpUsrXN6mTRssWbIEx48fx549e9CiRQv06dMHN27csHCmRES2iXWUiIjKqmPN4BEREYiIiKh0+UsvvWQwPW/ePHz22Wc4duwYevXqJXV6REQ2j3WUiIjKsmqDb4rCwkKsXLkSbm5uCA4OrnS9goICFBQU6Kc1Gg0AIDs7G05OTpLlp1arAQAXstSSxXhwfMYxPUbpaySV0vHlFsdS7wGp90en00k6fm3AOgqD8eXy3pbb86a1QI2T23PGOkplKYQQwtpJAIBCoUBycjKGDh1qMH/btm148cUXodPp4O3tjc2bN6Nz586VjpOQkIDExESJsyWi2io3Nxeurq6VLtdoNHBzc0NKSgpcXFyMGlOr1SIsLKzasaXGOkpEliDnOioXNn8GPzw8HBkZGbh58yZWrVqFESNG4MCBA/D09Kxw/djYWMTExOinNRoNfH19ETU3Dv5tAyXL80KWGvOi4xGzMAm+/irZxElKSoJKJV0ctVqN+Hhp90eur43c4kj9XtPpdJg0aZJk49uy2lZH5VB3APn+rsohjpz2xZJx8nU6xA5/NOtobWPzDb6zszP8/f3h7++Prl27onXr1vjss88QGxtb4fpKpRJKpbLc/GYt/eDfTroDUylff5Ws4qhUKgQEBEgexxL7I7fXRm5xpH6vabVayca2dbWtjsqp7jCObceR075YIo7uzqNbR2ubWncf/JKSEoNrQ4mIyDSso0RE8mbVM/harRZZWVn6abVajYyMDLi7u8PDwwOzZ8/G4MGD4e3tjZs3b2Lp0qW4dOkShg8fbsWsiYhsB+soERGVZdUGPz09HeHh4frp0ms+IyMjsXz5cpw6dQpr167FzZs34eHhgc6dO+PXX39FUFCQtVImIrIprKNERFSWVRv8sLAwVHUTn++++86C2RAR1T6so0REVFatuwafiIiIiIgqxwafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafiIiIiEhG2OATEREREckIG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjFi1wU9LS8OgQYPg4+MDhUKBzZs365cVFRXhnXfeQbt27eDs7AwfHx+MHTsWly9ftl7CREQ2hnWUiIjKsmqDn5eXh+DgYCxdurTcMp1Oh8OHDyM+Ph6HDx/Gd999h9OnT2Pw4MFWyJSIyDaxjhIRUVl1rBk8IiICERERFS5zc3PDzp07DeYtWbIEXbp0wfnz59G8eXNLpEhEZNNYR4mIqCyrNvimys3NhUKhQIMGDSpdp6CgAAUFBfppjUYDALj4VzYcnJwky+1Cltrgp1ziqNXSxikdX8r9ketrI7c4Ur/XdDqdpOPXFrWhjsqh7jw4PuPYXhw57Ysl4+SzjtYaCiGEsHYSAKBQKJCcnIyhQ4dWuDw/Px89evRAQEAA1q1bV+k4CQkJSExMlChLIqrtcnNz4erqWulyjUYDNzc3pKSkwMXFxagxtVotwsLCqh1baqyjRGQJcq6jclErzuAXFRVhxIgREEJg2bJlVa4bGxuLmJgY/bRGo4Gvry/i4uIQGBgoWY5qtRrx8fFISkqCSqViHBuKI6d9YZya0+l0mDRpkmTj2zpz1NGouXHwbytdHb2Qpca8aMu952IWJsHXX7o4pfsjlzjaw/KpCZZ+D8jlPZ2v0yF2+KNbR2sTm2/wSw9K2dnZ+OWXX6r9X51SqYRSqSw338/PDwEBAVKlqadSqRjHRuPIaV8Yx3RarVaysW2duepos5Z+8G8nXYNfylLvOV9/lUX2Ry5xcm/f/ymXmgBY7rWRy3tad+fRraO1jU03+KUHpczMTOzevRseHh7WTomIqFZhHSUievRYtcHXarXIysrST6vVamRkZMDd3R3e3t544YUXcPjwYWzbtg3FxcW4evUqAMDd3R316tWzVtpERDaDdZSIiMqyaoOfnp6O8PBw/XTpNZ+RkZFISEjA1q1bAQAdOnQw2G737t0ICwuzVJpERDaLdZSIiMqyaoMfFhaGqm7iYyM3+CEislmso0REVJZV/5ItERERERGZFxt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREREREMsIGn4iIiIhIRtjgExERERHJCBt8IiIiIiIZYYNPRERERCQjbPCJiIiIiGSEDT4RERERkYywwSciIiIikhE2+EREtUjPnj2xfv16yePcvHkTnp6euHjxouSxiIgs6VGoo1Zt8NPS0jBo0CD4+PhAoVBg8+bNBsu/++479OnTBx4eHlAoFMjIyLBKnkRE5lRd7avM1q1bce3aNbz44ov6eV5eXggKCio3VkJCAjp06KCvo05OTvo62qJFCygUikof48aNQ6NGjTB27FjMmjVLgmeAiOjhmLOOtmjRAgsWLCi3bmkdrWja1uuoVRv8vLw8BAcHY+nSpZUuf+qppzB37lwLZ0ZEJJ3qal9lFi1ahPHjx8PO7v9KtxACTZs2rbaO9u7dWz/v4MGDuHLlCq5cuYJNmzYBAE6fPq2ft3DhQgDA+PHjsW7dOty6dcvUXSQikpQ562hN2HodrWPRaGVEREQgIiKi0uVjxowBAJw7d85CGRERSa+62leRGzdu4JdfftEfNEo5Ojqif//+GDZsWIXbldbRadOm6ec1btxY/293d3cAgKenJxo0aGCwbeknA9u2bTMpVyIiqZmzjtaEqXU0OTkZr7zyykPHNZZVG3wpFBQUoKCgQD+dm5sLADhz5oykcbOzswEAJ0+ehE6nYxwbiiOnfWGcmrt79y6A+2e8jZGXl2f02KXrajQag/lKpRJKpdLocaqyZ88eODk5ITAw0CzjVeXBOhoSEoKUlBQAwF9/SltHL/5l2fdc1h8nkS9hnNL9kUsc7Un51ARLvwfk8p7O17GO1kSXLl3w66+/WrTBh7ARAERycnKFy9RqtQAgjhw5Uu04s2bNEgD44IMPPip8XLhwocoacvfuXeHl5WXyuC4uLuXmzZo166Fq34Pmz58vWrZsWW6+n5+fqFevnnB2dhYAhFKpFM7OzqJu3boiODhYv150dLQAytfR3bt3CwDi9u3b+nmso3zwwUdVDznX0QcfZevorFmzDKZLVVRHH/Tmm2+KsLCwavMzJ9mdwY+NjUVMTIx+OicnB35+fjh//jzc3Nwki6vRaODr64sLFy7A1dWVcWwojpz2hXFqTgiBO3fuwMfHp8r1HBwcoFarUVhYaPL4CoXCYJ65zjoB9z+BcHBwqHDZ9OnTMW7cOLRu3Rrz58/Hs88+i0WLFiEtLa1GsR6so//617+wc+dOHDt2jHWUcWQTR077Ysk4j0IdfdDD1NEHOTo6SvoJTkVk1+BX9lGOm5ubpG/6Uq6uroxjo3HktC+MUzPGNqcODg6VHgSspVGjRrh9+3aly/z9/QEA3t7e8Pf3118TWhMP1tG8vDx4eXnh2LFjrKOMI7s4ctoXS8V5FOpoqYepow+6deuWwTX7lsD74BMR1QIhISG4evVqpQcnqfzxxx8IDg62aEwiIilYs46GhIRYNKZVG3ytVouMjAz9/e3VajUyMjJw/vx5APf/x5ORkYE///wTwP3bD2VkZODq1avWSpmI6KFVV/sqEhISgkaNGmHv3r0G80tKSnDp0qVyY5XeYKC0jt64cQOAaXVUp9Ph0KFDeOaZZ2qwl0RE0jFnHZVSaR3t06ePxWICgFW/ZFv6pYSyj8jISCGEEKtXr65wuTFfuCiVn58vZs2aJfLz86XZCcax+Thy2hfGkYfqal9lZsyYIV588UWDeU2aNKlwrODgYBEcHFxtHa3qy2Hr168Xjz32mOzeC4zDOHLaF0vGsSXmrKN+fn5i/vz55dYt+6XamnzJtrSOWppCCCPvdURERFZ19epVBAUF4fDhw/Dz85M8XteuXTF16lS89NJLksciIrKER6WO8hp8IqJawsvLC5999lmVH0Gby82bN/Hcc89h1KhRksciIrKUR6WO8gw+EREREZGM8Aw+EREREZGMsMEnIiIiIpIRWTf4+/btg729PQYMGCBZjDlz5qBz586oX78+PD09MXToUJw+fdqsMZYtW4b27dvr/4BFt27dsH37drPGqMgHH3wAhUKBadOmmXXchIQEKBQKg0dAQIBZY5S6dOkSXn75ZXh4eMDR0RHt2rVDenq6WWO0aNGi3P4oFApERUWZNU5xcTHi4+OhUqng6OiIVq1aISkpCVJcZXfnzh1MmzYNfn5+cHR0RPfu3XHw4MGHGjMtLQ2DBg2Cj48PFAoFNm/ebLBcCIH33nsP3t7ecHR0RO/evZGZmflQMck8pK6llqijgHVqKeuocVhHjcM6SsaSdYP/2Wef4Y033kBaWhouX74sSYzU1FRERUVh//792LlzJ4qKitCnTx/k5eWZLUazZs3wwQcf4NChQ0hPT8czzzyDIUOG4MSJE2aLUdbBgwexYsUKtG/fXpLxg4KCcOXKFf1jz549Zo9x+/Zt9OjRA3Xr1sX27dvx559/4uOPP0bDhg3NGufgwYMG+7Jz504AwPDhw80aZ+7cuVi2bBmWLFmCkydPYu7cufjwww+xePFis8YBgIkTJ2Lnzp348ssvcfz4cfTp0we9e/fGpUuXajxmXl4egoODsXTp0gqXf/jhh1i0aBGWL1+OAwcOwNnZGX379kV+fn6NY5J5SF1LLVFHAcvXUtZR47GOGod1lIxm8RtzWsidO3eEi4uLOHXqlBg5cqSYPXu2ReJev35dABCpqamSxmnYsKH49NNPJRn7zp07onXr1mLnzp0iNDRUREdHm3X8yu4ja27vvPOOeOqppySPU1Z0dLRo1aqVKCkpMeu4AwYMEBMmTDCY99xzz4nRo0ebNY5OpxP29vZi27ZtBvOfeOIJERcXZ5YYAERycrJ+uqSkRHh5eYn//Oc/+nk5OTlCqVSKDRs2mCUm1Yw1aqml6qgQ0tVS1tGHwzpaPdZRqopsz+B/8803CAgIwGOPPYaXX34Zn3/+uSQfwZVV+tcj3d3dJRm/uLgYX3/9NfLy8tCtWzdJYkRFRWHAgAHo3bu3JOMDQGZmJnx8fNCyZUuMHj1akttVbd26FZ06dcLw4cPh6emJkJAQrFq1yuxxHlRYWIivvvoKEyZMgEKhMOvY3bt3x65du3DmzBkAwNGjR7Fnzx5ERESYNc69e/dQXFwMBwcHg/mOjo6SnCEE7v8FwqtXrxq859zc3PDkk09i3759ksQk41ijlkpdRwHpaynraM2xjtYM6ygZsPb/MKTSvXt3sWDBAiGEEEVFRaJRo0Zi9+7dksYsLi4WAwYMED169DD72MeOHRPOzs7C3t5euLm5iR9++MHsMYQQYsOGDaJt27bi7t27QgghyZmnH3/8UXzzzTfi6NGjYseOHaJbt26iefPmQqPRmDWOUqkUSqVSxMbGisOHD4sVK1YIBwcHsWbNGrPGedDGjRuFvb29uHTpktnHLi4uFu+8845QKBSiTp06QqFQiPfff9/scYQQolu3biI0NFRcunRJ3Lt3T3z55ZfCzs5OtGnTxizjo8yZp7179woA4vLlywbrDR8+XIwYMcIsMalmLF1LpayjQlimlrKOPhzWUeOwjlJVZNngnzp1StSpU0dcu3ZNPy8qKkq8/PLLksadPHmy8PPzExcuXDD72AUFBSIzM1Okp6eLmTNnikaNGokTJ06YNcb58+eFp6enOHr0qH6eFAemsm7fvi1cXV3N/jF53bp1Rbdu3QzmvfHGG6Jr165mjfOgPn36iIEDB0oy9oYNG0SzZs3Ehg0bxLFjx8QXX3wh3N3dJTnQZmVliZ49ewoAwt7eXnTu3FmMHj1aBAQEmGV8HphqB2vUUinrqBDS11LW0YfHOmoc1lGqiiwb/OnTp+t/oUofdnZ2wtHRUeTk5EgSMyoqSjRr1kz89ddfkoxfVq9evcSkSZPMOmZycnK55w2AUCgUwt7eXty7d8+s8R7UqVMnMXPmTLOO2bx5c/HKK68YzPvkk0+Ej4+PWeOUOnfunLCzsxObN2+WZPxmzZqJJUuWGMxLSkoSjz32mCTxhBBCq9XqDxYjRowQ/fv3N8u4ZQ9MZ8+eFQDEkSNHDNbr2bOnmDp1qllikuksXUstXUeFMH8tZR19OKyjxmMdparI7hr8e/fu4YsvvsDHH3+MjIwM/ePo0aPw8fHBhg0bzBpPCIEpU6YgOTkZv/zyC1QqlVnHr0xJSQkKCgrMOmavXr1w/Phxg+etU6dOGD16NDIyMmBvb2/WeKW0Wi3Onj0Lb29vs47bo0ePcrfaO3PmDPz8/Mwap9Tq1avh6ekp2a0EdTod7OwMf2Xt7e1RUlIiSTwAcHZ2hre3N27fvo2ffvoJQ4YMkSSOSqWCl5cXdu3apZ+n0Whw4MAByb5rQlWzZC21Vh0FzF9LWUcfDutozbGOkgFr/w/D3JKTk0W9evUqPLs0Y8YM0alTJ7PGe+2114Sbm5tISUkRV65c0T90Op3ZYsycOVOkpqYKtVotjh07JmbOnCkUCoX4+eefzRajMlJ8tPzWW2+JlJQUoVarxd69e0Xv3r1Fo0aNxPXr180a5/fffxd16tQRs2fPFpmZmWLdunXCyclJfPXVV2aNI8T96zqbN28u3nnnHbOPXSoyMlI0bdpUbNu2TajVavHdd9+JRo0aiRkzZpg91o4dO8T27dvFX3/9JX7++WcRHBwsnnzySVFYWFjjMe/cuSOOHDkijhw5IgCIefPmiSNHjojs7GwhhBAffPCBaNCggdiyZYs4duyYGDJkiFCpVPrrmMmyLFlLLVFHhbBeLWUdNQ7raPVYR8lYsmvwBw4cWOnHXwcOHBAADK6NfFgAKnysXr3abDEmTJgg/Pz8RL169UTjxo1Fr169LNLcCyHNgWnkyJHC29tb1KtXTzRt2lSMHDlSZGVlmTVGqe+//160bdtWKJVKERAQIFauXClJnJ9++kkAEKdPn5ZkfCGE0Gg0Ijo6WjRv3lw4ODiIli1biri4OFFQUGD2WBs3bhQtW7YU9erVE15eXiIqKuqhL8nYvXt3hb8rkZGRQoj7t3iLj48XTZo0EUqlUvTq1UvS55OqZslaaok6KoT1ainrqHFYR6vHOkrGUghhgXtHEhERERGRRcjuGnwiIiIiokcZG3wiIiIiIhlhg09EREREJCNs8ImIiIiIZIQNPhERERGRjLDBJyIiIiKSETb4REREREQywgafrK5FixZYsGCBVWKPGzcOQ4cOtWhMhUKBzZs3WzQmEckb6ygRPYgNPpUzbtw4KBQKKBQK1K1bF02aNMGzzz6Lzz//HCUlJdZOzySrVq1CcHAwXFxc0KBBA4SEhGDOnDlWzenKlSuIiIiwag5EJC3WUWmxjhJVjQ0+Vahfv364cuUKzp07h+3btyM8PBzR0dEYOHAg7t27Z+30DBQWFlY4//PPP8e0adMwdepUZGRkYO/evZgxYwa0Wq2FMzTk5eUFpVJp1RyISHqso9JhHSWqGht8qpBSqYSXlxeaNm2KJ554Au+++y62bNmC7du3Y82aNfr1cnJyMHHiRDRu3Biurq545plncPToUf3ys2fPYsiQIWjSpAlcXFzQuXNn/O9//6sydnVjJiQkoEOHDvj000+hUqng4OBQ4Thbt27FiBEj8Morr8Df3x9BQUEYNWoUZs+eXW7djz76CN7e3vDw8EBUVBSKior0y27fvo2xY8eiYcOGcHJyQkREBDIzMwEAQgg0btwY3377rX79Dh06wNvbWz+9Z88eKJVK6HQ6AIYfLZ87dw4KhQLfffcdwsPD4eTkhODgYOzbt88gv1WrVsHX1xdOTk4YNmwY5s2bhwYNGlT5PBKRdbGOso4SWQsbfDLaM888g+DgYHz33Xf6ecOHD8f169exfft2HDp0CE888QR69eqFW7duAQC0Wi369++PXbt24ciRI+jXrx8GDRqE8+fPVxqnujEBICsrC5s2bcJ3332HjIyMCsfx8vLC/v37kZ2dXeV+7d69G2fPnsXu3buxdu1arFmzxuDgO27cOKSnp2Pr1q3Yt28fhBDo378/ioqKoFAo0LNnT6SkpAC4fxA7efIk7t69i1OnTgEAUlNT0blzZzg5OVWaQ1xcHN5++21kZGSgTZs2GDVqlP4M3969ezF58mRER0cjIyMDzz77bIUHVyKyfayjrKNEFiGIyoiMjBRDhgypcNnIkSNFYGCgEEKIX3/9Vbi6uor8/HyDdVq1aiVWrFhR6fhBQUFi8eLF+mk/Pz8xf/58o8ecNWuWqFu3rrh+/XqV+3H58mXRtWtXAUC0adNGREZGio0bN4ri4mKDffXz8xP37t3Tzxs+fLgYOXKkEEKIM2fOCABi7969+uU3b94Ujo6O4ptvvhFCCLFo0SIRFBQkhBBi8+bN4sknnxRDhgwRy5YtE0II0bt3b/Huu+/qtwcgkpOThRBCqNVqAUB8+umn+uUnTpwQAMTJkyeFEPef8wEDBhjs2+jRo4Wbm1uV+09E1sM6yjpKZE08g08mEUJAoVAAAI4ePQqtVgsPDw+4uLjoH2q1GmfPngVw/8zT22+/jcDAQDRo0AAuLi44efJkpWeejBkTAPz8/NC4ceMqc/X29sa+fftw/PhxREdH4969e4iMjES/fv0MvuQWFBQEe3t7g+2uX78OADh58iTq1KmDJ598Ur/cw8MDjz32GE6ePAkACA0NxZ9//okbN24gNTUVYWFhCAsLQ0pKCoqKivDbb78hLCysylzbt29vEB+APofTp0+jS5cuBuuXnSai2oN1lHWUSGp1rJ0A1S4nT56ESqUCcP+g4+3trf9Y9UGl1zW+/fbb2LlzJz766CP4+/vD0dERL7zwQqVf6DJmTABwdnY2Oue2bduibdu2eP311zF58mQ8/fTTSE1NRXh4OACgbt26BusrFAqT7nLRrl07uLu7IzU1FampqZg9eza8vLwwd+5cHDx4EEVFRejevXuVYzyYQ+mBv7bdaYOIjMM6Wh7rKJF5scEno/3yyy84fvw43nzzTQDAE088gatXr6JOnTpo0aJFhdvs3bsX48aNw7BhwwDcP/CcO3eu0hjGjPkwHn/8cQBAXl6eUesHBgbi3r17OHDggP7g8vfff+P06dP6sRQKBZ5++mls2bIFJ06cwFNPPQUnJycUFBRgxYoV6NSpk0kH0rIee+wxHDx40GBe2Wkiqh1YR1lHiSyBl+hQhQoKCnD16lVcunQJhw8fxvvvv48hQ4Zg4MCBGDt2LACgd+/e6NatG4YOHYqff/4Z586dw2+//Ya4uDikp6cDAFq3bq3/AtfRo0fx0ksvVXlGxZgxjfXaa68hKSkJe/fuRXZ2Nvbv34+xY8eicePG6Natm1FjtG7dGkOGDMGrr76KPXv24OjRo3j55ZfRtGlTDBkyRL9eWFgYNmzYgA4dOsDFxQV2dnbo2bMn1q1bh9DQUJPyLuuNN97Ajz/+iHnz5iEzMxMrVqzA9u3b9WeoiMg2sY5Cnz/rKJFlscGnCu3YsQPe3t5o0aIF+vXrh927d2PRokXYsmWL/jpLhUKBH3/8ET179sT48ePRpk0bvPjii8jOzkaTJk0AAPPmzUPDhg3RvXt3DBo0CH379sUTTzxRaVxjxjRW7969sX//fgwfPhxt2rTB888/DwcHB+zatQseHh5Gj7N69Wp07NgRAwcORLdu3SCEwI8//mjwcXBoaCiKi4sNrhENCwsrN68mevTogeXLl2PevHkIDg7Gjh078Oabb1Z6Wzsisg2so/+HdZTIshRCCGHtJIjINK+++ipOnTqFX3/91dqpEBHVSqyjJGe8Bp+oFvjoo4/w7LPPwtnZGdu3b8fatWvxySefWDstIqJag3WUHiU8g09UC4wYMQIpKSm4c+cOWrZsiTfeeAOTJ0+2dlpERLUG6yg9StjgExERERHJCL9kS0REREQkI2zwiYiIiIhkhA0+EREREZGMsMEnIiIiIpIRNvhERERERDLCBp+IiIiISEbY4BMRERERyQgbfCIiIiIiGWGDT0REREQkI/8PfU/Pzn3jXUMAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dpq1 = pd.DataFrame(Cw1.items(),columns=['state', 'action'])\n",
        "dpq1.head(30)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ZRsKSnOo8Vav",
        "outputId": "3e22b2fb-3c4b-4942-caa9-11a9730497bb"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "              state         action\n",
              "0    (15, 3, False)  [2.0, 2380.0]\n",
              "1    (10, 6, False)  [1050.0, 4.0]\n",
              "2    (16, 4, False)  [2.0, 2320.0]\n",
              "3    (10, 5, False)  [1046.0, 4.0]\n",
              "4     (14, 6, True)  [214.0, 40.0]\n",
              "5    (17, 4, False)  [2184.0, 2.0]\n",
              "6     (15, 6, True)  [306.0, 12.0]\n",
              "7    (12, 6, False)  [2390.0, 2.0]\n",
              "8    (10, 1, False)   [2.0, 932.0]\n",
              "9   (21, 10, False)  [4864.0, 0.0]\n",
              "10   (20, 3, False)  [3076.0, 0.0]\n",
              "11   (11, 4, False)  [1220.0, 8.0]\n",
              "12   (18, 7, False)  [2104.0, 2.0]\n",
              "13    (9, 5, False)   [846.0, 4.0]\n",
              "14  (11, 10, False)  [4828.0, 4.0]\n",
              "15   (11, 1, False)  [1190.0, 4.0]\n",
              "16   (6, 10, False)  [2.0, 1600.0]\n",
              "17    (21, 6, True)  [1212.0, 0.0]\n",
              "18   (13, 2, False)  [2422.0, 4.0]\n",
              "19   (15, 2, False)  [2.0, 2264.0]\n",
              "20   (15, 9, False)  [2252.0, 2.0]\n",
              "21   (17, 9, False)  [2224.0, 2.0]\n",
              "22  (14, 10, False)  [9646.0, 4.0]\n",
              "23   (19, 6, False)  [2120.0, 0.0]\n",
              "24   (19, 4, False)  [2040.0, 0.0]\n",
              "25    (9, 7, False)   [812.0, 4.0]\n",
              "26  (18, 10, False)  [2.0, 8476.0]\n",
              "27   (18, 4, False)  [2058.0, 2.0]\n",
              "28   (19, 1, False)  [1994.0, 2.0]\n",
              "29   (15, 6, False)  [2216.0, 2.0]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7f7fb83c-a398-4d81-8236-253d227c20c2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>state</th>\n",
              "      <th>action</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(15, 3, False)</td>\n",
              "      <td>[2.0, 2380.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(10, 6, False)</td>\n",
              "      <td>[1050.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(16, 4, False)</td>\n",
              "      <td>[2.0, 2320.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(10, 5, False)</td>\n",
              "      <td>[1046.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(14, 6, True)</td>\n",
              "      <td>[214.0, 40.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>(17, 4, False)</td>\n",
              "      <td>[2184.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>(15, 6, True)</td>\n",
              "      <td>[306.0, 12.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>(12, 6, False)</td>\n",
              "      <td>[2390.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>(10, 1, False)</td>\n",
              "      <td>[2.0, 932.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>(21, 10, False)</td>\n",
              "      <td>[4864.0, 0.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>(20, 3, False)</td>\n",
              "      <td>[3076.0, 0.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>(11, 4, False)</td>\n",
              "      <td>[1220.0, 8.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>(18, 7, False)</td>\n",
              "      <td>[2104.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>(9, 5, False)</td>\n",
              "      <td>[846.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>(11, 10, False)</td>\n",
              "      <td>[4828.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>(11, 1, False)</td>\n",
              "      <td>[1190.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>(6, 10, False)</td>\n",
              "      <td>[2.0, 1600.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>(21, 6, True)</td>\n",
              "      <td>[1212.0, 0.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>(13, 2, False)</td>\n",
              "      <td>[2422.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>(15, 2, False)</td>\n",
              "      <td>[2.0, 2264.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>(15, 9, False)</td>\n",
              "      <td>[2252.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>(17, 9, False)</td>\n",
              "      <td>[2224.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>(14, 10, False)</td>\n",
              "      <td>[9646.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>(19, 6, False)</td>\n",
              "      <td>[2120.0, 0.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>(19, 4, False)</td>\n",
              "      <td>[2040.0, 0.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>(9, 7, False)</td>\n",
              "      <td>[812.0, 4.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>(18, 10, False)</td>\n",
              "      <td>[2.0, 8476.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>(18, 4, False)</td>\n",
              "      <td>[2058.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>(19, 1, False)</td>\n",
              "      <td>[1994.0, 2.0]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>(15, 6, False)</td>\n",
              "      <td>[2216.0, 2.0]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7f7fb83c-a398-4d81-8236-253d227c20c2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7f7fb83c-a398-4d81-8236-253d227c20c2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7f7fb83c-a398-4d81-8236-253d227c20c2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-8366a80c-f310-4bcc-b66c-07038c802359\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8366a80c-f310-4bcc-b66c-07038c802359')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-8366a80c-f310-4bcc-b66c-07038c802359 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import copy\n",
        "\n",
        "def mc_off(behaviour_policy, env, discount=1.0, Q1 = copy.deepcopy(optimal_Q1), C1 = copy.deepcopy(Cw1) ):\n",
        "\n",
        "    C1[(9, 10, False)][1] = 0\n",
        "    target_policy = create_target_policy(Q1)\n",
        "\n",
        "    for i_episode in range(1, 4):\n",
        "\n",
        "        print(f\"\\nEP:{i_episode}\")\n",
        "        #Episodi inventati per capire il codice.\n",
        "\n",
        "        if  i_episode == 1:\n",
        "\n",
        "            episode = [((4, 3, True), 1, 70.0),((6, 3, True), 1, 45.0), ((14, 3, True), 1, 0.0), ((12, 10, False), 1, 0.0), ((19, 10, False), 1, 1.0)]\n",
        "\n",
        "        if i_episode == 2 :\n",
        "\n",
        "            episode = [((18, 7, False), 1, 70.0), ((12, 10, False), 0, - 3000.0), ((19, 10, False), 0, 3000.0)]\n",
        "\n",
        "        if i_episode == 3 :\n",
        "\n",
        "            episode = [((18, 7, False), 0, 0.5), ((9, 10, False), 0, 67.0), ((12, 10, False), 0, 2.5), ((12, 10, False), 0, 0.0), ((20, 10, False), 0, 0.0)]\n",
        "\n",
        "\n",
        "        print(episode[::-1])\n",
        "\n",
        "        G = 0.0\n",
        "        W = 1.0\n",
        "\n",
        "        for t in range(len(episode))[::-1]:\n",
        "            state, action, reward = episode[t]\n",
        "\n",
        "            if action != np.argmax(target_policy(state)):\n",
        "                print(\"\\t  --- in break ---- \")\n",
        "                break\n",
        "\n",
        "            G = discount*G + reward\n",
        "\n",
        "\n",
        "            print(\" \")\n",
        "            print(\"G=\", G)\n",
        "            W = W * (target_policy(state)[action]/behaviour_policy(state)[action])\n",
        "            print(f\"  IN   Q[{state}] =\", Q1[state])\n",
        "\n",
        "            print(\"IN W=\",W)\n",
        "            C1[state][action] = C1[state][action] + W\n",
        "            print(f\"C[{state}][{action}] + W = \", C1[state][action])\n",
        "            print(f\"(W/C[{state}][{action}]) = \", W/C1[state][action])\n",
        "\n",
        "\n",
        "            print(f\"Formula  Q[{state}][{action}] = Q[{state}][{action}] + W/C[{state}][{action}] * (G - Q[{state}][{action}]) \")\n",
        "            Q1[state][action] = Q1[state][action]  + (W/C1[state][action]) * (G - Q1[state][action])\n",
        "            print(f\"  OUT modify Q[{state}][{action}] = \",Q1[state][action])\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "    return print(\"\\nEnd\")\n",
        "\n",
        "\n",
        "behaviour_policy = create_behaviour_policy(env.action_space.n)\n",
        "mc_off( behaviour_policy, env)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "17a3bdf9-fc7a-4afa-d7d7-76436b3d2d67",
        "id": "3j0TCf1S5dTt"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "EP:1\n",
            "[((19, 10, False), 1, 1.0), ((12, 10, False), 1, 0.0), ((14, 3, True), 1, 0.0), ((6, 3, True), 1, 45.0), ((4, 3, True), 1, 70.0)]\n",
            "\t  --- in break ---- \n",
            "\n",
            "EP:2\n",
            "[((19, 10, False), 0, 3000.0), ((12, 10, False), 0, -3000.0), ((18, 7, False), 1, 70.0)]\n",
            " \n",
            "G= 3000.0\n",
            "  IN   Q[(19, 10, False)] = [-0.00896226 -1.        ]\n",
            "IN W= 2.0\n",
            "C[(19, 10, False)][0] + W =  8482.0\n",
            "(W/C[(19, 10, False)][0]) =  0.0002357934449422306\n",
            "Formula  Q[(19, 10, False)][0] = Q[(19, 10, False)][0] + W/C[(19, 10, False)][0] * (G - Q[(19, 10, False)][0]) \n",
            "  OUT modify Q[(19, 10, False)][0] =  0.698420183918887\n",
            " \n",
            "G= 0.0\n",
            "  IN   Q[(12, 10, False)] = [-0.58630528 -1.        ]\n",
            "IN W= 4.0\n",
            "C[(12, 10, False)][0] + W =  9818.0\n",
            "(W/C[(12, 10, False)][0]) =  0.00040741495212874313\n",
            "Formula  Q[(12, 10, False)][0] = Q[(12, 10, False)][0] + W/C[(12, 10, False)][0] * (G - Q[(12, 10, False)][0]) \n",
            "  OUT modify Q[(12, 10, False)][0] =  -0.5860664086371975\n",
            "\t  --- in break ---- \n",
            "\n",
            "EP:3\n",
            "[((20, 10, False), 0, 0.0), ((12, 10, False), 0, 0.0), ((12, 10, False), 0, 2.5), ((9, 10, False), 0, 67.0), ((18, 7, False), 0, 0.5)]\n",
            " \n",
            "G= 0.0\n",
            "  IN   Q[(20, 10, False)] = [0.42391135 0.        ]\n",
            "IN W= 2.0\n",
            "C[(20, 10, False)][0] + W =  12816.0\n",
            "(W/C[(20, 10, False)][0]) =  0.00015605493133583021\n",
            "Formula  Q[(20, 10, False)][0] = Q[(20, 10, False)][0] + W/C[(20, 10, False)][0] * (G - Q[(20, 10, False)][0]) \n",
            "  OUT modify Q[(20, 10, False)][0] =  0.4238451935081158\n",
            " \n",
            "G= 0.0\n",
            "  IN   Q[(12, 10, False)] = [-0.58606641 -1.        ]\n",
            "IN W= 4.0\n",
            "C[(12, 10, False)][0] + W =  9822.0\n",
            "(W/C[(12, 10, False)][0]) =  0.00040724903278354713\n",
            "Formula  Q[(12, 10, False)][0] = Q[(12, 10, False)][0] + W/C[(12, 10, False)][0] * (G - Q[(12, 10, False)][0]) \n",
            "  OUT modify Q[(12, 10, False)][0] =  -0.5858277336591331\n",
            " \n",
            "G= 2.5\n",
            "  IN   Q[(12, 10, False)] = [-0.58582773 -1.        ]\n",
            "IN W= 8.0\n",
            "C[(12, 10, False)][0] + W =  9830.0\n",
            "(W/C[(12, 10, False)][0]) =  0.0008138351983723296\n",
            "Formula  Q[(12, 10, False)][0] = Q[(12, 10, False)][0] + W/C[(12, 10, False)][0] * (G - Q[(12, 10, False)][0]) \n",
            "  OUT modify Q[(12, 10, False)][0] =  -0.5833163784333678\n",
            "\t  --- in break ---- \n",
            "\n",
            "End\n"
          ]
        }
      ]
    }
  ]
}