Homework/期末專題實作(三)/電商網頁銷售改進效果之AB Test分析.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CUUk4M9ESPiL"
   },
   "source": [
    "# **[作業]電商如何以A/B Test 驗證新網頁設計有效**\n",
    "## **Can eCommerce UX change boost the conversion rate from 0.13 to 0.15?**\n",
    "知識點:\n",
    "\n",
    "*   effect size\n",
    "*   sample size for A/B test \n",
    "*   type I error = 0.05 and Power= 0.8\n",
    "*   z-score, confidence interval\n",
    "\n",
    "參考：[A/B testing: A step-by-step guide in Python](https://medium.com/@RenatoFillinich/ab-testing-with-python-e5964dd66143) by Renato Fillinich @ medium.com\n",
    "\n",
    "數據 : ab_data.csv from Kaggle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_CFV1buAeAMz"
   },
   "source": [
    "# **[作業目標]**\n",
    "\n",
    "1.   了解 Binomial 分布，以及用常態分布求統計解的方法\n",
    "2.   判讀A/B Test 結果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "19cuAxsOeI3k"
   },
   "source": [
    "# **[作業重點]**\n",
    "\n",
    "\n",
    "\n",
    "1.   如何決定最小樣本數\n",
    "2.   如何以 Z 值，p-Value和信賴區間(Confidence Interval)判斷A/B結果是否顯著"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b5pbmX1PCiUD",
    "outputId": "523d70e1-293c-4d36-bfc1-2acfcf4d6f1e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4720\n"
     ]
    }
   ],
   "source": [
    "# Packages imports\n",
    " \n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats as stats\n",
    "import statsmodels.stats.api as sms\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from math import ceil\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Some plot styling preferences\n",
    "#plt.style.use('seaborn-whitegrid')\n",
    "#font = {'family' : 'Helvetica',\n",
    "#        'weight' : 'bold',\n",
    "#        'size'   : 14}\n",
    "\n",
    "#mpl.rc('font', **font)\n",
    "\n",
    "#求樣本大小\n",
    "effect_size = sms.proportion_effectsize(0.13, 0.15)    # Calculating effect size based on our expected rates\n",
    "\n",
    "required_n = sms.NormalIndPower().solve_power( effect_size, \n",
    "                                               power=0.8, \n",
    "                                               alpha=0.05, \n",
    "                                               ratio=1 \n",
    "                                             )         # Calculating sample size needed\n",
    "\n",
    "required_n = ceil(required_n)                          # Rounding up to next whole number                          \n",
    "\n",
    "print(required_n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 198
    },
    "id": "NzCIP30MEIWw",
    "outputId": "0fac92d3-76f1-4ad2-c881-a944e20b74eb"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#展示實驗資料\n",
    "\n",
    "df = pd.read_csv( 'ab_data.csv' )\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count   Dtype \n",
      "---  ------        --------------   ----- \n",
      " 0   user_id       294478 non-null  int64 \n",
      " 1   timestamp     294478 non-null  object\n",
      " 2   group         294478 non-null  object\n",
      " 3   landing_page  294478 non-null  object\n",
      " 4   converted     294478 non-null  int64 \n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 345
    },
    "id": "i1FXBm0sEMz4",
    "outputId": "f954c865-9cbf-453b-918b-e41685189926"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Data =  (294478, 5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>landing_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>control</th>\n",
       "      <td>1928</td>\n",
       "      <td>145274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>treatment</th>\n",
       "      <td>145311</td>\n",
       "      <td>1965</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "landing_page  new_page  old_page\n",
       "group                           \n",
       "control           1928    145274\n",
       "treatment       145311      1965"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To make sure all the control group are seeing the old page and viceversa\n",
    "# 用 crosstab 將 landing_page 當作 column，group 當作 row\n",
    "\n",
    "print( 'Shape of Data = ', df.shape )\n",
    "pd.crosstab( df['group'], df['landing_page'] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "smK1d9OoEQ4c",
    "outputId": "aaf70cfa-c9cf-46b3-e10b-186bce4e13b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 3894 users that appear multiple times in the dataset\n"
     ]
    }
   ],
   "source": [
    "# 偵測重複出現使用者\n",
    "session_counts = df['user_id'].value_counts( ascending=False )\n",
    "multi_users = session_counts[session_counts > 1].count()\n",
    "\n",
    "print(f'There are {multi_users} users that appear multiple times in the dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1Iu-jUPxEUFX",
    "outputId": "fe577125-b873-4c10-bf70-c7199b1f57ce"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The updated dataset now has 286690 entries\n"
     ]
    }
   ],
   "source": [
    "# 除去重複出現使用者\n",
    "users_to_drop = session_counts[session_counts > 1].index\n",
    "\n",
    "df = df[~df['user_id'].isin(users_to_drop)]\n",
    "print( f'The updated dataset now has {df.shape[0]} entries' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "id": "gOh_R80CEYVs",
    "outputId": "47915ff9-eaa4-4f59-eb5d-e941be57d61d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>763854</td>\n",
       "      <td>2017-01-21 03:43:17.188315</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>690555</td>\n",
       "      <td>2017-01-18 06:38:13.079449</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>861520</td>\n",
       "      <td>2017-01-06 21:13:40.044766</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>630778</td>\n",
       "      <td>2017-01-05 16:42:36.995204</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>656634</td>\n",
       "      <td>2017-01-04 15:31:21.676130</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9435</th>\n",
       "      <td>908512</td>\n",
       "      <td>2017-01-14 22:02:29.922674</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9436</th>\n",
       "      <td>873211</td>\n",
       "      <td>2017-01-05 00:57:16.167151</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9437</th>\n",
       "      <td>631276</td>\n",
       "      <td>2017-01-20 18:56:58.167809</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9438</th>\n",
       "      <td>662301</td>\n",
       "      <td>2017-01-03 08:10:57.768806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9439</th>\n",
       "      <td>944623</td>\n",
       "      <td>2017-01-19 10:56:01.648653</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9440 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "0      763854  2017-01-21 03:43:17.188315    control     old_page          0\n",
       "1      690555  2017-01-18 06:38:13.079449    control     old_page          0\n",
       "2      861520  2017-01-06 21:13:40.044766    control     old_page          0\n",
       "3      630778  2017-01-05 16:42:36.995204    control     old_page          0\n",
       "4      656634  2017-01-04 15:31:21.676130    control     old_page          0\n",
       "...       ...                         ...        ...          ...        ...\n",
       "9435   908512  2017-01-14 22:02:29.922674  treatment     new_page          0\n",
       "9436   873211  2017-01-05 00:57:16.167151  treatment     new_page          0\n",
       "9437   631276  2017-01-20 18:56:58.167809  treatment     new_page          0\n",
       "9438   662301  2017-01-03 08:10:57.768806  treatment     new_page          0\n",
       "9439   944623  2017-01-19 10:56:01.648653  treatment     new_page          1\n",
       "\n",
       "[9440 rows x 5 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 選取 控制組和實驗組各半 4720 * 2 = 9440\n",
    "control_sample = df[df['group'] == 'control'].sample(n=required_n, random_state=22)\n",
    "treatment_sample = df[df['group'] == 'treatment'].sample(n=required_n, random_state=22)\n",
    "\n",
    "ab_test = pd.concat([control_sample, treatment_sample], axis=0)\n",
    "ab_test.reset_index(drop=True, inplace=True)\n",
    "ab_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "I0DU6C8uEbbZ",
    "outputId": "213e691f-a140-4ac0-e615-ff5e28b6396e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9440 entries, 0 to 9439\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count  Dtype \n",
      "---  ------        --------------  ----- \n",
      " 0   user_id       9440 non-null   int64 \n",
      " 1   timestamp     9440 non-null   object\n",
      " 2   group         9440 non-null   object\n",
      " 3   landing_page  9440 non-null   object\n",
      " 4   converted     9440 non-null   int64 \n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 368.9+ KB\n"
     ]
    }
   ],
   "source": [
    "ab_test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "viU4ZRx6W3Yh",
    "outputId": "6a47ab36-3255-4888-92e4-25fdbd166a7f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "treatment    4720\n",
       "control      4720\n",
       "Name: group, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 確認 ab_test 控制組實驗組各半\n",
    "ab_test['group'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 101
    },
    "id": "ewFAGIwMEfJJ",
    "outputId": "081f4f2a-d31e-4fd7-bcf0-6a57389d02fd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "</style><table id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >conversion_rate</th>        <th class=\"col_heading level0 col1\" >std_deviation</th>        <th class=\"col_heading level0 col2\" >std_error</th>    </tr>    <tr>        <th class=\"index_name level0\" >group</th>        <th class=\"blank\" ></th>        <th class=\"blank\" ></th>        <th class=\"blank\" ></th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199level0_row0\" class=\"row_heading level0 row0\" >control</th>\n",
       "                        <td id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199row0_col0\" class=\"data row0 col0\" >0.123</td>\n",
       "                        <td id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199row0_col1\" class=\"data row0 col1\" >0.329</td>\n",
       "                        <td id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199row0_col2\" class=\"data row0 col2\" >0.005</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199level0_row1\" class=\"row_heading level0 row1\" >treatment</th>\n",
       "                        <td id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199row1_col0\" class=\"data row1 col0\" >0.126</td>\n",
       "                        <td id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199row1_col1\" class=\"data row1 col1\" >0.331</td>\n",
       "                        <td id=\"T_7421a9a2_8d77_11eb_a42f_c809a8005199row1_col2\" class=\"data row1 col2\" >0.005</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1e3261cd308>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 計算轉換率(conversion rate)、平均值，標準差，標準誤\n",
    "conversion_rates = ab_test.groupby('group')['converted']\n",
    "\n",
    "std_p = lambda x: np.std(x, ddof=0)              # Std. deviation of the proportion\n",
    "se_p = lambda x: stats.sem(x, ddof=0)            # Std. error of the proportion (std / sqrt(n))\n",
    "\n",
    "conversion_rates = conversion_rates.agg([np.mean, std_p, se_p])\n",
    "conversion_rates.columns = ['conversion_rate', 'std_deviation', 'std_error']\n",
    "\n",
    "conversion_rates.style.format('{:.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 478
    },
    "id": "IVM21KkUEitI",
    "outputId": "a980c634-be8e-4c2f-ff32-f57c59a78047"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAGcCAYAAABEA6NcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7gddX3v8ffHBERUQGRXkAQTLUcbW6wYuQhV0FrBG9aqxTtqRVQE6u3gpd6O7bHWWrRSU46Coggi9RI1ilZEj8ol4VIgUDwRQSJQgihYUCDwPX/MRJbbfVkhe629suf9ep717Jn5/WbWd62wmc+e+c1MqgpJktRd95rtAiRJ0uwyDEiS1HGGAUmSOs4wIElSxxkGJEnqOMOAJEkdZxiQRJKvJXnpbNexqZIsSlJJ5s92LdLmxDAgzYAkL0iyKsl/J7m23bnuO9t19auqDqyqT852He7MpdlhGJA2UZLXA8cAfwc8CNgF+BfgoNmsq9eo7FyTzJvtGgZhVL5f6Z4yDEibIMm2wHuA11bV56vqlqq6o6q+XFVvavvcO8kxSa5pX8ckuXfbtl+StUnekOT69qjCy9q2vZJc17sDTfLnSS5qp++V5OgkP0rysySnJtm+bdvwF/YrkvwEOCPJVkk+3fb9RZKVSR7U9j8zyV/1bPftSa5qazqx/Zy9231pkp8kuSHJ26b4fj6R5KNJViS5Bdg/ydOSXJDk5iRXJ3lXzyrfbX/+oj3Ksne7nZcnuSzJz5OcnuQh0/zTvLz9rq9N8oZ2GzsmuTXJA3vqe0ySdUm2mKD2+yT5ZPuelyV5c5K1Pe1XJvmf7b/HLUnmJ3lmktXt93tmkj/o6V9Jfn/cd/PednrDfwdvbb/TK5O8cJrPKM0Yw4C0afYGtgK+MEWftwF7AX8MPArYA3h7T/uOwLbAzsArgGOTPKCqzgZuAZ7Y0/cFwGfa6SOAZwFPAB4M/Bw4dtx7PwH4A+ApwEvb91kIPBA4DPjVBPUe0r72Bx4K3A/4yLg++wIPB54EvKN3pzeBFwB/C9wf+F77mV4CbAc8DXh1kme1fR/f/tyuqu5XVWe1bW8Fng2MAf8XOHmK96OtfVfgz4Cjk/xpVV0HnAk8r6ffi4BTquqOCbbxTmARzXfw5LbveM9vP8N2bb+TgaPaOlcAX06y5TS1brAjsAPNfwcvBY5L8vA+15U2TVX58uXrHr6AFwLXTdPnR8BTe+afAlzZTu9Hs0Oe39N+PbBXO/1e4Ph2+v40O9KHtPOXAU/qWW8n4A5gPs1OrICH9rS/HPgBsNsENZ4J/FU7/S3gNT1tD59guwt62s8FDp7ks38COHGa7+cY4J/a6Q3b7/0+vga8omf+XsCtG76HcdvasP4jepa9H/h4O/2XwPfb6XnAdcAek9R1BfCUnvm/Atb2zF8JvLxn/m+AU8fV+VNgv3a+gN8f9928t+e/g/XAfXvaTwX+Zrb/G/fVjZdHBqRN8zNgh2nOGT8YuKpn/qp22W+2UVXre+ZvpflrHJqjAM9uTys8Gzi/qjZs6yHAF9pD0r+gCQd30oxb2ODqnulPAacDp7SH0N8/0eHxSeqdP267101S70R6ayDJnkm+3R6ev4nmCMUOU6z/EOBDPZ/zRiA0f0H385693/eXgCVJNvy1f1NVnTvJNh48bjtXT9Cnd9lvfW9VdVfbPlWdvX5eVbdMUrc0UIYBadOcBfya5nD9ZK6h2aFtsEu7bFpVdSnNTuFAfvsUATQ7mgOrarue11ZV9dPeTfRs646qendVLQEeBzyd5nB9P/WuB/6rn5on+hjj5j8DLAcWVtW2wDKanftEfaH5nK8a9znvU1U/mOI9F/ZM/+b7rqpf0/zF/ULgxTQBaTLXAgsm2eYGvfX+1veWJO06G/49bgW27um/47htPSDJfSeqWxo0w4C0CarqJuAdNOf5n5Vk6yRbJDkwyfvbbicDb08ylmSHtv+nN+JtPkMzPuDxwOd6li8D/nbDYLp2+5NewZBk/yR/1A5IvJnm0P+dE3Q9GfjrJIuT3I/mKonPjjt6sSnuD9xYVb9OsgdNyNlgHXAXzfn3DZYBb0nyyPZzbJvkudO8x9+0/xaPBF4GfLan7USaMRHPZOp/h1Pb931Akp2Bw6d5z1OBpyV5UnvE5Q3AbTSnZgAuBF6QZF6SA2jGc4z37iRbJvkTmrD2uQn6SDPOMCBtoqr6IPB6mkGB62j+kj0c+GLb5b3AKuAi4GLg/HZZv06mOad8RlXd0LP8QzR/YX8jyS+Bs4E9p9jOjsBpNEHgMuA7TLwzPJ7mL+bvAj+mOfLxuo2odzqvAd7T1vwOmp0oAFV1K81gw++3pwX2qqovAH9Pc3rjZuASmiMlU/kOsIZm/MMHquobPe/xfZrAcX5VXTnFNt4DrKX5Dv6d5ru7bbLOVXU5zSDDfwZuAJ4BPKOqbm+7HNku+wXNkYkvjtvEdTSDQK8BTgIOq6r/nOZzSjMiVRMdlZOkuSvJGcBnqupjG7HOq2kGSk70F/2m1rMf8OmqWjBdX2kQPDIgqVOSPBbYnd8+dTBRv52S7JPmvgsPpznsP9UlpNJmy7tmSeqMJJ+kGex5ZFX9cpruWwL/CiymObR/Cs2dJaU5x9MEkiR1nKcJJEnqOMOAJEkdZxiQJKnjDAOSJHWcYUCSpI4zDEiS1HGGAUmSOs4wIElSxxkGJEnqOMOAJEkdZxiQJKnjDAOSJHWcYUCSpI4zDEiS1HGGAUmSOs4wIElSx82f7QJmyw477FCLFi2a7TIkSRqK884774aqGpuorbNhYNGiRaxatWq2y5AkaSiSXDVZm6cJJEnqOMOAJEkdZxiQJKnjDAOSJHWcYUCSpI4zDEiS1HGGAUmSOs4wIElSxxkGJEnqOMOAJEkdZxiQJKnjDAOSJHWcYUCSpI4bmTCQ5IAklydZk+ToCdofkeSsJLcleeO4tu2SnJbkP5NclmTv4VUuSdLmbSQeYZxkHnAs8GRgLbAyyfKqurSn243AEcCzJtjEh4CvV9VzkmwJbD3omiVJmitG5cjAHsCaqrqiqm4HTgEO6u1QVddX1Urgjt7lSbYBHg98vO13e1X9YjhlS5K0+RuVMLAzcHXP/Np2WT8eCqwDTkhyQZKPJbnvTBcoSdJcNSphIBMsqz7XnQ/sDny0qh4N3AL8zpgDgCSHJlmVZNW6devuWaWSJM0xoxIG1gILe+YXANdsxLprq+qcdv40mnDwO6rquKpaWlVLx8bG7nGxkiTNJaMSBlYCuyZZ3A4APBhY3s+KVXUdcHWSh7eLngRcOsUqkiSpx0hcTVBV65McDpwOzAOOr6rVSQ5r25cl2RFYBWwD3JXkKGBJVd0MvA44qQ0SVwAvm5UPIknSZmgkwgBAVa0AVoxbtqxn+jqa0wcTrXshsHSgBUqSNEeNymkCSZI0SwwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUseNTBhIckCSy5OsSXL0BO2PSHJWktuSvHGC9nlJLkjyleFULEnS3DASYSDJPOBY4EBgCfD8JEvGdbsROAL4wCSbORK4bGBFSpI0R41EGAD2ANZU1RVVdTtwCnBQb4equr6qVgJ3jF85yQLgacDHhlGsJElzyaiEgZ2Bq3vm17bL+nUM8Gbgrqk6JTk0yaokq9atW7fxVUqSNAeNShjIBMuqrxWTpwPXV9V50/WtquOqamlVLR0bG9vYGiVJmpNGJQysBRb2zC8Arulz3X2AZya5kub0whOTfHpmy5Mkae4alTCwEtg1yeIkWwIHA8v7WbGq3lJVC6pqUbveGVX1osGVKknS3DJ/tgsAqKr1SQ4HTgfmAcdX1eokh7Xty5LsCKwCtgHuSnIUsKSqbp61wiVJmgNS1dep+Tln6dKltWrVqtkuQ5KkoUhyXlUtnahtVE4TSJKkWWIYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjpu/mwXIEkz4Sfv+aPZLkGaEbu84+Khv6dHBiRJ6jjDgCRJHWcYkCSp4xwzMMMe86YTZ7sEaZOd9w8vme0SJA2RRwYkSeo4w4AkSR03MmEgyQFJLk+yJsnRE7Q/IslZSW5L8sae5QuTfDvJZUlWJzlyuJVLkrR5G4kxA0nmAccCTwbWAiuTLK+qS3u63QgcATxr3OrrgTdU1flJ7g+cl+Sb49aVJEmTGJUjA3sAa6rqiqq6HTgFOKi3Q1VdX1UrgTvGLb+2qs5vp38JXAbsPJyyJUna/I1KGNgZuLpnfi33YIeeZBHwaOCcGalKkqQOGJUwkAmW1UZtILkf8G/AUVV18yR9Dk2yKsmqdevW3YMyJUmae0YlDKwFFvbMLwCu6XflJFvQBIGTqurzk/WrquOqamlVLR0bG7vHxUqSNJeMShhYCeyaZHGSLYGDgeX9rJgkwMeBy6rqgwOsUZKkOWkkriaoqvVJDgdOB+YBx1fV6iSHte3LkuwIrAK2Ae5KchSwBNgNeDFwcZIL202+tapWDP2DSJK0GRqJMADQ7rxXjFu2rGf6OprTB+N9j4nHHEiSpD6MymkCSZI0SwwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUsdNe2lhkt8D9gEeDPwKuARYVVV3Dbg2SZI0BJOGgST7A0cD2wMXANcDW9E8QvhhSU4D/nGy5wBIkqTNw1RHBp4KvLKqfjK+Icl84OnAk2meCSBJkjZTk4aBqnrTFG3rgS8OpCJJkjRU/YwZuDfwF8Ci3v5V9Z7BlSVJkoaln2cTfAm4CTgPuG2w5UiSpGHrJwwsqKoDBl6JJEmaFf3cZ+AHSf5o4JVIkqRZ0c+RgX2BQ5L8mOY0QYCqqt0GWpkkSRqKfsLAgQOvQpIkzZppTxNU1VXAdsAz2td27TJJkjQHTBsGkhwJnAT8Xvv6dJLXDbowSZI0HP2cJngFsGdV3QKQ5O+Bs4B/HmRhkiRpOPq5miDAnT3zd7bLJEnSHNDPkYETgHOSfKGdfxbw8cGVJEmShmnaMFBVH0xyJs0lhgFeVlUXDLowSZI0HFM9wnibqro5yfbAle1rQ9v2VXXj4MuTJEmDNtWRgc/QPKb4PKB6lqedf+gA65IkSUMy1SOMn97+XDy8ciRJ0rD1c5+Bb/WzTJIkbZ6mGjOwFbA1sEOSB3D35YTbAA8eQm2SJGkIphoz8CrgKJod/3ncHQZuBo4dcF2SJGlIphoz8KEkHwHeWlX/a4g1SZKkIZpyzEBV3Qk8dUi1SJKkWdDP7Yi/keQvkgz0FsRJDkhyeZI1SY6eoP0RSc5KcluSN27MupIkaXL93I749cB9gTuT/Ir2PgNVtc1MFZFkHs04hCcDa4GVSZZX1aU93W4EjqC5HfLGritJkiYx7ZGBqrp/Vd2rqraoqm3a+RkLAq09gDVVdUVV3Q6cAhw0ro7rq2olcMfGritJkibXz5EBkjwTeHw7e2ZVfWWG69gZuLpnfi2w50yvm+RQ4FCAXXbZZeOrlCRpDurnpkPvA44ELm1fR7bLZtJE4xFqgmWbtG5VHVdVS6tq6djYWN/FSZI0l/VzZOCpwB9X1V0AST4JXADM5EC9tcDCnvkFwDVDWFeSpM7r52oCgO16prcdQB0rgV2TLE6yJXAwsHwI60qS1Hn9HBn438AFSb5Nc0j+8cBbZrKIqlqf5HDgdGAecHxVrU5yWNu+LMmOwCqa2yHfleQoYEn7mOXfWXcm65MkaS6bNgxU1clJzgQe2y76n1V13UwXUlUrgBXjli3rmb6O5hRAX+tKkqT+9HU1AbA3sC/NwLx5wBcGVpEkSRqqfq4m+BfgMOBi4BLgVUl8UJEkSXNEP0cGngD8YVUV/OZqgosHWpUkSRqafq4muBzovUPPQuCiwZQjSZKGrZ8jAw8ELktybjv/WOCsJMsBquqZgypOkiQNXj9h4B0Dr0KSJM2afi4t/E6SB3H3pYXnVtX1gy1LkiQNSz9XEzwPOBd4LvA84Jwkzxl0YZIkaTj6OU3wNuCxG44GJBkD/h04bZCFSZKk4ejnaoJ7jTst8LM+15MkSZuBfo4MfD3J6cDJ7fxf4q1/JUmaM6YMA0kCfJhm8OC+NA8qOq6qvB2xJElzxJRhoKoqyRer6jHA54dUkyRJGqJ+zv2fneSx03eTJEmbo37GDOwPHJbkSuAWmlMFVVW7DbIwSZI0HP2EgQMHXoUkSZo1/dyB8Koku9MMICzg+1V1/sArkyRJQ9HPHQjfAXyS5oFFOwAnJHn7oAuTJEnD0c9pgucDj66qXwMkeR9wPvDeQRYmSZKGo5+rCa4EtuqZvzfwo4FUI0mShq6fIwO3AauTfJNmzMCTge8l+TBAVR0xwPokSdKA9RMGvtC+NjhzMKVIkqTZ0M/VBJ8cRiGSJGl2TDpmIMmXkzwjyRYTtD00yXuSvHyw5UmSpEGb6sjAK4HXA8ckuRFYRzOQcDGwBvhIVX1p8CVKkqRBmjQMVNV1wJuBNydZBOwE/Ar4YVXdOpTqJEnSwPUzgJCqupLmEkNJkjTH9HOfAUmSNIcZBiRJ6riRCQNJDkhyeZI1SY6eoD1JPty2X9Q+PGlD218nWZ3kkiQnJ9lq/PqSJGlik44ZSHIxzR0HJ1RVu81UEUnmAcfS3N1wLbAyyfKqurSn24HAru1rT+CjwJ5JdgaOAJZU1a+SnAocDHxipuqTJGkum2oA4dPbn69tf36q/flCYKavJtgDWFNVVwAkOQU4COgNAwcBJ1ZVAWcn2S7JTm3bfOA+Se4AtgaumeH6JEmasyY9TVBVV1XVVcA+VfXmqrq4fR0NPGWG69gZuLpnfm27bNo+VfVT4APAT4BrgZuq6hsTvUmSQ5OsSrJq3bp1M1a8JEmbs37GDNw3yb4bZpI8DrjvDNeRCZaNP0UxYZ8kD6A5arAYeDBNvS+a6E2q6riqWlpVS8fGxjapYEmS5op+7jPwCuD4JNvS7KBvAmb6NsRrgYU98wv43UP9k/X5U+DHVbUOIMnngccBn57hGiVJmpP6eVDRecCjkmwDpKpuGkAdK4FdkywGfkozAPAF4/osBw5vxxPsSXM64NokPwH2SrI1zR0SnwSsGkCNkiTNSdOeJkjyoCQfBz5bVTclWZLkFTNZRFWtBw4HTgcuA06tqtVJDktyWNttBXAFzXMR/g/wmnbdc4DTgPOBi9vPdNxM1idJ0lzWz2mCTwAnAG9r538IfBb4+EwWUlUraHb4vcuW9UwXd1/ZMH7ddwLvnMl6JEnqin4GEO5QVacCd8Fv/oq/c6BVSZKkoeknDNyS5IG0o/uT7EUziFCSJM0B/ZwmeD3N4L2HJfk+MAY8d6BVSZKkoeknDKwGngA8nOZa/8sZoWcaSJKkTdPPTv2sqlpfVaur6pKqugM4a9CFSZKk4ZjqQUU70twC+D5JHs3ddwDchub+/5IkaQ6Y6jTBU4BDaO7094/cHQZuBt462LIkSdKwTBoGquqTST4FPL+qThpiTZIkaYimHDNQVXcBrxpSLZIkaRb0M4Dwm0nemGRhku03vAZemSRJGop+Li3c8ITC3lsBF/DQmS9HkiQNWz9PLVw8jEIkSdLs6OephVsneXuS49r5XZM8ffClSZKkYehnzMAJwO3A49r5tcB7B1aRJEkaqn7CwMOq6v3AHQBV9SvuvueAJEnazPUTBm5Pch/ufmrhw4DbBlqVJEkamn6uJngX8HVgYZKTgH1o7kwoSZLmgH6uJvhGkvOAvWhODxxZVTcMvDJJkjQU04aBJMuBk4HlVXXL4EuSJEnD1M+YgX8E/gS4NMnnkjwnyVYDrkuSJA1JP6cJvgN8J8k84InAK4HjaR5lLEmSNnP9DCCkvZrgGcBfArsDnxxkUZIkaXj6GTPwWWBPmisKjgXObJ9mKEmS5oB+jgycALygqu4cdDGSJGn4+hkz8PUkj0uyqLd/VZ04wLokSdKQ9HOa4FPAw4ALgQ1HBwowDEiSNAf0c5pgKbCkqmrQxUiSpOHr5z4DlwA7DroQSZI0O/o5MrADzQ2HzqXnAUVV9cyBVSVJkoam3wcVDVySA4APAfOAj1XV+8a1p21/KnArcEhVnd+2bQd8DPhDmvEML6+qs4ZRtyRJm7u+7kCY5EHAY9tF51bV9TNZRHt3w2OBJwNrgZVJllfVpT3dDgR2bV97Ah9tf0ITEr5eVc9JsiWw9UzWJ0nSXDbtmIEkzwPOBZ4LPA84J8lzZriOPYA1VXVFVd0OnAIcNK7PQcCJ1Tgb2C7JTkm2AR4PfBygqm6vql/McH2SJM1Z/ZwmeBvw2A1HA5KMAf8OnDaDdewMXN0zv5a7/+qfqs/OwHpgHXBCkkcB59E8ZtknLEqS1Id+ria417jTAj/rc72NkQmWjb+UcbI+82mel/DRqno0cAtw9IRvkhyaZFWSVevWrduUeiVJmjP62al/PcnpSQ5JcgjwVeBrM1zHWmBhz/wC4Jo++6wF1lbVOe3y02jCwe+oquOqamlVLR0bG5uRwiVJ2txNGwaq6k3AvwK7AY8CjquqN89wHSuBXZMsbgcAHgwsH9dnOfCSNPYCbqqqa6vqOuDqJA9v+z0JuBRJktSXSccMJPl94EFV9f2q+jzw+Xb545M8rKp+NFNFVNX6JIcDp9NcWnh8Va1OcljbvgxYQXNZ4RqaSwtf1rOJ1wEntUHiinFtkiRpClMNIDwGeOsEy29t254xk4VU1QqaHX7vsmU90wW8dpJ1L6S5bbIkSdpIU50mWFRVF41fWFWrgEUDq0iSJA3VVGFgqyna7jPThUiSpNkxVRhYmeSV4xcmeQXNtfySJGkOmGrMwFHAF5K8kLt3/kuBLYE/H3RhkiRpOCYNA1X1X8DjkuxP8wAggK9W1RlDqUySJA1FPw8q+jbw7SHUIkmSZsFM31ZYkiRtZgwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHTcyYSDJAUkuT7ImydETtCfJh9v2i5LsPq59XpILknxleFVLkrT5G4kwkGQecCxwILAEeH6SJeO6HQjs2r4OBT46rv1I4LIBlypJ0pwzEmEA2ANYU1VXVNXtwCnAQeP6HAScWI2zge2S7ASQZAHwNOBjwyxakqS5YFTCwM7A1T3za9tl/fY5BngzcNdUb5Lk0CSrkqxat27dplUsSdIcMSphIBMsq376JHk6cH1VnTfdm1TVcVW1tKqWjo2N3ZM6JUmac0YlDKwFFvbMLwCu6bPPPsAzk1xJc3rhiUk+PbhSJUmaW0YlDKwEdk2yOMmWwMHA8nF9lgMvaa8q2Au4qaquraq3VNWCqlrUrndGVb1oqNVLkrQZmz/bBQBU1fokhwOnA/OA46tqdZLD2vZlwArgqcAa4FbgZbNVryRJc8lIhAGAqlpBs8PvXbasZ7qA106zjTOBMwdQniRJc9aonCaQJEmzxDAgSVLHGQYkSeo4w4AkSR1nGJAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLHGQYkSeo4w4AkSR1nGJAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLHGQYkSeo4w4AkSR1nGJAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLHGQYkSeo4w4AkSR1nGJAkqeNGJgwkOSDJ5UnWJDl6gvYk+XDbflGS3dvlC5N8O8llSVYnOXL41UuStPkaiTCQZB5wLHAgsAR4fpIl47odCOzavg4FPtouXw+8oar+ANgLeO0E60qSpEmMRBgA9gDWVNUVVXU7cApw0Lg+BwEnVuNsYLskO1XVtVV1PkBV/RK4DNh5mMVLkrQ5G5UwsDNwdc/8Wn53hz5tnySLgEcD50z0JkkOTbIqyap169ZtYsmSJM0NoxIGMsGy2pg+Se4H/BtwVFXdPNGbVNVxVbW0qpaOjY3d42IlSZpLRiUMrAUW9swvAK7pt0+SLWiCwElV9fkB1ilJ0pwzKmFgJbBrksVJtgQOBpaP67MceEl7VcFewE1VdW2SAB8HLquqDw63bEmSNn/zZ7sAgKpan+Rw4HRgHnB8Va1OcljbvgxYATwVWAPcCrysXX0f4MXAxUkubJe9tapWDPMzSJK0uRqJMADQ7rxXjFu2rGe6gNdOsN73mHg8gSRJ6sOonCaQJEmzxDAgSVLHGQYkSeo4w4AkSR1nGJAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLHGQYkSeo4w4AkSR1nGJAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLHGQYkSeo4w4AkSR1nGJAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLHGQYkSeo4w4AkSR1nGJAkqeNGJgwkOSDJ5UnWJDl6gvYk+XDbflGS3ftdV5IkTW4kwkCSecCxwIHAEuD5SZaM63YgsGv7OhT46EasK0mSJjESYQDYA1hTVVdU1e3AKcBB4/ocBJxYjbOB7ZLs1Oe6kiRpEqMSBnYGru6ZX9su66dPP+tKkqRJzJ/tAlqZYFn12aefdZsNJIfSnGIA+O8kl/ddoUbJDsANs13EXJYPvHS2S9Bo8ndvGN450W5tRjxksoZRCQNrgYU98wuAa/rss2Uf6wJQVccBx21qsZpdSVZV1dLZrkPqGn/35q5ROU2wEtg1yeIkWwIHA8vH9VkOvKS9qmAv4KaqurbPdSVJ0iRG4sgfhowAAAWQSURBVMhAVa1PcjhwOjAPOL6qVic5rG1fBqwAngqsAW4FXjbVurPwMSRJ2iylasLT69LISnJoe8pH0hD5uzd3GQYkSeq4URkzIEmSZolhQJudJIuSvOAerHdIko8MoiZpFCTZLslrRm1bo/A+mpphQJujRcCEYSDJSAyKlWbJdsDv7Fjb27bPyLYGYFjvoykYBjR0SV7SPmzqP5J8KslDknyrXfatJLu0/T7RPpzqB0muSPKcdhPvA/4kyYVJ/rr9i/9zSb4MfCPJ9km+2G7v7CS7zdqHlYbrfcDD2t+NlUm+neQzwMVJ5iX5h3b5RUleBZDkfu3v3flJLk5y0ATb+ock+yX5TpJTk/wwyfuSvDDJue16D2u3N5bk39r3WZlkn3b5u5Icn+TM9vf5iIneZ7hfl36jqnz5GtoLeCRwObBDO7898GXgpe38y4EvttOfAD5HE1qX0DyDAmA/4Cs92zyE5qZU27fz/wy8s51+InBhT7+PzPZ34MvXoF40R80uaaf3A24BFrfzhwJvb6fvDawCFtNcYr5Nu3wHmsu307utnu39AtipXf+nwLvbtiOBY9rpzwD7ttO7AJe10+8CftCuuwPwM2CL8e/ja3ZeHlLVsD0ROK2qbgCoqhuT7A08u23/FPD+nv5frKq7gEuTPGiK7X6zqm5sp/cF/qLd/hlJHphk2xn9FNLm4dyq+nE7/WfAbj1H2LaleQrsWuDvkjweuIvm2S6T/a6trOZmbyT5EfCNdvnFwP7t9J8CS5Lf3FJ3myT3b6e/WlW3AbcluX6K99GQGQY0bGGSZ0f06G2/bdy6k7llmn5eQ6suGv978bqqOr23Q5JDgDHgMVV1R5Irga0m2V7v7+NdPfN3cff+5F7A3lX1q3HvM379O3EfNDIcM6Bh+xbwvCQPBEiyPc2hw4Pb9hcC35tmG78E7j9F+3fb7ZBkP+CGqrp5E2qWNhdT/W6cDrw6yRYASf5HkvvSHCG4vg0C+3P3w2ym+z2bzDeAwzfMJPnjTahZQ2Iq01BVc5vpvwW+k+RO4ALgCOD4JG8C1tHeanoKFwHrk/wHzbiCn49rfxdwQpKLaG5d7SP41AlV9bMk309yCfAr4L96mj9Gc37+/DR/pq8DngWcBHw5ySrgQuA/J9jW14Cv9lnGEcCx7e/ffJpwflifNX+tqt7U/yfWTPEOhJIkdZynCSRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiRJ6jjDgCRJHWcYkCSp4wwDkiR1nGFAkqSOMwxIktRxhgFJkjrOMCBJUscZBiTdI0kelOQzSa5Icl6Ss5L8+WzXJWnjGQYkbbQkAb4IfLeqHlpVjwEOBhaM6zd/NuqTtHFSVbNdg6TNTJInAe+oqidM0HYI8DRgK+C+wHOA44GHArcCh1bVRUneBfx3VX2gXe8S4OntZr4OnAM8Gvgh8JKqunWQn0nqMo8MSLonHgmcP0X73sBLq+qJwLuBC6pqN+CtwIl9bP/hwHHtOjcDr9nEeiVNwTAgaZMlOTbJfyRZ2S76ZlXd2E7vC3wKoKrOAB6YZNtpNnl1VX2/nf50uw1JA2IYkHRPrAZ23zBTVa8FngSMtYtu6embCdYvYD2//f+grca1j+8vaUAMA5LuiTOArZK8umfZ1pP0/S7wQoAk+wE3VNXNwJW0gSLJ7sDinnV2SbJ3O/184HszVrmk3+EAQkn3SJKdgH8C9gTW0RwNWAbcB1haVYe3/bYHTqDZ2fcOILwP8CXg94CVNKcCDmw3v4ImRDwO+H/Aix1AKA2OYUDSSEmyCPhKVf3hLJcidYanCSRJ6jiPDEiS1HEeGZAkqeMMA5IkdZxhQJKkjjMMSJLUcYYBSZI6zjAgSVLH/X/+unL9XRdbWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 繪出 conversion rate 棒狀圖\n",
    "plt.figure(figsize=(8,6))\n",
    "\n",
    "sns.barplot(x=ab_test['group'], y=ab_test['converted'], ci=False)\n",
    "\n",
    "plt.ylim(0, 0.17)\n",
    "plt.title('Conversion rate by group', pad=20)\n",
    "plt.xlabel('Group', labelpad=15)\n",
    "plt.ylabel('Converted (proportion)', labelpad=15);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3SJ2HsmYEm4M",
    "outputId": "f9f481d7-f3c9-49dc-c75a-ea6a5bd12fec"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z statistic: -0.34\n",
      "p-value: 0.732\n",
      "ci 95% for control group: [0.114, 0.133]\n",
      "ci 95% for treatment group: [0.116, 0.135]\n"
     ]
    }
   ],
   "source": [
    "# 以函數計算 z_stat, pval, confidence interval\n",
    "from statsmodels.stats.proportion import proportions_ztest, proportion_confint\n",
    "control_results = ab_test[ab_test['group'] == 'control']['converted']\n",
    "treatment_results = ab_test[ab_test['group'] == 'treatment']['converted']\n",
    "n_con = control_results.count()\n",
    "n_treat = treatment_results.count()\n",
    "successes = [control_results.sum(), treatment_results.sum()]\n",
    "nobs = [n_con, n_treat]\n",
    "\n",
    "z_stat, pval = proportions_ztest(successes, nobs=nobs)\n",
    "(lower_con, lower_treat), (upper_con, upper_treat) = proportion_confint(successes, nobs=nobs, alpha=0.05)\n",
    "\n",
    "# 判讀統計結果\n",
    "print(f'z statistic: {z_stat:.2f}')\n",
    "print(f'p-value: {pval:.3f}')\n",
    "print(f'ci 95% for control group: [{lower_con:.3f}, {upper_con:.3f}]')\n",
    "print(f'ci 95% for treatment group: [{lower_treat:.3f}, {upper_treat:.3f}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ckQFAPewwRJL"
   },
   "source": [
    "# 作業：判讀程式最後統計結果，A/B test 是否顯著\n",
    "\n",
    "z statistic: -0.34\n",
    "\n",
    "p-value: 0.732\n",
    "\n",
    "ci 95% for control group: [0.114, 0.133]\n",
    "\n",
    "ci 95% for treatment group: [0.116, 0.135]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ans: 不顯著，因為 $\\text{p-value} = 0.732 > 0.05$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "25S0aik5wjew"
   },
   "source": [
    "# 作業：試以(0.12, 0.11)計算結果是否顯著\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15970\n"
     ]
    }
   ],
   "source": [
    "# 求樣本大小\n",
    "effect_size = sms.proportion_effectsize(0.12, 0.11)    # Calculating effect size based on our expected rates\n",
    "\n",
    "required_n = sms.NormalIndPower().solve_power( effect_size, \n",
    "                                               power=0.8, \n",
    "                                               alpha=0.05, \n",
    "                                               ratio=1 \n",
    "                                             )         # Calculating sample size needed\n",
    "\n",
    "required_n = ceil(required_n)                          # Rounding up to next whole number                          \n",
    "\n",
    "print(required_n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>763854</td>\n",
       "      <td>2017-01-21 03:43:17.188315</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>690555</td>\n",
       "      <td>2017-01-18 06:38:13.079449</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>861520</td>\n",
       "      <td>2017-01-06 21:13:40.044766</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>630778</td>\n",
       "      <td>2017-01-05 16:42:36.995204</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>656634</td>\n",
       "      <td>2017-01-04 15:31:21.676130</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31935</th>\n",
       "      <td>780954</td>\n",
       "      <td>2017-01-19 07:49:58.295232</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31936</th>\n",
       "      <td>700881</td>\n",
       "      <td>2017-01-13 12:31:37.243352</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31937</th>\n",
       "      <td>829626</td>\n",
       "      <td>2017-01-14 09:07:57.784950</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31938</th>\n",
       "      <td>773197</td>\n",
       "      <td>2017-01-05 00:23:56.177295</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31939</th>\n",
       "      <td>720502</td>\n",
       "      <td>2017-01-15 15:54:12.049241</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>31940 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       user_id                   timestamp      group landing_page  converted\n",
       "0       763854  2017-01-21 03:43:17.188315    control     old_page          0\n",
       "1       690555  2017-01-18 06:38:13.079449    control     old_page          0\n",
       "2       861520  2017-01-06 21:13:40.044766    control     old_page          0\n",
       "3       630778  2017-01-05 16:42:36.995204    control     old_page          0\n",
       "4       656634  2017-01-04 15:31:21.676130    control     old_page          0\n",
       "...        ...                         ...        ...          ...        ...\n",
       "31935   780954  2017-01-19 07:49:58.295232  treatment     new_page          0\n",
       "31936   700881  2017-01-13 12:31:37.243352  treatment     new_page          0\n",
       "31937   829626  2017-01-14 09:07:57.784950  treatment     new_page          0\n",
       "31938   773197  2017-01-05 00:23:56.177295  treatment     new_page          0\n",
       "31939   720502  2017-01-15 15:54:12.049241  treatment     new_page          0\n",
       "\n",
       "[31940 rows x 5 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 選取 控制組和實驗組各半 15970 * 2 = 31940\n",
    "control_sample = df[df['group'] == 'control'].sample(n=required_n, random_state=22)\n",
    "treatment_sample = df[df['group'] == 'treatment'].sample(n=required_n, random_state=22)\n",
    "\n",
    "ab_test = pd.concat([control_sample, treatment_sample], axis=0)\n",
    "ab_test.reset_index(drop=True, inplace=True)\n",
    "ab_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z statistic: 0.07\n",
      "p-value: 0.945\n",
      "ci 95% for control group: [0.116, 0.126]\n",
      "ci 95% for treatment group: [0.116, 0.126]\n"
     ]
    }
   ],
   "source": [
    "# 以函數計算 z_stat, pval, confidence interval\n",
    "\n",
    "control_results = ab_test[ab_test['group'] == 'control']['converted']\n",
    "treatment_results = ab_test[ab_test['group'] == 'treatment']['converted']\n",
    "n_con = control_results.count()\n",
    "n_treat = treatment_results.count()\n",
    "successes = [control_results.sum(), treatment_results.sum()]\n",
    "nobs = [n_con, n_treat]\n",
    "\n",
    "z_stat, pval = proportions_ztest(successes, nobs=nobs)\n",
    "(lower_con, lower_treat), (upper_con, upper_treat) = proportion_confint(successes, nobs=nobs, alpha=0.05)\n",
    "\n",
    "print(f'z statistic: {z_stat:.2f}')\n",
    "print(f'p-value: {pval:.3f}')\n",
    "print(f'ci 95% for control group: [{lower_con:.3f}, {upper_con:.3f}]')\n",
    "print(f'ci 95% for treatment group: [{lower_treat:.3f}, {upper_treat:.3f}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ans: 不顯著，因為 $\\text{p-value} = 0.945 > 0.05$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sjbijdxtw_3N"
   },
   "source": [
    "# 作業：樣本數是以那些模組/函數算的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ans: statsmodels.stats.api 及 math"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "ABTestExercise_作業.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}