Academy2019_Final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Финал Академии ИИ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Доступные данные\n",
    "\n",
    "Как и в задаче он-лайн этапа, будут доступны обучающая и тестовая выборки:\n",
    "- `academy2019_final_train.csv`, `academy2019_final_test.csv`: признаки от организаторов\n",
    "- `academy2019_final_train.jsonlines`, `academy2019_final_test.jsonlines`: расширенный набор данных в формате строчек с JSON объектами\n",
    "\n",
    "Дополнительные поля, которые добавлены в JSONLines:\n",
    "\n",
    "- `kills_log`, `deaths_log`: списки убийств и смертей героя во время игры, указано время и координаты события на карте\n",
    "- `wards`: список установленных игроком \"вардов\" — специальных игровых предметов, которые дают обзор карты в некотором радиусе\n",
    "- `gold_by_reason`: информация о том, по какой причине сколько золота было получено\n",
    "- `actions`: действия мышкой за первые 10 минут игры, в колоночном формате со следующими полями:\n",
    "  - `time`: время действия\n",
    "  - `order_type`: тип действия\n",
    "  - `camera_(x,y)`: положение камеры в момент действия\n",
    "  - `cursor_(x,y)`: положение курсора на экране в момент действия\n",
    "  - `pos_(x,y)`: координаты действия на игровой карте; в случае если действие не относится к карте — (0, 0)\n",
    "- `pings`: \"пинги\" — сообщения, которые делал игрок союзникам о некоторой важной точке на карте, в колоночном формате\n",
    "\n",
    "Данные он-лайн этапа доступны могут также быть полезны для решения задачи."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Предсказание\n",
    "\n",
    "Необходимо для каждого тестового примера оценить вероятность `P(skilled = 1)`. Для этого в моделях SciKit-Learn можно использовать методы `model.predict_proba()`. \n",
    "\n",
    "Для оценки качества предсказаний будет использоваться метрика ROC-AUC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Чтение таблиц с признаками"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "\n",
    "df_train = pandas.read_csv('data/final/academy2019_final_train.csv', index_col='id')\n",
    "df_test = pandas.read_csv('data/final/academy2019_final_test.csv', index_col='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>skilled</th>\n",
       "      <th>player_team</th>\n",
       "      <th>winner_team</th>\n",
       "      <th>duration</th>\n",
       "      <th>pre_game_duration</th>\n",
       "      <th>first_blood_time</th>\n",
       "      <th>first_blood_claimed</th>\n",
       "      <th>hero_id</th>\n",
       "      <th>hero_pick_order</th>\n",
       "      <th>leaver_status</th>\n",
       "      <th>...</th>\n",
       "      <th>avg_deaths_x16</th>\n",
       "      <th>avg_assists_x16</th>\n",
       "      <th>avg_gpm_x16</th>\n",
       "      <th>avg_xpm_x16</th>\n",
       "      <th>best_kills_x16</th>\n",
       "      <th>best_assists_x16</th>\n",
       "      <th>best_gpm_x16</th>\n",
       "      <th>best_xpm_x16</th>\n",
       "      <th>win_streak</th>\n",
       "      <th>best_win_streak</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>radiant</td>\n",
       "      <td>radiant</td>\n",
       "      <td>1761</td>\n",
       "      <td>90</td>\n",
       "      <td>52</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>514</td>\n",
       "      <td>601</td>\n",
       "      <td>24</td>\n",
       "      <td>28</td>\n",
       "      <td>681</td>\n",
       "      <td>797</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>dire</td>\n",
       "      <td>dire</td>\n",
       "      <td>2609</td>\n",
       "      <td>90</td>\n",
       "      <td>108</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>412</td>\n",
       "      <td>515</td>\n",
       "      <td>23</td>\n",
       "      <td>40</td>\n",
       "      <td>624</td>\n",
       "      <td>796</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>radiant</td>\n",
       "      <td>radiant</td>\n",
       "      <td>1811</td>\n",
       "      <td>90</td>\n",
       "      <td>102</td>\n",
       "      <td>1</td>\n",
       "      <td>108</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>15</td>\n",
       "      <td>458</td>\n",
       "      <td>499</td>\n",
       "      <td>5</td>\n",
       "      <td>15</td>\n",
       "      <td>458</td>\n",
       "      <td>499</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>radiant</td>\n",
       "      <td>radiant</td>\n",
       "      <td>2668</td>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>372</td>\n",
       "      <td>483</td>\n",
       "      <td>14</td>\n",
       "      <td>27</td>\n",
       "      <td>462</td>\n",
       "      <td>650</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>dire</td>\n",
       "      <td>radiant</td>\n",
       "      <td>1716</td>\n",
       "      <td>90</td>\n",
       "      <td>52</td>\n",
       "      <td>0</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>11</td>\n",
       "      <td>15</td>\n",
       "      <td>290</td>\n",
       "      <td>387</td>\n",
       "      <td>16</td>\n",
       "      <td>37</td>\n",
       "      <td>513</td>\n",
       "      <td>744</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 57 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    skilled player_team winner_team  duration  pre_game_duration  \\\n",
       "id                                                                 \n",
       "0         0     radiant     radiant      1761                 90   \n",
       "1         0        dire        dire      2609                 90   \n",
       "2         1     radiant     radiant      1811                 90   \n",
       "3         0     radiant     radiant      2668                 90   \n",
       "7         1        dire     radiant      1716                 90   \n",
       "\n",
       "    first_blood_time  first_blood_claimed  hero_id  hero_pick_order  \\\n",
       "id                                                                    \n",
       "0                 52                    0        6                5   \n",
       "1                108                    0        7                4   \n",
       "2                102                    1      108                5   \n",
       "3                  0                    0        9                6   \n",
       "7                 52                    0       31                2   \n",
       "\n",
       "    leaver_status       ...         avg_deaths_x16  avg_assists_x16  \\\n",
       "id                      ...                                           \n",
       "0               0       ...                      7               11   \n",
       "1               0       ...                      9               14   \n",
       "2               0       ...                      7               15   \n",
       "3               0       ...                      4               18   \n",
       "7               0       ...                     11               15   \n",
       "\n",
       "    avg_gpm_x16  avg_xpm_x16  best_kills_x16  best_assists_x16  best_gpm_x16  \\\n",
       "id                                                                             \n",
       "0           514          601              24                28           681   \n",
       "1           412          515              23                40           624   \n",
       "2           458          499               5                15           458   \n",
       "3           372          483              14                27           462   \n",
       "7           290          387              16                37           513   \n",
       "\n",
       "    best_xpm_x16  win_streak  best_win_streak  \n",
       "id                                             \n",
       "0            797           5                5  \n",
       "1            796           1                5  \n",
       "2            499           1                1  \n",
       "3            650           5                5  \n",
       "7            744           0                5  \n",
       "\n",
       "[5 rows x 57 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Пример обработки расширенного набора данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "with open('data/final/academy2019_final_train.jsonlines') as fin:\n",
    "    for line in fin:\n",
    "        record = json.loads(line)\n",
    "        \n",
    "        record_id = record['id']\n",
    "        df_actions = pandas.DataFrame(record['actions'])\n",
    "        df_pings = pandas.DataFrame(record['pings'])\n",
    "        \n",
    "        # обработка записи\n",
    "        \n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>order_type</th>\n",
       "      <th>camera_x</th>\n",
       "      <th>camera_y</th>\n",
       "      <th>cursor_x</th>\n",
       "      <th>cursor_y</th>\n",
       "      <th>pos_x</th>\n",
       "      <th>pos_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-192.85</td>\n",
       "      <td>16</td>\n",
       "      <td>130</td>\n",
       "      <td>122</td>\n",
       "      <td>306</td>\n",
       "      <td>211</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-192.52</td>\n",
       "      <td>16</td>\n",
       "      <td>130</td>\n",
       "      <td>122</td>\n",
       "      <td>320</td>\n",
       "      <td>213</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-192.25</td>\n",
       "      <td>16</td>\n",
       "      <td>130</td>\n",
       "      <td>122</td>\n",
       "      <td>334</td>\n",
       "      <td>211</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-191.99</td>\n",
       "      <td>16</td>\n",
       "      <td>130</td>\n",
       "      <td>122</td>\n",
       "      <td>348</td>\n",
       "      <td>210</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-189.79</td>\n",
       "      <td>16</td>\n",
       "      <td>130</td>\n",
       "      <td>122</td>\n",
       "      <td>321</td>\n",
       "      <td>211</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     time  order_type  camera_x  camera_y  cursor_x  cursor_y  pos_x  pos_y\n",
       "0 -192.85          16       130       122       306       211      0      0\n",
       "1 -192.52          16       130       122       320       213      0      0\n",
       "2 -192.25          16       130       122       334       211      0      0\n",
       "3 -191.99          16       130       122       348       210      0      0\n",
       "4 -189.79          16       130       122       321       211      0      0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_actions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Облачные вычисления\n",
    "\n",
    "Для решения задачи можно использовать ТОЛЬКО:\n",
    "- рабочие персональные компьютеры\n",
    "- [Google Colab](https://colab.research.google.com)\n",
    "\n",
    "Данные в Colab доступны по адресам:\n",
    "- данные финала `https://s3.eu-central-1.amazonaws.com/ai-academy-2019/public/final/{filename}`\n",
    "- данные он-лайн этапа `https://s3.eu-central-1.amazonaws.com/ai-academy-2019/public/online/{filename}`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Пример чтения таблицы в Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pandas.read_csv(\n",
    "    'https://s3.eu-central-1.amazonaws.com/ai-academy-2019/public/final/academy2019_final_train.csv',\n",
    "    index_col='id',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Пример чтения jsonlines файла"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from urllib.request import urlopen\n",
    "\n",
    "with urlopen('https://s3.eu-central-1.amazonaws.com/ai-academy-2019/public/final/academy2019_final_train.jsonlines') as fin:\n",
    "    for line in fin:\n",
    "        record = json.loads(line)\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Вопросы по задачам и обсуждение\n",
    "\n",
    "Добавляйтесь в Discord чат [Party Parrot](https://discord.gg/APmbju), в канале `#ai_academy_final` можно обсуждать постановку задачи.\n",
    "\n",
    "**Помните**, что решать задачу необходимо самостоятельно. Обсуждение решения во время соревнования не допускается."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}