trading_ml_model_showcase.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook I demonstrate my pipeline for training a machine learning model for algo trading. \n",
    "\n",
    "This notebook walks you through:\n",
    "* loading bars data\n",
    "* extracting features based on technical indicators\n",
    "* labelling target variables for modelling using the triple barrier method\n",
    "* dynamically adjusting stop loss and take profit levels\n",
    "* selecting useful features using recursive feature elimination with random forests OR based on your own reasoning\n",
    "* training an XGBoost model to make buy decisions\n",
    "* backtesting and evaluating model performance\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from stonks import feature_extraction, feature_selection, backtest, labelling, models, visualisation\n",
    "import warnings\n",
    "pd.set_option('display.max_rows', 100)\n",
    "pd.set_option('display.max_columns', 100)\n",
    "warnings.simplefilter(action='ignore', category=pd.errors.PerformanceWarning)\n",
    "warnings.simplefilter(action='ignore', category=RuntimeWarning)\n",
    "warnings.simplefilter(action='ignore', category=pd.errors.SettingWithCopyWarning)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading stock price data.\n",
    "I get minute bars data from the Alpaca API. For your convenience, I am going to load data from a csv file. \n",
    "This gives you 1 minute bars for PLTR (Palantir) from 2022-05-01."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>trade_count</th>\n",
       "      <th>vwap</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2022-05-02 09:30:00+00:00</th>\n",
       "      <td>10.380</td>\n",
       "      <td>10.3900</td>\n",
       "      <td>10.2800</td>\n",
       "      <td>10.3300</td>\n",
       "      <td>583607</td>\n",
       "      <td>1561</td>\n",
       "      <td>10.333511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-05-02 09:31:00+00:00</th>\n",
       "      <td>10.325</td>\n",
       "      <td>10.3263</td>\n",
       "      <td>10.2600</td>\n",
       "      <td>10.2699</td>\n",
       "      <td>201313</td>\n",
       "      <td>731</td>\n",
       "      <td>10.280256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-05-02 09:32:00+00:00</th>\n",
       "      <td>10.260</td>\n",
       "      <td>10.2950</td>\n",
       "      <td>10.2175</td>\n",
       "      <td>10.2900</td>\n",
       "      <td>271984</td>\n",
       "      <td>1168</td>\n",
       "      <td>10.254898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-05-02 09:33:00+00:00</th>\n",
       "      <td>10.280</td>\n",
       "      <td>10.2900</td>\n",
       "      <td>10.2200</td>\n",
       "      <td>10.2300</td>\n",
       "      <td>152645</td>\n",
       "      <td>685</td>\n",
       "      <td>10.259573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2022-05-02 09:34:00+00:00</th>\n",
       "      <td>10.230</td>\n",
       "      <td>10.2850</td>\n",
       "      <td>10.1918</td>\n",
       "      <td>10.2800</td>\n",
       "      <td>311727</td>\n",
       "      <td>1016</td>\n",
       "      <td>10.237203</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             open     high      low    close  volume  \\\n",
       "timestamp                                                              \n",
       "2022-05-02 09:30:00+00:00  10.380  10.3900  10.2800  10.3300  583607   \n",
       "2022-05-02 09:31:00+00:00  10.325  10.3263  10.2600  10.2699  201313   \n",
       "2022-05-02 09:32:00+00:00  10.260  10.2950  10.2175  10.2900  271984   \n",
       "2022-05-02 09:33:00+00:00  10.280  10.2900  10.2200  10.2300  152645   \n",
       "2022-05-02 09:34:00+00:00  10.230  10.2850  10.1918  10.2800  311727   \n",
       "\n",
       "                           trade_count       vwap  \n",
       "timestamp                                          \n",
       "2022-05-02 09:30:00+00:00         1561  10.333511  \n",
       "2022-05-02 09:31:00+00:00          731  10.280256  \n",
       "2022-05-02 09:32:00+00:00         1168  10.254898  \n",
       "2022-05-02 09:33:00+00:00          685  10.259573  \n",
       "2022-05-02 09:34:00+00:00         1016  10.237203  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist = pd.read_csv('data/bars_PLTR.csv', index_col='timestamp')\n",
    "hist.index = pd.DatetimeIndex(hist.index)\n",
    "hist.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you would like to get your own minute bars, you can create a Free Alpaca account and use this code:\n",
    "# from alpaca_trade_api.rest import REST\n",
    "# from alpaca.data.timeframe import TimeFrame, TimeFrameUnit\n",
    "\n",
    "# API_KEY = \"YOURKEY\"\n",
    "# SECRET_KEY = \"YOURKEY\"\n",
    "# api = REST(key_id=API_KEY,secret_key=SECRET_KEY,base_url=\"https://paper-api.alpaca.markets\")\n",
    "# symbol = ['PLTR']\n",
    "# hist=api.get_bars(symbol,timeframe=TimeFrame(amount=1, unit=TimeFrameUnit.Minute), start='2022-05-01').df\n",
    "# hist = hist.drop(columns = ['symbol'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have bars data we can extract features that we will use to train our machine learning model.\n",
    "\n",
    "The features consist of various technical indicators such as moving averages, volatility indicators (rolling standard error, ATR), oversold/ overbought indicators etc.\n",
    "\n",
    "I create features algorithmically by iterating over different time windows e.g. macd_12_26 compares exponential moving averages over the last 12 and 26 minutes, but it could be [26,50], [100,1000], [100,9000] minutes.\n",
    "\n",
    "The features are added to the dataframe and given a descriptive feature name. For example:\n",
    "* macd_12_26 - is the moving average convergence divergence for the 12 and 26 minute windows\n",
    "* vol_std_pct_50 - is volatility measured as a rolling exponentially averaged standard deviation over 50 minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn off warnings so pandas does not print a long list of them for each column added with .loc\n",
    "# window limit ensures not too many features are generated\n",
    "hist = feature_extraction.get_features(hist, window_limit=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "      <th>trade_count</th>\n",
       "      <th>vwap</th>\n",
       "      <th>returns</th>\n",
       "      <th>volume_pct_change</th>\n",
       "      <th>minutes_open</th>\n",
       "      <th>date</th>\n",
       "      <th>datetime</th>\n",
       "      <th>market_open</th>\n",
       "      <th>close_market_open</th>\n",
       "      <th>datetime_market_open</th>\n",
       "      <th>pct_change_daily</th>\n",
       "      <th>30_min_bin</th>\n",
       "      <th>first_30_min</th>\n",
       "      <th>last_30_min</th>\n",
       "      <th>ATR_1</th>\n",
       "      <th>RSI_1</th>\n",
       "      <th>ema_1</th>\n",
       "      <th>ema_volume_1</th>\n",
       "      <th>vol_std_1</th>\n",
       "      <th>vol_std_volume_1</th>\n",
       "      <th>vol_pct_1</th>\n",
       "      <th>vol_pct_volume_1</th>\n",
       "      <th>returns_vol_std_1</th>\n",
       "      <th>swing_low_1</th>\n",
       "      <th>swing_high_1</th>\n",
       "      <th>swing_low_pct_1</th>\n",
       "      <th>swing_high_pct_1</th>\n",
       "      <th>min_vol_std_1</th>\n",
       "      <th>max_vol_std_1</th>\n",
       "      <th>close_ema_vol_std_1</th>\n",
       "      <th>close_ema_pct_1</th>\n",
       "      <th>ema_vol_std_shift_1</th>\n",
       "      <th>vol_std_pct_1</th>\n",
       "      <th>vol_std_pct_ratio_1</th>\n",
       "      <th>returns_ema_1</th>\n",
       "      <th>exp_ema_pct_shift_1</th>\n",
       "      <th>exp_ema_pct_abs_shift_1</th>\n",
       "      <th>exp_ema_pct_dir_shift_1</th>\n",
       "      <th>exp_pct_volatility_1</th>\n",
       "      <th>exp_pct_volatility_old_1</th>\n",
       "      <th>ATR_2</th>\n",
       "      <th>RSI_2</th>\n",
       "      <th>ADX_2</th>\n",
       "      <th>ema_2</th>\n",
       "      <th>ema_volume_2</th>\n",
       "      <th>...</th>\n",
       "      <th>macd_26_100_signal</th>\n",
       "      <th>volume_macd_26_100</th>\n",
       "      <th>volume_macd_26_100_signal</th>\n",
       "      <th>ema_volume_vol_std_26_100</th>\n",
       "      <th>ema_volume_vol_std_new_26_100</th>\n",
       "      <th>volume_macd_pct_26_100</th>\n",
       "      <th>ema_vol_std_shift_26_100</th>\n",
       "      <th>ema_vol_std_shift_new_26_100</th>\n",
       "      <th>macd_pct_26_100</th>\n",
       "      <th>vol_std_pct_ratio_26_100</th>\n",
       "      <th>macd_50_100</th>\n",
       "      <th>macd_50_100_signal</th>\n",
       "      <th>volume_macd_50_100</th>\n",
       "      <th>volume_macd_50_100_signal</th>\n",
       "      <th>ema_volume_vol_std_50_100</th>\n",
       "      <th>ema_volume_vol_std_new_50_100</th>\n",
       "      <th>volume_macd_pct_50_100</th>\n",
       "      <th>ema_vol_std_shift_50_100</th>\n",
       "      <th>ema_vol_std_shift_new_50_100</th>\n",
       "      <th>macd_pct_50_100</th>\n",
       "      <th>vol_std_pct_ratio_50_100</th>\n",
       "      <th>macd_50_200</th>\n",
       "      <th>macd_50_200_signal</th>\n",
       "      <th>volume_macd_50_200</th>\n",
       "      <th>volume_macd_50_200_signal</th>\n",
       "      <th>ema_volume_vol_std_50_200</th>\n",
       "      <th>ema_volume_vol_std_new_50_200</th>\n",
       "      <th>volume_macd_pct_50_200</th>\n",
       "      <th>ema_vol_std_shift_50_200</th>\n",
       "      <th>ema_vol_std_shift_new_50_200</th>\n",
       "      <th>macd_pct_50_200</th>\n",
       "      <th>vol_std_pct_ratio_50_200</th>\n",
       "      <th>macd_60_150</th>\n",
       "      <th>macd_60_150_signal</th>\n",
       "      <th>volume_macd_60_150</th>\n",
       "      <th>volume_macd_60_150_signal</th>\n",
       "      <th>ema_volume_vol_std_60_150</th>\n",
       "      <th>ema_volume_vol_std_new_60_150</th>\n",
       "      <th>volume_macd_pct_60_150</th>\n",
       "      <th>ema_vol_std_shift_60_150</th>\n",
       "      <th>ema_vol_std_shift_new_60_150</th>\n",
       "      <th>macd_pct_60_150</th>\n",
       "      <th>vol_std_pct_ratio_60_150</th>\n",
       "      <th>returns_EMA_20</th>\n",
       "      <th>returns_EMA_50</th>\n",
       "      <th>returns_EMA_80</th>\n",
       "      <th>returns_EMA_200</th>\n",
       "      <th>returns_EMA_1200</th>\n",
       "      <th>candle</th>\n",
       "      <th>candle_pct</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2023-09-14 15:56:00+00:00</th>\n",
       "      <td>15.810</td>\n",
       "      <td>15.82</td>\n",
       "      <td>15.80</td>\n",
       "      <td>15.815</td>\n",
       "      <td>135683</td>\n",
       "      <td>645</td>\n",
       "      <td>15.810052</td>\n",
       "      <td>0.000943</td>\n",
       "      <td>-0.579341</td>\n",
       "      <td>386</td>\n",
       "      <td>2023-09-14</td>\n",
       "      <td>2023-09-14 15:56:00+00:00</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>15.72</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>0.604326</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.02</td>\n",
       "      <td>100.0</td>\n",
       "      <td>15.815</td>\n",
       "      <td>135683.0</td>\n",
       "      <td>0.010112</td>\n",
       "      <td>9.985271e+04</td>\n",
       "      <td>0.063970</td>\n",
       "      <td>54.453957</td>\n",
       "      <td>0.000707</td>\n",
       "      <td>15.815</td>\n",
       "      <td>15.815</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.473484</td>\n",
       "      <td>0.000639</td>\n",
       "      <td>1.652933</td>\n",
       "      <td>0.000473</td>\n",
       "      <td>0.000040</td>\n",
       "      <td>0.000553</td>\n",
       "      <td>-0.000040</td>\n",
       "      <td>0.000621</td>\n",
       "      <td>0.000642</td>\n",
       "      <td>0.024013</td>\n",
       "      <td>84.758237</td>\n",
       "      <td>69.780324</td>\n",
       "      <td>15.809802</td>\n",
       "      <td>1.782907e+05</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>39829.602996</td>\n",
       "      <td>1</td>\n",
       "      <td>0.427968</td>\n",
       "      <td>0.528489</td>\n",
       "      <td>32.749058</td>\n",
       "      <td>0.430885</td>\n",
       "      <td>0.834881</td>\n",
       "      <td>0.141242</td>\n",
       "      <td>0.515374</td>\n",
       "      <td>0.013499</td>\n",
       "      <td>1</td>\n",
       "      <td>19398.426918</td>\n",
       "      <td>1</td>\n",
       "      <td>0.208436</td>\n",
       "      <td>0.274531</td>\n",
       "      <td>19.170408</td>\n",
       "      <td>0.260799</td>\n",
       "      <td>0.382620</td>\n",
       "      <td>0.085537</td>\n",
       "      <td>0.681032</td>\n",
       "      <td>0.028604</td>\n",
       "      <td>1</td>\n",
       "      <td>29718.365237</td>\n",
       "      <td>1</td>\n",
       "      <td>0.208328</td>\n",
       "      <td>0.420582</td>\n",
       "      <td>29.369041</td>\n",
       "      <td>0.380522</td>\n",
       "      <td>0.810763</td>\n",
       "      <td>0.181250</td>\n",
       "      <td>0.468487</td>\n",
       "      <td>0.019220</td>\n",
       "      <td>1</td>\n",
       "      <td>21890.650486</td>\n",
       "      <td>1</td>\n",
       "      <td>0.177078</td>\n",
       "      <td>0.306144</td>\n",
       "      <td>22.847503</td>\n",
       "      <td>0.293635</td>\n",
       "      <td>0.499343</td>\n",
       "      <td>0.121814</td>\n",
       "      <td>0.587327</td>\n",
       "      <td>0.000097</td>\n",
       "      <td>6.004226e-05</td>\n",
       "      <td>0.000046</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.031626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-09-14 15:57:00+00:00</th>\n",
       "      <td>15.810</td>\n",
       "      <td>15.83</td>\n",
       "      <td>15.81</td>\n",
       "      <td>15.815</td>\n",
       "      <td>337571</td>\n",
       "      <td>1019</td>\n",
       "      <td>15.815863</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.487939</td>\n",
       "      <td>387</td>\n",
       "      <td>2023-09-14</td>\n",
       "      <td>2023-09-14 15:57:00+00:00</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>15.72</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>0.604326</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.02</td>\n",
       "      <td>100.0</td>\n",
       "      <td>15.815</td>\n",
       "      <td>337571.0</td>\n",
       "      <td>0.008487</td>\n",
       "      <td>1.179064e+05</td>\n",
       "      <td>0.053674</td>\n",
       "      <td>45.266605</td>\n",
       "      <td>0.000578</td>\n",
       "      <td>15.815</td>\n",
       "      <td>15.815</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000537</td>\n",
       "      <td>0.839247</td>\n",
       "      <td>0.000236</td>\n",
       "      <td>0.000039</td>\n",
       "      <td>0.000547</td>\n",
       "      <td>-0.000039</td>\n",
       "      <td>0.000620</td>\n",
       "      <td>0.000539</td>\n",
       "      <td>0.022006</td>\n",
       "      <td>84.758237</td>\n",
       "      <td>79.037335</td>\n",
       "      <td>15.813267</td>\n",
       "      <td>2.844776e+05</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>50760.978045</td>\n",
       "      <td>1</td>\n",
       "      <td>0.527122</td>\n",
       "      <td>0.584948</td>\n",
       "      <td>36.885707</td>\n",
       "      <td>0.447240</td>\n",
       "      <td>0.858515</td>\n",
       "      <td>0.146903</td>\n",
       "      <td>0.520181</td>\n",
       "      <td>0.013881</td>\n",
       "      <td>1</td>\n",
       "      <td>23603.341977</td>\n",
       "      <td>1</td>\n",
       "      <td>0.245106</td>\n",
       "      <td>0.300549</td>\n",
       "      <td>21.368362</td>\n",
       "      <td>0.267608</td>\n",
       "      <td>0.390516</td>\n",
       "      <td>0.087952</td>\n",
       "      <td>0.684665</td>\n",
       "      <td>0.029299</td>\n",
       "      <td>1</td>\n",
       "      <td>36340.470182</td>\n",
       "      <td>1</td>\n",
       "      <td>0.253245</td>\n",
       "      <td>0.462735</td>\n",
       "      <td>32.899422</td>\n",
       "      <td>0.389648</td>\n",
       "      <td>0.824248</td>\n",
       "      <td>0.185636</td>\n",
       "      <td>0.471854</td>\n",
       "      <td>0.019680</td>\n",
       "      <td>1</td>\n",
       "      <td>26325.126668</td>\n",
       "      <td>1</td>\n",
       "      <td>0.210460</td>\n",
       "      <td>0.336275</td>\n",
       "      <td>25.376422</td>\n",
       "      <td>0.300452</td>\n",
       "      <td>0.508171</td>\n",
       "      <td>0.124718</td>\n",
       "      <td>0.590504</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>5.886496e-05</td>\n",
       "      <td>0.000045</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.031626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-09-14 15:58:00+00:00</th>\n",
       "      <td>15.815</td>\n",
       "      <td>15.82</td>\n",
       "      <td>15.81</td>\n",
       "      <td>15.815</td>\n",
       "      <td>133852</td>\n",
       "      <td>586</td>\n",
       "      <td>15.814584</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.603485</td>\n",
       "      <td>388</td>\n",
       "      <td>2023-09-14</td>\n",
       "      <td>2023-09-14 15:58:00+00:00</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>15.72</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>0.604326</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>100.0</td>\n",
       "      <td>15.815</td>\n",
       "      <td>133852.0</td>\n",
       "      <td>0.006421</td>\n",
       "      <td>1.138556e+05</td>\n",
       "      <td>0.040608</td>\n",
       "      <td>57.747367</td>\n",
       "      <td>0.000433</td>\n",
       "      <td>15.815</td>\n",
       "      <td>15.815</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000406</td>\n",
       "      <td>0.756656</td>\n",
       "      <td>0.000118</td>\n",
       "      <td>0.000039</td>\n",
       "      <td>0.000542</td>\n",
       "      <td>-0.000039</td>\n",
       "      <td>0.000618</td>\n",
       "      <td>0.000408</td>\n",
       "      <td>0.016003</td>\n",
       "      <td>84.758237</td>\n",
       "      <td>83.665841</td>\n",
       "      <td>15.814422</td>\n",
       "      <td>1.840605e+05</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>49551.479452</td>\n",
       "      <td>1</td>\n",
       "      <td>0.516140</td>\n",
       "      <td>0.580937</td>\n",
       "      <td>36.079936</td>\n",
       "      <td>0.461353</td>\n",
       "      <td>0.879658</td>\n",
       "      <td>0.151823</td>\n",
       "      <td>0.523673</td>\n",
       "      <td>0.014230</td>\n",
       "      <td>1</td>\n",
       "      <td>23590.088159</td>\n",
       "      <td>1</td>\n",
       "      <td>0.245720</td>\n",
       "      <td>0.302286</td>\n",
       "      <td>21.180460</td>\n",
       "      <td>0.273796</td>\n",
       "      <td>0.397712</td>\n",
       "      <td>0.090157</td>\n",
       "      <td>0.687809</td>\n",
       "      <td>0.029948</td>\n",
       "      <td>1</td>\n",
       "      <td>36663.471200</td>\n",
       "      <td>1</td>\n",
       "      <td>0.256029</td>\n",
       "      <td>0.469810</td>\n",
       "      <td>32.918452</td>\n",
       "      <td>0.398177</td>\n",
       "      <td>0.836981</td>\n",
       "      <td>0.189735</td>\n",
       "      <td>0.474827</td>\n",
       "      <td>0.020110</td>\n",
       "      <td>1</td>\n",
       "      <td>26564.922521</td>\n",
       "      <td>1</td>\n",
       "      <td>0.212922</td>\n",
       "      <td>0.340911</td>\n",
       "      <td>25.366155</td>\n",
       "      <td>0.306827</td>\n",
       "      <td>0.516440</td>\n",
       "      <td>0.127434</td>\n",
       "      <td>0.593361</td>\n",
       "      <td>0.000088</td>\n",
       "      <td>5.771075e-05</td>\n",
       "      <td>0.000045</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-09-14 15:59:00+00:00</th>\n",
       "      <td>15.815</td>\n",
       "      <td>15.84</td>\n",
       "      <td>15.81</td>\n",
       "      <td>15.825</td>\n",
       "      <td>634831</td>\n",
       "      <td>1728</td>\n",
       "      <td>15.826510</td>\n",
       "      <td>0.000632</td>\n",
       "      <td>3.742783</td>\n",
       "      <td>389</td>\n",
       "      <td>2023-09-14</td>\n",
       "      <td>2023-09-14 15:59:00+00:00</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>15.72</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>0.667939</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.03</td>\n",
       "      <td>100.0</td>\n",
       "      <td>15.825</td>\n",
       "      <td>634831.0</td>\n",
       "      <td>0.008568</td>\n",
       "      <td>2.798473e+05</td>\n",
       "      <td>0.054166</td>\n",
       "      <td>67.271600</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>15.825</td>\n",
       "      <td>15.825</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.167069</td>\n",
       "      <td>0.000541</td>\n",
       "      <td>1.333522</td>\n",
       "      <td>0.000375</td>\n",
       "      <td>0.000045</td>\n",
       "      <td>0.000543</td>\n",
       "      <td>-0.000045</td>\n",
       "      <td>0.000617</td>\n",
       "      <td>0.000544</td>\n",
       "      <td>0.023002</td>\n",
       "      <td>96.774374</td>\n",
       "      <td>90.850191</td>\n",
       "      <td>15.821474</td>\n",
       "      <td>4.845742e+05</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>75570.246253</td>\n",
       "      <td>1</td>\n",
       "      <td>0.683712</td>\n",
       "      <td>0.578776</td>\n",
       "      <td>43.383958</td>\n",
       "      <td>0.482695</td>\n",
       "      <td>0.902452</td>\n",
       "      <td>0.159493</td>\n",
       "      <td>0.534017</td>\n",
       "      <td>0.014742</td>\n",
       "      <td>1</td>\n",
       "      <td>33285.146473</td>\n",
       "      <td>1</td>\n",
       "      <td>0.301143</td>\n",
       "      <td>0.306582</td>\n",
       "      <td>25.234319</td>\n",
       "      <td>0.282454</td>\n",
       "      <td>0.406599</td>\n",
       "      <td>0.093391</td>\n",
       "      <td>0.694027</td>\n",
       "      <td>0.030847</td>\n",
       "      <td>1</td>\n",
       "      <td>51617.780930</td>\n",
       "      <td>1</td>\n",
       "      <td>0.348370</td>\n",
       "      <td>0.475441</td>\n",
       "      <td>39.132757</td>\n",
       "      <td>0.409727</td>\n",
       "      <td>0.850752</td>\n",
       "      <td>0.195408</td>\n",
       "      <td>0.480664</td>\n",
       "      <td>0.020707</td>\n",
       "      <td>1</td>\n",
       "      <td>36572.305613</td>\n",
       "      <td>1</td>\n",
       "      <td>0.276173</td>\n",
       "      <td>0.349819</td>\n",
       "      <td>29.951190</td>\n",
       "      <td>0.315391</td>\n",
       "      <td>0.526146</td>\n",
       "      <td>0.131206</td>\n",
       "      <td>0.598650</td>\n",
       "      <td>0.000114</td>\n",
       "      <td>6.897742e-05</td>\n",
       "      <td>0.000052</td>\n",
       "      <td>0.000029</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.063231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2023-09-14 16:00:00+00:00</th>\n",
       "      <td>15.830</td>\n",
       "      <td>15.83</td>\n",
       "      <td>15.77</td>\n",
       "      <td>15.770</td>\n",
       "      <td>4225103</td>\n",
       "      <td>48</td>\n",
       "      <td>15.829994</td>\n",
       "      <td>-0.003476</td>\n",
       "      <td>5.655477</td>\n",
       "      <td>390</td>\n",
       "      <td>2023-09-14</td>\n",
       "      <td>2023-09-14 16:00:00+00:00</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>15.72</td>\n",
       "      <td>2023-09-14 09:30:00+00:00</td>\n",
       "      <td>0.318066</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.770</td>\n",
       "      <td>4225103.0</td>\n",
       "      <td>0.030652</td>\n",
       "      <td>2.340970e+06</td>\n",
       "      <td>0.194067</td>\n",
       "      <td>100.880000</td>\n",
       "      <td>0.002378</td>\n",
       "      <td>15.770</td>\n",
       "      <td>15.770</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.794335</td>\n",
       "      <td>0.001944</td>\n",
       "      <td>3.589777</td>\n",
       "      <td>-0.001550</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>0.000572</td>\n",
       "      <td>-0.000010</td>\n",
       "      <td>0.000630</td>\n",
       "      <td>0.001946</td>\n",
       "      <td>0.041501</td>\n",
       "      <td>10.005558</td>\n",
       "      <td>72.548767</td>\n",
       "      <td>15.787158</td>\n",
       "      <td>2.978260e+06</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>293925.370142</td>\n",
       "      <td>1</td>\n",
       "      <td>0.690200</td>\n",
       "      <td>0.371371</td>\n",
       "      <td>61.975962</td>\n",
       "      <td>0.447960</td>\n",
       "      <td>0.845208</td>\n",
       "      <td>0.147362</td>\n",
       "      <td>0.529219</td>\n",
       "      <td>0.014145</td>\n",
       "      <td>1</td>\n",
       "      <td>112090.191571</td>\n",
       "      <td>1</td>\n",
       "      <td>0.263212</td>\n",
       "      <td>0.191607</td>\n",
       "      <td>38.331671</td>\n",
       "      <td>0.272251</td>\n",
       "      <td>0.394012</td>\n",
       "      <td>0.089612</td>\n",
       "      <td>0.690352</td>\n",
       "      <td>0.030080</td>\n",
       "      <td>1</td>\n",
       "      <td>170893.422629</td>\n",
       "      <td>1</td>\n",
       "      <td>0.522036</td>\n",
       "      <td>0.292126</td>\n",
       "      <td>58.440711</td>\n",
       "      <td>0.400342</td>\n",
       "      <td>0.837852</td>\n",
       "      <td>0.190557</td>\n",
       "      <td>0.476909</td>\n",
       "      <td>0.020193</td>\n",
       "      <td>1</td>\n",
       "      <td>116268.058032</td>\n",
       "      <td>1</td>\n",
       "      <td>0.321313</td>\n",
       "      <td>0.216451</td>\n",
       "      <td>45.305569</td>\n",
       "      <td>0.308433</td>\n",
       "      <td>0.517314</td>\n",
       "      <td>0.127953</td>\n",
       "      <td>0.595458</td>\n",
       "      <td>-0.000057</td>\n",
       "      <td>-5.224009e-07</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>0.000011</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>-0.060</td>\n",
       "      <td>-0.379027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 668 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                             open   high    low   close   volume  trade_count  \\\n",
       "timestamp                                                                       \n",
       "2023-09-14 15:56:00+00:00  15.810  15.82  15.80  15.815   135683          645   \n",
       "2023-09-14 15:57:00+00:00  15.810  15.83  15.81  15.815   337571         1019   \n",
       "2023-09-14 15:58:00+00:00  15.815  15.82  15.81  15.815   133852          586   \n",
       "2023-09-14 15:59:00+00:00  15.815  15.84  15.81  15.825   634831         1728   \n",
       "2023-09-14 16:00:00+00:00  15.830  15.83  15.77  15.770  4225103           48   \n",
       "\n",
       "                                vwap   returns  volume_pct_change  \\\n",
       "timestamp                                                           \n",
       "2023-09-14 15:56:00+00:00  15.810052  0.000943          -0.579341   \n",
       "2023-09-14 15:57:00+00:00  15.815863  0.000000           1.487939   \n",
       "2023-09-14 15:58:00+00:00  15.814584  0.000000          -0.603485   \n",
       "2023-09-14 15:59:00+00:00  15.826510  0.000632           3.742783   \n",
       "2023-09-14 16:00:00+00:00  15.829994 -0.003476           5.655477   \n",
       "\n",
       "                           minutes_open        date                  datetime  \\\n",
       "timestamp                                                                       \n",
       "2023-09-14 15:56:00+00:00           386  2023-09-14 2023-09-14 15:56:00+00:00   \n",
       "2023-09-14 15:57:00+00:00           387  2023-09-14 2023-09-14 15:57:00+00:00   \n",
       "2023-09-14 15:58:00+00:00           388  2023-09-14 2023-09-14 15:58:00+00:00   \n",
       "2023-09-14 15:59:00+00:00           389  2023-09-14 2023-09-14 15:59:00+00:00   \n",
       "2023-09-14 16:00:00+00:00           390  2023-09-14 2023-09-14 16:00:00+00:00   \n",
       "\n",
       "                                        market_open  close_market_open  \\\n",
       "timestamp                                                                \n",
       "2023-09-14 15:56:00+00:00 2023-09-14 09:30:00+00:00              15.72   \n",
       "2023-09-14 15:57:00+00:00 2023-09-14 09:30:00+00:00              15.72   \n",
       "2023-09-14 15:58:00+00:00 2023-09-14 09:30:00+00:00              15.72   \n",
       "2023-09-14 15:59:00+00:00 2023-09-14 09:30:00+00:00              15.72   \n",
       "2023-09-14 16:00:00+00:00 2023-09-14 09:30:00+00:00              15.72   \n",
       "\n",
       "                               datetime_market_open  pct_change_daily  \\\n",
       "timestamp                                                               \n",
       "2023-09-14 15:56:00+00:00 2023-09-14 09:30:00+00:00          0.604326   \n",
       "2023-09-14 15:57:00+00:00 2023-09-14 09:30:00+00:00          0.604326   \n",
       "2023-09-14 15:58:00+00:00 2023-09-14 09:30:00+00:00          0.604326   \n",
       "2023-09-14 15:59:00+00:00 2023-09-14 09:30:00+00:00          0.667939   \n",
       "2023-09-14 16:00:00+00:00 2023-09-14 09:30:00+00:00          0.318066   \n",
       "\n",
       "                           30_min_bin  first_30_min  last_30_min  ATR_1  \\\n",
       "timestamp                                                                 \n",
       "2023-09-14 15:56:00+00:00          12             0            1   0.02   \n",
       "2023-09-14 15:57:00+00:00          12             0            1   0.02   \n",
       "2023-09-14 15:58:00+00:00          12             0            1   0.01   \n",
       "2023-09-14 15:59:00+00:00          12             0            1   0.03   \n",
       "2023-09-14 16:00:00+00:00          12             0            1   0.06   \n",
       "\n",
       "                           RSI_1   ema_1  ema_volume_1  vol_std_1  \\\n",
       "timestamp                                                           \n",
       "2023-09-14 15:56:00+00:00  100.0  15.815      135683.0   0.010112   \n",
       "2023-09-14 15:57:00+00:00  100.0  15.815      337571.0   0.008487   \n",
       "2023-09-14 15:58:00+00:00  100.0  15.815      133852.0   0.006421   \n",
       "2023-09-14 15:59:00+00:00  100.0  15.825      634831.0   0.008568   \n",
       "2023-09-14 16:00:00+00:00    0.0  15.770     4225103.0   0.030652   \n",
       "\n",
       "                           vol_std_volume_1  vol_pct_1  vol_pct_volume_1  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00      9.985271e+04   0.063970         54.453957   \n",
       "2023-09-14 15:57:00+00:00      1.179064e+05   0.053674         45.266605   \n",
       "2023-09-14 15:58:00+00:00      1.138556e+05   0.040608         57.747367   \n",
       "2023-09-14 15:59:00+00:00      2.798473e+05   0.054166         67.271600   \n",
       "2023-09-14 16:00:00+00:00      2.340970e+06   0.194067        100.880000   \n",
       "\n",
       "                           returns_vol_std_1  swing_low_1  swing_high_1  \\\n",
       "timestamp                                                                 \n",
       "2023-09-14 15:56:00+00:00           0.000707       15.815        15.815   \n",
       "2023-09-14 15:57:00+00:00           0.000578       15.815        15.815   \n",
       "2023-09-14 15:58:00+00:00           0.000433       15.815        15.815   \n",
       "2023-09-14 15:59:00+00:00           0.000439       15.825        15.825   \n",
       "2023-09-14 16:00:00+00:00           0.002378       15.770        15.770   \n",
       "\n",
       "                           swing_low_pct_1  swing_high_pct_1  min_vol_std_1  \\\n",
       "timestamp                                                                     \n",
       "2023-09-14 15:56:00+00:00            100.0             100.0            0.0   \n",
       "2023-09-14 15:57:00+00:00            100.0             100.0            0.0   \n",
       "2023-09-14 15:58:00+00:00            100.0             100.0            0.0   \n",
       "2023-09-14 15:59:00+00:00            100.0             100.0            0.0   \n",
       "2023-09-14 16:00:00+00:00            100.0             100.0            0.0   \n",
       "\n",
       "                           max_vol_std_1  close_ema_vol_std_1  \\\n",
       "timestamp                                                       \n",
       "2023-09-14 15:56:00+00:00            0.0                  0.0   \n",
       "2023-09-14 15:57:00+00:00            0.0                  0.0   \n",
       "2023-09-14 15:58:00+00:00            0.0                  0.0   \n",
       "2023-09-14 15:59:00+00:00            0.0                  0.0   \n",
       "2023-09-14 16:00:00+00:00            0.0                  0.0   \n",
       "\n",
       "                           close_ema_pct_1  ema_vol_std_shift_1  \\\n",
       "timestamp                                                         \n",
       "2023-09-14 15:56:00+00:00              0.0             1.473484   \n",
       "2023-09-14 15:57:00+00:00              0.0             0.000000   \n",
       "2023-09-14 15:58:00+00:00              0.0             0.000000   \n",
       "2023-09-14 15:59:00+00:00              0.0             1.167069   \n",
       "2023-09-14 16:00:00+00:00              0.0            -1.794335   \n",
       "\n",
       "                           vol_std_pct_1  vol_std_pct_ratio_1  returns_ema_1  \\\n",
       "timestamp                                                                      \n",
       "2023-09-14 15:56:00+00:00       0.000639             1.652933       0.000473   \n",
       "2023-09-14 15:57:00+00:00       0.000537             0.839247       0.000236   \n",
       "2023-09-14 15:58:00+00:00       0.000406             0.756656       0.000118   \n",
       "2023-09-14 15:59:00+00:00       0.000541             1.333522       0.000375   \n",
       "2023-09-14 16:00:00+00:00       0.001944             3.589777      -0.001550   \n",
       "\n",
       "                           exp_ema_pct_shift_1  exp_ema_pct_abs_shift_1  \\\n",
       "timestamp                                                                 \n",
       "2023-09-14 15:56:00+00:00             0.000040                 0.000553   \n",
       "2023-09-14 15:57:00+00:00             0.000039                 0.000547   \n",
       "2023-09-14 15:58:00+00:00             0.000039                 0.000542   \n",
       "2023-09-14 15:59:00+00:00             0.000045                 0.000543   \n",
       "2023-09-14 16:00:00+00:00             0.000010                 0.000572   \n",
       "\n",
       "                           exp_ema_pct_dir_shift_1  exp_pct_volatility_1  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00                -0.000040              0.000621   \n",
       "2023-09-14 15:57:00+00:00                -0.000039              0.000620   \n",
       "2023-09-14 15:58:00+00:00                -0.000039              0.000618   \n",
       "2023-09-14 15:59:00+00:00                -0.000045              0.000617   \n",
       "2023-09-14 16:00:00+00:00                -0.000010              0.000630   \n",
       "\n",
       "                           exp_pct_volatility_old_1     ATR_2      RSI_2  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00                  0.000642  0.024013  84.758237   \n",
       "2023-09-14 15:57:00+00:00                  0.000539  0.022006  84.758237   \n",
       "2023-09-14 15:58:00+00:00                  0.000408  0.016003  84.758237   \n",
       "2023-09-14 15:59:00+00:00                  0.000544  0.023002  96.774374   \n",
       "2023-09-14 16:00:00+00:00                  0.001946  0.041501  10.005558   \n",
       "\n",
       "                               ADX_2      ema_2  ema_volume_2  ...  \\\n",
       "timestamp                                                      ...   \n",
       "2023-09-14 15:56:00+00:00  69.780324  15.809802  1.782907e+05  ...   \n",
       "2023-09-14 15:57:00+00:00  79.037335  15.813267  2.844776e+05  ...   \n",
       "2023-09-14 15:58:00+00:00  83.665841  15.814422  1.840605e+05  ...   \n",
       "2023-09-14 15:59:00+00:00  90.850191  15.821474  4.845742e+05  ...   \n",
       "2023-09-14 16:00:00+00:00  72.548767  15.787158  2.978260e+06  ...   \n",
       "\n",
       "                           macd_26_100_signal  volume_macd_26_100  \\\n",
       "timestamp                                                           \n",
       "2023-09-14 15:56:00+00:00                   1        39829.602996   \n",
       "2023-09-14 15:57:00+00:00                   1        50760.978045   \n",
       "2023-09-14 15:58:00+00:00                   1        49551.479452   \n",
       "2023-09-14 15:59:00+00:00                   1        75570.246253   \n",
       "2023-09-14 16:00:00+00:00                   1       293925.370142   \n",
       "\n",
       "                           volume_macd_26_100_signal  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                          1   \n",
       "2023-09-14 15:57:00+00:00                          1   \n",
       "2023-09-14 15:58:00+00:00                          1   \n",
       "2023-09-14 15:59:00+00:00                          1   \n",
       "2023-09-14 16:00:00+00:00                          1   \n",
       "\n",
       "                           ema_volume_vol_std_26_100  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                   0.427968   \n",
       "2023-09-14 15:57:00+00:00                   0.527122   \n",
       "2023-09-14 15:58:00+00:00                   0.516140   \n",
       "2023-09-14 15:59:00+00:00                   0.683712   \n",
       "2023-09-14 16:00:00+00:00                   0.690200   \n",
       "\n",
       "                           ema_volume_vol_std_new_26_100  \\\n",
       "timestamp                                                  \n",
       "2023-09-14 15:56:00+00:00                       0.528489   \n",
       "2023-09-14 15:57:00+00:00                       0.584948   \n",
       "2023-09-14 15:58:00+00:00                       0.580937   \n",
       "2023-09-14 15:59:00+00:00                       0.578776   \n",
       "2023-09-14 16:00:00+00:00                       0.371371   \n",
       "\n",
       "                           volume_macd_pct_26_100  ema_vol_std_shift_26_100  \\\n",
       "timestamp                                                                     \n",
       "2023-09-14 15:56:00+00:00               32.749058                  0.430885   \n",
       "2023-09-14 15:57:00+00:00               36.885707                  0.447240   \n",
       "2023-09-14 15:58:00+00:00               36.079936                  0.461353   \n",
       "2023-09-14 15:59:00+00:00               43.383958                  0.482695   \n",
       "2023-09-14 16:00:00+00:00               61.975962                  0.447960   \n",
       "\n",
       "                           ema_vol_std_shift_new_26_100  macd_pct_26_100  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00                      0.834881         0.141242   \n",
       "2023-09-14 15:57:00+00:00                      0.858515         0.146903   \n",
       "2023-09-14 15:58:00+00:00                      0.879658         0.151823   \n",
       "2023-09-14 15:59:00+00:00                      0.902452         0.159493   \n",
       "2023-09-14 16:00:00+00:00                      0.845208         0.147362   \n",
       "\n",
       "                           vol_std_pct_ratio_26_100  macd_50_100  \\\n",
       "timestamp                                                          \n",
       "2023-09-14 15:56:00+00:00                  0.515374     0.013499   \n",
       "2023-09-14 15:57:00+00:00                  0.520181     0.013881   \n",
       "2023-09-14 15:58:00+00:00                  0.523673     0.014230   \n",
       "2023-09-14 15:59:00+00:00                  0.534017     0.014742   \n",
       "2023-09-14 16:00:00+00:00                  0.529219     0.014145   \n",
       "\n",
       "                           macd_50_100_signal  volume_macd_50_100  \\\n",
       "timestamp                                                           \n",
       "2023-09-14 15:56:00+00:00                   1        19398.426918   \n",
       "2023-09-14 15:57:00+00:00                   1        23603.341977   \n",
       "2023-09-14 15:58:00+00:00                   1        23590.088159   \n",
       "2023-09-14 15:59:00+00:00                   1        33285.146473   \n",
       "2023-09-14 16:00:00+00:00                   1       112090.191571   \n",
       "\n",
       "                           volume_macd_50_100_signal  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                          1   \n",
       "2023-09-14 15:57:00+00:00                          1   \n",
       "2023-09-14 15:58:00+00:00                          1   \n",
       "2023-09-14 15:59:00+00:00                          1   \n",
       "2023-09-14 16:00:00+00:00                          1   \n",
       "\n",
       "                           ema_volume_vol_std_50_100  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                   0.208436   \n",
       "2023-09-14 15:57:00+00:00                   0.245106   \n",
       "2023-09-14 15:58:00+00:00                   0.245720   \n",
       "2023-09-14 15:59:00+00:00                   0.301143   \n",
       "2023-09-14 16:00:00+00:00                   0.263212   \n",
       "\n",
       "                           ema_volume_vol_std_new_50_100  \\\n",
       "timestamp                                                  \n",
       "2023-09-14 15:56:00+00:00                       0.274531   \n",
       "2023-09-14 15:57:00+00:00                       0.300549   \n",
       "2023-09-14 15:58:00+00:00                       0.302286   \n",
       "2023-09-14 15:59:00+00:00                       0.306582   \n",
       "2023-09-14 16:00:00+00:00                       0.191607   \n",
       "\n",
       "                           volume_macd_pct_50_100  ema_vol_std_shift_50_100  \\\n",
       "timestamp                                                                     \n",
       "2023-09-14 15:56:00+00:00               19.170408                  0.260799   \n",
       "2023-09-14 15:57:00+00:00               21.368362                  0.267608   \n",
       "2023-09-14 15:58:00+00:00               21.180460                  0.273796   \n",
       "2023-09-14 15:59:00+00:00               25.234319                  0.282454   \n",
       "2023-09-14 16:00:00+00:00               38.331671                  0.272251   \n",
       "\n",
       "                           ema_vol_std_shift_new_50_100  macd_pct_50_100  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00                      0.382620         0.085537   \n",
       "2023-09-14 15:57:00+00:00                      0.390516         0.087952   \n",
       "2023-09-14 15:58:00+00:00                      0.397712         0.090157   \n",
       "2023-09-14 15:59:00+00:00                      0.406599         0.093391   \n",
       "2023-09-14 16:00:00+00:00                      0.394012         0.089612   \n",
       "\n",
       "                           vol_std_pct_ratio_50_100  macd_50_200  \\\n",
       "timestamp                                                          \n",
       "2023-09-14 15:56:00+00:00                  0.681032     0.028604   \n",
       "2023-09-14 15:57:00+00:00                  0.684665     0.029299   \n",
       "2023-09-14 15:58:00+00:00                  0.687809     0.029948   \n",
       "2023-09-14 15:59:00+00:00                  0.694027     0.030847   \n",
       "2023-09-14 16:00:00+00:00                  0.690352     0.030080   \n",
       "\n",
       "                           macd_50_200_signal  volume_macd_50_200  \\\n",
       "timestamp                                                           \n",
       "2023-09-14 15:56:00+00:00                   1        29718.365237   \n",
       "2023-09-14 15:57:00+00:00                   1        36340.470182   \n",
       "2023-09-14 15:58:00+00:00                   1        36663.471200   \n",
       "2023-09-14 15:59:00+00:00                   1        51617.780930   \n",
       "2023-09-14 16:00:00+00:00                   1       170893.422629   \n",
       "\n",
       "                           volume_macd_50_200_signal  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                          1   \n",
       "2023-09-14 15:57:00+00:00                          1   \n",
       "2023-09-14 15:58:00+00:00                          1   \n",
       "2023-09-14 15:59:00+00:00                          1   \n",
       "2023-09-14 16:00:00+00:00                          1   \n",
       "\n",
       "                           ema_volume_vol_std_50_200  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                   0.208328   \n",
       "2023-09-14 15:57:00+00:00                   0.253245   \n",
       "2023-09-14 15:58:00+00:00                   0.256029   \n",
       "2023-09-14 15:59:00+00:00                   0.348370   \n",
       "2023-09-14 16:00:00+00:00                   0.522036   \n",
       "\n",
       "                           ema_volume_vol_std_new_50_200  \\\n",
       "timestamp                                                  \n",
       "2023-09-14 15:56:00+00:00                       0.420582   \n",
       "2023-09-14 15:57:00+00:00                       0.462735   \n",
       "2023-09-14 15:58:00+00:00                       0.469810   \n",
       "2023-09-14 15:59:00+00:00                       0.475441   \n",
       "2023-09-14 16:00:00+00:00                       0.292126   \n",
       "\n",
       "                           volume_macd_pct_50_200  ema_vol_std_shift_50_200  \\\n",
       "timestamp                                                                     \n",
       "2023-09-14 15:56:00+00:00               29.369041                  0.380522   \n",
       "2023-09-14 15:57:00+00:00               32.899422                  0.389648   \n",
       "2023-09-14 15:58:00+00:00               32.918452                  0.398177   \n",
       "2023-09-14 15:59:00+00:00               39.132757                  0.409727   \n",
       "2023-09-14 16:00:00+00:00               58.440711                  0.400342   \n",
       "\n",
       "                           ema_vol_std_shift_new_50_200  macd_pct_50_200  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00                      0.810763         0.181250   \n",
       "2023-09-14 15:57:00+00:00                      0.824248         0.185636   \n",
       "2023-09-14 15:58:00+00:00                      0.836981         0.189735   \n",
       "2023-09-14 15:59:00+00:00                      0.850752         0.195408   \n",
       "2023-09-14 16:00:00+00:00                      0.837852         0.190557   \n",
       "\n",
       "                           vol_std_pct_ratio_50_200  macd_60_150  \\\n",
       "timestamp                                                          \n",
       "2023-09-14 15:56:00+00:00                  0.468487     0.019220   \n",
       "2023-09-14 15:57:00+00:00                  0.471854     0.019680   \n",
       "2023-09-14 15:58:00+00:00                  0.474827     0.020110   \n",
       "2023-09-14 15:59:00+00:00                  0.480664     0.020707   \n",
       "2023-09-14 16:00:00+00:00                  0.476909     0.020193   \n",
       "\n",
       "                           macd_60_150_signal  volume_macd_60_150  \\\n",
       "timestamp                                                           \n",
       "2023-09-14 15:56:00+00:00                   1        21890.650486   \n",
       "2023-09-14 15:57:00+00:00                   1        26325.126668   \n",
       "2023-09-14 15:58:00+00:00                   1        26564.922521   \n",
       "2023-09-14 15:59:00+00:00                   1        36572.305613   \n",
       "2023-09-14 16:00:00+00:00                   1       116268.058032   \n",
       "\n",
       "                           volume_macd_60_150_signal  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                          1   \n",
       "2023-09-14 15:57:00+00:00                          1   \n",
       "2023-09-14 15:58:00+00:00                          1   \n",
       "2023-09-14 15:59:00+00:00                          1   \n",
       "2023-09-14 16:00:00+00:00                          1   \n",
       "\n",
       "                           ema_volume_vol_std_60_150  \\\n",
       "timestamp                                              \n",
       "2023-09-14 15:56:00+00:00                   0.177078   \n",
       "2023-09-14 15:57:00+00:00                   0.210460   \n",
       "2023-09-14 15:58:00+00:00                   0.212922   \n",
       "2023-09-14 15:59:00+00:00                   0.276173   \n",
       "2023-09-14 16:00:00+00:00                   0.321313   \n",
       "\n",
       "                           ema_volume_vol_std_new_60_150  \\\n",
       "timestamp                                                  \n",
       "2023-09-14 15:56:00+00:00                       0.306144   \n",
       "2023-09-14 15:57:00+00:00                       0.336275   \n",
       "2023-09-14 15:58:00+00:00                       0.340911   \n",
       "2023-09-14 15:59:00+00:00                       0.349819   \n",
       "2023-09-14 16:00:00+00:00                       0.216451   \n",
       "\n",
       "                           volume_macd_pct_60_150  ema_vol_std_shift_60_150  \\\n",
       "timestamp                                                                     \n",
       "2023-09-14 15:56:00+00:00               22.847503                  0.293635   \n",
       "2023-09-14 15:57:00+00:00               25.376422                  0.300452   \n",
       "2023-09-14 15:58:00+00:00               25.366155                  0.306827   \n",
       "2023-09-14 15:59:00+00:00               29.951190                  0.315391   \n",
       "2023-09-14 16:00:00+00:00               45.305569                  0.308433   \n",
       "\n",
       "                           ema_vol_std_shift_new_60_150  macd_pct_60_150  \\\n",
       "timestamp                                                                  \n",
       "2023-09-14 15:56:00+00:00                      0.499343         0.121814   \n",
       "2023-09-14 15:57:00+00:00                      0.508171         0.124718   \n",
       "2023-09-14 15:58:00+00:00                      0.516440         0.127434   \n",
       "2023-09-14 15:59:00+00:00                      0.526146         0.131206   \n",
       "2023-09-14 16:00:00+00:00                      0.517314         0.127953   \n",
       "\n",
       "                           vol_std_pct_ratio_60_150  returns_EMA_20  \\\n",
       "timestamp                                                             \n",
       "2023-09-14 15:56:00+00:00                  0.587327        0.000097   \n",
       "2023-09-14 15:57:00+00:00                  0.590504        0.000092   \n",
       "2023-09-14 15:58:00+00:00                  0.593361        0.000088   \n",
       "2023-09-14 15:59:00+00:00                  0.598650        0.000114   \n",
       "2023-09-14 16:00:00+00:00                  0.595458       -0.000057   \n",
       "\n",
       "                           returns_EMA_50  returns_EMA_80  returns_EMA_200  \\\n",
       "timestamp                                                                    \n",
       "2023-09-14 15:56:00+00:00    6.004226e-05        0.000046         0.000026   \n",
       "2023-09-14 15:57:00+00:00    5.886496e-05        0.000045         0.000026   \n",
       "2023-09-14 15:58:00+00:00    5.771075e-05        0.000045         0.000026   \n",
       "2023-09-14 15:59:00+00:00    6.897742e-05        0.000052         0.000029   \n",
       "2023-09-14 16:00:00+00:00   -5.224009e-07        0.000008         0.000011   \n",
       "\n",
       "                           returns_EMA_1200  candle  candle_pct  \n",
       "timestamp                                                        \n",
       "2023-09-14 15:56:00+00:00          0.000014   0.005    0.031626  \n",
       "2023-09-14 15:57:00+00:00          0.000014   0.005    0.031626  \n",
       "2023-09-14 15:58:00+00:00          0.000014   0.000    0.000000  \n",
       "2023-09-14 15:59:00+00:00          0.000015   0.010    0.063231  \n",
       "2023-09-14 16:00:00+00:00          0.000012  -0.060   -0.379027  \n",
       "\n",
       "[5 rows x 668 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hist.tail()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in order to train the model we need to label desired outcomes (targets).\n",
    "\n",
    "So far the best method I have found is the triple barrier method (see Lopez De Prado, Advances in Financial Machine Learning).\n",
    "\n",
    "The triple barrier considers future stock price change over a specified time window, e.g. 50 minutes. If the price hits the upper barrier first (increases by X amount) it gets a 1, if it hits the lower barrier first it gets a -1. If it doesnt cross either barrier it gets a 0.\n",
    "\n",
    "The barriers could be set to a fixed number, such as 1% or 1 dollar above current price. However, I will set them dynamically by considering the recent trend and volatility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected volatility over a 50 minute time period (% change as decimal points):\n",
      "count    134004.000000\n",
      "mean          0.009144\n",
      "std           0.002030\n",
      "min           0.005583\n",
      "25%           0.008001\n",
      "50%           0.008723\n",
      "75%           0.009877\n",
      "max           0.020173\n",
      "Name: multiplier_volatility, dtype: float64\n",
      "\n",
      "Expected EMA shift over a 50 minute time period (% change as decimal points):\n",
      "count    133954.000000\n",
      "mean          0.007825\n",
      "std           0.001627\n",
      "min           0.003336\n",
      "25%           0.006859\n",
      "50%           0.007548\n",
      "75%           0.008510\n",
      "max           0.016046\n",
      "Name: multiplier, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# This multiplier gives us the expected volatility over a 50 minute time period expressed as pct. \n",
    "# I calculate the standard deviation over the last 50 min expressed as % of the 50 minute EMA.\n",
    "# then I take the exponential moving average of the last 5000 rows of volatilities\n",
    "\n",
    "hist['multiplier_volatility'] = hist['exp_pct_volatility_50']\n",
    "print('Expected volatility over a 50 minute time period (% change as decimal points):')\n",
    "print(hist['multiplier_volatility'].describe())\n",
    "print('')\n",
    "\n",
    "\n",
    "# In order to take a reasonable take profit level we can also add the trend to volatility - by how many pct, does the ema typically increase over a 50m window?\n",
    "hist['multiplier'] = hist['exp_ema_pct_abs_shift_50']\n",
    "print('Expected EMA shift over a 50 minute time period (% change as decimal points):')\n",
    "print(hist['multiplier'].describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# then we obtain our barriers by adding the multipliers to the current price. I have found that the volatility multiplier is sufficient.\n",
    "hist['higher_bound'] = hist.close * (1 + (hist.multiplier * 0 + hist.multiplier_volatility * 1))\n",
    "hist['lower_bound'] = hist.close * (1 - (hist.multiplier * 0 + hist.multiplier_volatility * 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0.0    86013\n",
       "-1.0    24022\n",
       " 1.0    23993\n",
       "Name: barrier, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets use the triple barrier function to label every minute bar based on whether price will hit our desired target in the next 50 minutes\n",
    "window = 26\n",
    "hist = labelling.get_barrier(hist,hist.close.values,hist.higher_bound.values,hist.lower_bound.values,window=window,side=None)\n",
    "hist['barrier'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some algorithms like Keras LSTMs like this labelling more\n",
    "hist['barrier'].loc[hist['barrier'] == 1] = 2\n",
    "hist['barrier'].loc[hist['barrier'] == 0] = 1\n",
    "hist['barrier'].loc[hist['barrier'] == -1] = 0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have created our features and target variables, we have to reduce the number of features we are considering to avoid overfitting and make the model interpretable.\n",
    "\n",
    "We can use different methods for it like recursive feature elemination (RFE), genetic algorithms or mutual information. Below is an example using RFE.\n",
    "\n",
    "Running the below cell takes 18m on my machine. If you want to skip this just use the model features I get as a result of my run (provided below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting estimator with 668 features.\n",
      "Fitting estimator with 648 features.\n",
      "Fitting estimator with 628 features.\n",
      "Fitting estimator with 608 features.\n",
      "Fitting estimator with 588 features.\n",
      "Fitting estimator with 568 features.\n",
      "Fitting estimator with 548 features.\n",
      "Fitting estimator with 528 features.\n",
      "Fitting estimator with 508 features.\n",
      "Fitting estimator with 488 features.\n",
      "Fitting estimator with 468 features.\n",
      "Fitting estimator with 448 features.\n",
      "Fitting estimator with 428 features.\n",
      "Fitting estimator with 408 features.\n",
      "Fitting estimator with 388 features.\n",
      "Fitting estimator with 368 features.\n",
      "Fitting estimator with 348 features.\n",
      "Fitting estimator with 328 features.\n",
      "Fitting estimator with 308 features.\n",
      "Fitting estimator with 288 features.\n",
      "Fitting estimator with 268 features.\n",
      "Fitting estimator with 248 features.\n",
      "Fitting estimator with 228 features.\n",
      "Fitting estimator with 208 features.\n",
      "Fitting estimator with 188 features.\n",
      "Fitting estimator with 168 features.\n",
      "Fitting estimator with 148 features.\n",
      "Fitting estimator with 128 features.\n",
      "Fitting estimator with 108 features.\n",
      "Fitting estimator with 88 features.\n",
      "Fitting estimator with 68 features.\n",
      "Fitting estimator with 48 features.\n",
      "Fitting estimator with 28 features.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Index(['minutes_open', '30_min_bin', 'last_30_min', 'returns_vol_std_26',\n",
       "       'volume_macd_50_200', 'volume_macd_pct_50_200', 'volume_macd_60_150',\n",
       "       'ema_volume_vol_std_60_150', 'ema_volume_vol_std_new_60_150',\n",
       "       'volume_macd_pct_60_150'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_ranking = feature_selection.select_best_features(hist.drop(columns=['date','datetime','market_open','datetime_market_open']), \n",
    "                                       method='RFE', \n",
    "                                       estimator_type = 'random_forest', \n",
    "                                       #class_weight = {0:5,1:1,2:10},\n",
    "                                       max_depth = 4, \n",
    "                                       n_features_to_select=10,\n",
    "                                       step = 20)\n",
    "\n",
    "f = pd.Series(feature_ranking,index=hist.drop(columns=['barrier','date','datetime','market_open','datetime_market_open']).columns)\n",
    "model_features = list(f.loc[f==1].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['minutes_open', '30_min_bin', 'last_30_min', 'returns_vol_std_26', 'volume_macd_50_200', 'volume_macd_pct_50_200', 'volume_macd_60_150', 'ema_volume_vol_std_60_150', 'ema_volume_vol_std_new_60_150', 'volume_macd_pct_60_150']\n"
     ]
    }
   ],
   "source": [
    "# here are the features you get from RFE\n",
    "#print(model_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# alternatively here are the features I get as a result of RFE:\n",
    "model_features = ['minutes_open', '30_min_bin', 'returns_vol_std_20',\n",
    "       'volume_macd_50_100', 'ema_volume_vol_std_50_200',\n",
    "       'volume_macd_pct_50_200', 'volume_macd_60_150',\n",
    "       'ema_volume_vol_std_60_150', 'ema_volume_vol_std_new_60_150',\n",
    "       'volume_macd_pct_60_150']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example shows how RFE can be useful to identify features, but also should not be used on its own.\n",
    "\n",
    "The algorithm identified a useful time window of comparing the 60m and 150m EMAs. \n",
    "\n",
    "However, it found redundant features that share a lot of information with each other as 'volume_macd_60_150', 'volume_macd_pct_60_150', 'ema_volume_vol_std_60_150' are the same variable (MACD of volume), but transformed to be expressed as pct or standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add barrier to the list of features as the next function will use it\n",
    "model_features = model_features + ['barrier']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to train our model. I am using gradient boosting to predict which barrier the stock price will hit first. \n",
    "\n",
    "The backtest days parameter controls what time period to set aside for backtesting. Lets set aside the last 30 days for testing. \n",
    "\n",
    "Be mindful that the classification metrics are hard to interpret. I usually get a low f-1 score for the target class (2) ~30, but the winrate may still come out to 50%. Or the winrate may be ~30% with a 2:1 win to loss ratio and decent returns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.42      0.22      0.29      1266\n",
      "         1.0       0.83      0.96      0.89      5901\n",
      "         2.0       0.44      0.33      0.38      1410\n",
      "\n",
      "    accuracy                           0.75      8577\n",
      "   macro avg       0.56      0.50      0.52      8577\n",
      "weighted avg       0.70      0.75      0.72      8577\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ypred, model_index,model = models.predict_barrier(hist, \n",
    "                                                  features = model_features, \n",
    "                                                  max_depth=4, \n",
    "                                                  model_type='boosting', \n",
    "                                                  parallel=True, \n",
    "                                                  normalise=False,\n",
    "                                                  tree_method='hist',\n",
    "                                                  backtest_days=30,\n",
    "                                                  prod_model=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ema_volume_vol_std_60_150        0.036540\n",
       "ema_volume_vol_std_new_60_150    0.047213\n",
       "ema_volume_vol_std_50_200        0.066125\n",
       "30_min_bin                       0.068681\n",
       "volume_macd_pct_50_200           0.069833\n",
       "volume_macd_pct_60_150           0.090011\n",
       "returns_vol_std_20               0.109757\n",
       "volume_macd_50_100               0.130391\n",
       "minutes_open                     0.185698\n",
       "volume_macd_60_150               0.195751\n",
       "dtype: float32"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# which features are most useful for our predictions?\n",
    "pd.Series(model.feature_importances_,index=model.feature_names_in_).sort_values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    7516\n",
       "1    1061\n",
       "Name: strategy1_buy, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the model output to create a dataframe for backtesting \n",
    "backtest_hist = hist.loc[model_index]\n",
    "backtest_hist['ml_prediction'] = ypred\n",
    "backtest_hist['strategy1_sell'] = 0\n",
    "backtest_hist['strategy1_buy'] = 0\n",
    "\n",
    "# whenever the model predicts that we will hit the upper barrier, we will take it as a buy signal\n",
    "backtest_hist['strategy1_buy'].loc[(backtest_hist.ml_prediction == 2)] = 1\n",
    "\n",
    "# optionally, we can use the same model to generate sell signals\n",
    "#backtest_hist['strategy1_sell'].loc[(backtest_hist.ml_prediction == 0)] = 1\n",
    "\n",
    "# how many buy signals (1) do we get in our sample?\n",
    "backtest_hist.strategy1_buy.value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# before we backtest, we need to set the stop loss and take profit levels\n",
    "# these can be different from the barriers we used during labelling\n",
    "# lets use 26 minute volatility and a 3:1 ratio of take profit to stop loss\n",
    "\n",
    "backtest_hist['multiplier'] = backtest_hist['exp_ema_pct_abs_shift_26']\n",
    "backtest_hist['multiplier_volatility'] = backtest_hist['exp_pct_volatility_26']\n",
    "\n",
    "backtest_hist['higher_bound'] = backtest_hist.close * (1 + ((0 * backtest_hist.multiplier + 3 * backtest_hist.multiplier_volatility) * 1))\n",
    "backtest_hist['lower_bound'] = backtest_hist.close * (1 - ((0 * backtest_hist.multiplier + 1 * backtest_hist.multiplier_volatility) * 1))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The backtest iterates over our test dataframe and simulates trading decisions. A buy is made based on the ML model predictions.\n",
    "\n",
    "A sell decision can be made based on the ML model and/ or crossing stop loss take profit (SLTP) levels. The backtest params below use the ML model to make buy decisions and stop-loss/ take-profit to sell. We set the holding period to 30 min, meaning that if no barrier is hit in 30 minutes we will sell. In addition, we will hold stock if there is a new buy signal, even if we are below/ above SLTP level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cash money:\n",
      "9793.476002842042\n",
      "112\n",
      "112\n",
      "Winrate:\n",
      "0.2857142857142857\n",
      "Avg win:\n",
      "0.012199857873811865\n",
      "Avg loss:\n",
      "-0.006830136406268766\n"
     ]
    }
   ],
   "source": [
    "trades = backtest.run_backtest(\n",
    "    backtest_hist=backtest_hist.loc[backtest_hist.lower_bound.notna()], \n",
    "    buy_index=backtest_hist.loc[backtest_hist.strategy1_buy==1].index, \n",
    "    sell_index=[],\n",
    "    take_profit_series=backtest_hist.higher_bound,\n",
    "    stop_loss_series=backtest_hist.lower_bound,\n",
    "    risk=1,\n",
    "    cash = 10000,\n",
    "    holding_period= 30,\n",
    "    min_holding_period=0,\n",
    "    hold_on_buy_signal=False,\n",
    "    sltp=True,\n",
    "    sltp_update=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the automated features the model is not performing well. The wins are 2x bigger than losses, but not enough to compensate for the low winrate. Lets try create a model with different features.\n",
    "\n",
    "Lets use:\n",
    "* Swing high/ swing low pct - how far was the recent swing high/ low from the EMA line expressed as pct?\n",
    "* Volume and Closing price MACDs expressed as pct\n",
    "* Volatility metric vol_std_pct_ based on an EMA of standard deviation expressed as pct\n",
    "* Trend and volatility variables we used during target labelling:\n",
    "    * 'exp_ema_pct_abs_shift_50' is the expected change in EMA over 50 minutes based on the last 5000 such shifts.\n",
    "    * 'exp_pct_volatility_50' is the expected 50 minute volatility based on the last 5000 such volatilities.\n",
    "* 30_min_bin and minutes_open capture how long the market has been open on that day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hand picked features by logic\n",
    "model_features = [\n",
    "    'swing_high_pct_12', 'swing_low_pct_50','swing_low_pct_100', 'swing_high_pct_100','swing_high_pct_26',\n",
    "    'volume_macd_pct_12_100','volume_macd_pct_26_100', 'macd_pct_26_100',\n",
    "    'vol_std_pct_100', 'vol_std_pct_26', \n",
    "    'exp_ema_pct_abs_shift_50', 'exp_pct_volatility_50',\n",
    "    '30_min_bin', 'minutes_open'\n",
    "    ]\n",
    "model_features = model_features + ['barrier']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.42      0.31      0.36      1266\n",
      "         1.0       0.83      0.95      0.89      5901\n",
      "         2.0       0.48      0.33      0.39      1410\n",
      "\n",
      "    accuracy                           0.75      8577\n",
      "   macro avg       0.58      0.53      0.54      8577\n",
      "weighted avg       0.71      0.75      0.73      8577\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ypred, model_index,model = models.predict_barrier(hist, \n",
    "                                                  features = model_features, \n",
    "                                                  max_depth=4, \n",
    "                                                  model_type='boosting', \n",
    "                                                  parallel=True, \n",
    "                                                  normalise=False,\n",
    "                                                  tree_method='hist',\n",
    "                                                  backtest_days=30,\n",
    "                                                  prod_model=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    7620\n",
       "1     957\n",
       "Name: strategy1_buy, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the model output to create a dataframe for backtesting \n",
    "backtest_hist = hist.loc[model_index]\n",
    "backtest_hist['ml_prediction'] = ypred\n",
    "backtest_hist['strategy1_sell'] = 0\n",
    "backtest_hist['strategy1_buy'] = 0\n",
    "\n",
    "# whenever the model predicts that we will hit the upper barrier, we will take it as a buy signal\n",
    "backtest_hist['strategy1_buy'].loc[(backtest_hist.ml_prediction == 2)] = 1\n",
    "\n",
    "# optionally, we can use the same model to generate sell signals\n",
    "#backtest_hist['strategy1_sell'].loc[(backtest_hist.ml_prediction == 0)] = 1\n",
    "\n",
    "# how many buy signals (1) do we get in our sample?\n",
    "backtest_hist.strategy1_buy.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "backtest_hist['multiplier'] = backtest_hist['exp_ema_pct_abs_shift_26']\n",
    "backtest_hist['multiplier_volatility'] = backtest_hist['exp_pct_volatility_26']\n",
    "\n",
    "backtest_hist['higher_bound'] = backtest_hist.close * (1 + ((0 * backtest_hist.multiplier + 3 * backtest_hist.multiplier_volatility) * 1))\n",
    "backtest_hist['lower_bound'] = backtest_hist.close * (1 - ((0 * backtest_hist.multiplier + 1 * backtest_hist.multiplier_volatility) * 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cash money:\n",
      "10700.544221768389\n",
      "98\n",
      "98\n",
      "Winrate:\n",
      "0.2755102040816326\n",
      "Avg win:\n",
      "0.015003647388568185\n",
      "Avg loss:\n",
      "-0.0066050339319929\n"
     ]
    }
   ],
   "source": [
    "trades = backtest.run_backtest(\n",
    "    backtest_hist=backtest_hist.loc[backtest_hist.lower_bound.notna()], \n",
    "    buy_index=backtest_hist.loc[backtest_hist.strategy1_buy==1].index, \n",
    "    sell_index=[],\n",
    "    take_profit_series=backtest_hist.higher_bound,\n",
    "    stop_loss_series=backtest_hist.lower_bound,\n",
    "    risk=1,\n",
    "    cash = 10000,\n",
    "    holding_period= 30,\n",
    "    min_holding_period=0,\n",
    "    hold_on_buy_signal=False,\n",
    "    sltp=True,\n",
    "    sltp_update=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The algorithm gives me 10700, so 7% return in a month, with a winrate of 28%, and the average win size of 1.5% and loss size of 0.66 (more than 2:1 ratio). You can review the simulated trades here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>buy_index</th>\n",
       "      <th>buy_price</th>\n",
       "      <th>position</th>\n",
       "      <th>stop_loss</th>\n",
       "      <th>take_profit</th>\n",
       "      <th>sell_index</th>\n",
       "      <th>price</th>\n",
       "      <th>$ gain</th>\n",
       "      <th>returns</th>\n",
       "      <th>holding_duration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>2023-09-13 09:31:00+00:00</td>\n",
       "      <td>15.6588</td>\n",
       "      <td>673.040723</td>\n",
       "      <td>15.575146</td>\n",
       "      <td>15.909761</td>\n",
       "      <td>2023-09-13 09:33:00+00:00</td>\n",
       "      <td>15.5700</td>\n",
       "      <td>-59.766016</td>\n",
       "      <td>-0.005671</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>2023-09-13 09:34:00+00:00</td>\n",
       "      <td>15.5500</td>\n",
       "      <td>673.906370</td>\n",
       "      <td>15.466987</td>\n",
       "      <td>15.799038</td>\n",
       "      <td>2023-09-13 09:41:00+00:00</td>\n",
       "      <td>15.8200</td>\n",
       "      <td>181.954720</td>\n",
       "      <td>0.017363</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>2023-09-13 15:37:00+00:00</td>\n",
       "      <td>15.5550</td>\n",
       "      <td>685.387256</td>\n",
       "      <td>15.475966</td>\n",
       "      <td>15.792102</td>\n",
       "      <td>2023-09-13 16:00:00+00:00</td>\n",
       "      <td>15.5900</td>\n",
       "      <td>23.988554</td>\n",
       "      <td>0.002250</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>2023-09-14 09:31:00+00:00</td>\n",
       "      <td>15.7750</td>\n",
       "      <td>677.349434</td>\n",
       "      <td>15.695299</td>\n",
       "      <td>16.014103</td>\n",
       "      <td>2023-09-14 09:41:00+00:00</td>\n",
       "      <td>16.0314</td>\n",
       "      <td>173.672395</td>\n",
       "      <td>0.016254</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>2023-09-14 09:42:00+00:00</td>\n",
       "      <td>15.9588</td>\n",
       "      <td>680.430842</td>\n",
       "      <td>15.877983</td>\n",
       "      <td>16.201252</td>\n",
       "      <td>2023-09-14 09:48:00+00:00</td>\n",
       "      <td>15.8550</td>\n",
       "      <td>-70.628721</td>\n",
       "      <td>-0.006504</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>2023-09-14 09:49:00+00:00</td>\n",
       "      <td>15.8233</td>\n",
       "      <td>681.794000</td>\n",
       "      <td>15.742925</td>\n",
       "      <td>16.064424</td>\n",
       "      <td>2023-09-14 09:53:00+00:00</td>\n",
       "      <td>15.7300</td>\n",
       "      <td>-63.611380</td>\n",
       "      <td>-0.005896</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>2023-09-14 09:54:00+00:00</td>\n",
       "      <td>15.7550</td>\n",
       "      <td>680.712131</td>\n",
       "      <td>15.674822</td>\n",
       "      <td>15.995535</td>\n",
       "      <td>2023-09-14 09:56:00+00:00</td>\n",
       "      <td>15.6400</td>\n",
       "      <td>-78.281895</td>\n",
       "      <td>-0.007299</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>2023-09-14 09:57:00+00:00</td>\n",
       "      <td>15.7200</td>\n",
       "      <td>677.247947</td>\n",
       "      <td>15.639920</td>\n",
       "      <td>15.960239</td>\n",
       "      <td>2023-09-14 10:08:00+00:00</td>\n",
       "      <td>15.6395</td>\n",
       "      <td>-54.518460</td>\n",
       "      <td>-0.005121</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>2023-09-14 10:09:00+00:00</td>\n",
       "      <td>15.6050</td>\n",
       "      <td>678.745227</td>\n",
       "      <td>15.525264</td>\n",
       "      <td>15.844207</td>\n",
       "      <td>2023-09-14 10:18:00+00:00</td>\n",
       "      <td>15.5250</td>\n",
       "      <td>-54.299618</td>\n",
       "      <td>-0.005127</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>2023-09-14 10:19:00+00:00</td>\n",
       "      <td>15.5001</td>\n",
       "      <td>679.835591</td>\n",
       "      <td>15.420723</td>\n",
       "      <td>15.738231</td>\n",
       "      <td>2023-09-14 10:38:00+00:00</td>\n",
       "      <td>15.7399</td>\n",
       "      <td>163.024575</td>\n",
       "      <td>0.015471</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   buy_index  buy_price    position  stop_loss  take_profit  \\\n",
       "88 2023-09-13 09:31:00+00:00    15.6588  673.040723  15.575146    15.909761   \n",
       "89 2023-09-13 09:34:00+00:00    15.5500  673.906370  15.466987    15.799038   \n",
       "90 2023-09-13 15:37:00+00:00    15.5550  685.387256  15.475966    15.792102   \n",
       "91 2023-09-14 09:31:00+00:00    15.7750  677.349434  15.695299    16.014103   \n",
       "92 2023-09-14 09:42:00+00:00    15.9588  680.430842  15.877983    16.201252   \n",
       "93 2023-09-14 09:49:00+00:00    15.8233  681.794000  15.742925    16.064424   \n",
       "94 2023-09-14 09:54:00+00:00    15.7550  680.712131  15.674822    15.995535   \n",
       "95 2023-09-14 09:57:00+00:00    15.7200  677.247947  15.639920    15.960239   \n",
       "96 2023-09-14 10:09:00+00:00    15.6050  678.745227  15.525264    15.844207   \n",
       "97 2023-09-14 10:19:00+00:00    15.5001  679.835591  15.420723    15.738231   \n",
       "\n",
       "                  sell_index    price      $ gain   returns  holding_duration  \n",
       "88 2023-09-13 09:33:00+00:00  15.5700  -59.766016 -0.005671                 2  \n",
       "89 2023-09-13 09:41:00+00:00  15.8200  181.954720  0.017363                 7  \n",
       "90 2023-09-13 16:00:00+00:00  15.5900   23.988554  0.002250                23  \n",
       "91 2023-09-14 09:41:00+00:00  16.0314  173.672395  0.016254                10  \n",
       "92 2023-09-14 09:48:00+00:00  15.8550  -70.628721 -0.006504                 6  \n",
       "93 2023-09-14 09:53:00+00:00  15.7300  -63.611380 -0.005896                 4  \n",
       "94 2023-09-14 09:56:00+00:00  15.6400  -78.281895 -0.007299                 2  \n",
       "95 2023-09-14 10:08:00+00:00  15.6395  -54.518460 -0.005121                11  \n",
       "96 2023-09-14 10:18:00+00:00  15.5250  -54.299618 -0.005127                 9  \n",
       "97 2023-09-14 10:38:00+00:00  15.7399  163.024575  0.015471                19  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trades[0].tail(10)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next stage in strategy development is to deploy the model in a real trading environment, which is what I am doing right now using the Alpaca Trading API.\n",
    "\n",
    "Trading strategies developed in this way tend to rapidly decay over time with changing market conditions.\n",
    "\n",
    "In order to ensure consistent returns, I am currently automating the process of iterating over hundreds of such strategies using different features and stocks, and then retraining and deploying the best models.\n",
    "\n",
    "Contact me if you have some questions, feedback or insights to share: \\\n",
    "email: gluzman64@gmail.com \\\n",
    "linkedin: linkedin.com/in/egluzman/ \\\n",
    "whatsapp: +447463457579"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".stonks",
   "language": "python",
   "name": ".stonks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "39fd4f7935656d6ca2e81686471bb91fda69f889511528cd43d174a39a9c9681"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}