EEG_ML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "This file is used for classification of emotion-based EEG signals from PD and HC using various Machine Learning (ML) algorithms. The input to ML algorithms are *Raw*, *CSP*, and *SPV* features. While *Raw* and *SPV* features are obtained using `matlab_files/eeg_csp_raw_run.m`, *SPV* features are obtained using `eeg_spectral.ipynb`. The detailed architecture, input and output dimensions are explained in the paper. The highlighted part in the following image indicate the pipeline of the current file.\n",
    "\n",
    "Note: The configurations, like hyperparameter values, present in this file are not the final configurations as reported in the paper. The present configurations are maybe mutated for additional experiments. However, results reported in the paper can be reproduced by plugging in the appropriate configurations as mentioned in the paper.\n",
    "\n",
    "\n",
    "![ML pipeline associated with the current notebook](images/ml.png)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We explore the following ML algorithms for the classification task:\n",
    "1. k-Nearest Neighbour (kNN), where the test sample is assigned the label corresponding to the mode of its k-*closest* neighbours based on a suitable distance metric.\n",
    "2. Support Vector Machine (SVM), where input data are transformed to a high-dimensional space where the two classes are linearly separable and the inter-class distance is maximum.\n",
    "3. Gaussian Naive Bayes (GNB), a generative classifier assuming class-conditional feature independence.\n",
    "4. Decision Tree (DT), which uses a tree-like graph structure where each leaf node represents a category label.\n",
    "5. Linear Discriminant Analysis (LDA), which linearly transforms data to achieve maximal inter-class distance.\n",
    "6. Logistic Regression (LR), which maps the input to class labels via the sigmoid function."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Imports"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy.io as sio\n",
    "import os\n",
    "#from skopt import BayesSearchCV\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import cross_validate\n",
    "from sklearn.metrics import make_scorer, accuracy_score, precision_score, recall_score, f1_score\n",
    "#from skopt.space import Real, Categorical, Integer\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sns\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\""
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eOJVKOd7QHG_",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "khrpfSHYq_jN",
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Path to pre-processed features\n",
    "data = '../data/iir_feat_tensor_full_with_full_labels.mat'\n",
    "\n",
    "# path to save final features, results, etc.\n",
    "savePath = '../results/MLResults/IIR/'\n",
    "#data = pd.read_csv(data, header=None)\n",
    "#data.head()\n",
    "#X = data.iloc[1:,0:-2]\n",
    "#y = data.iloc[1:,-1]\n",
    "\n",
    "# Load data\n",
    "matContent = sio.loadmat(data)\n",
    "X = matContent['nc_feat']\n",
    "y = np.squeeze(matContent['nc_multi_labels'])\n",
    "\n",
    "# Experiment name\n",
    "experiment = 'MultiClass_IIR_NC'\n",
    "\n",
    "filename = savePath+'ML_'+experiment+'.mat'\n",
    "\n",
    "cm_title = 'Categorical Emotion Classification for IIR features - NC (NB)'\n",
    "\n",
    "X = np.reshape(X, (X.shape[0],X.shape[1]*X.shape[2]))\n",
    "del matContent\n",
    "\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Data Standardization and PCA"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "CZDHWi-mfhkT",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Function for dimensionality reduction\n",
    "def reduce_dim(X, y, pca_thresh, Scaler):\n",
    "    y=y\n",
    "    #y = np.ravel(y)\n",
    "    if Scaler==\"MinMax\":\n",
    "        from sklearn.preprocessing import MinMaxScaler\n",
    "        sc = MinMaxScaler()\n",
    "        X = sc.fit_transform(X)\n",
    "    elif Scaler==\"MaxAbs\":\n",
    "        from sklearn.preprocessing import MaxAbsScaler\n",
    "        sc = MaxAbsScaler()\n",
    "        X = sc.fit_transform(X)\n",
    "    elif Scaler == \"Standard\":\n",
    "        from sklearn.preprocessing import StandardScaler\n",
    "        sc = StandardScaler()\n",
    "        X = sc.fit_transform(X)\n",
    "    elif Scaler == \"Normalizer\":\n",
    "        from sklearn.preprocessing import Normalizer\n",
    "        sc = Normalizer()\n",
    "        X = sc.fit_transform(X)\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "    pca = PCA(pca_thresh)\n",
    "    X = pca.fit_transform(X)\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "X, y = reduce_dim(X, y, 0.95, 'Normalizer')\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "18jbTqgmQsTf",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# KNN Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "2743XUHHbM8W",
    "outputId": "b062ef90-75bf-4696-a335-64a3b7a0313d",
    "scrolled": true,
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# K Nearest Neighbors\n",
    "\n",
    "#scoring = {'accuracy' : make_scorer(accuracy_score), 'f1_score' : make_scorer(f1_score, average='weighted')}\n",
    "# Parameter space for Grid search\n",
    "depth = np.logspace(0.001,10)\n",
    "param = {'n_neighbors':[2,3,4,5,6,7,8,9,10], 'weights':['uniform','distance'],\n",
    "         'metric':['euclidean','chebychev','minkowski']}\n",
    "\n",
    "# Perform Grid search\n",
    "opt = GridSearchCV(KNeighborsClassifier(), param_grid=param, cv=10, verbose=2, n_jobs=-1)\n",
    "clf = opt.fit(X,y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:', best_params)\n",
    "\n",
    "# Initialise Stratified K Fold\n",
    "kfold = StratifiedKFold(n_splits=10, shuffle=True)\n",
    "acc_list = [] \n",
    "f1_list = []\n",
    "\n",
    "# KNN Classifier\n",
    "#estimator = KNeighborsClassifier(n_neighbors=best_params.get('n_neighbors'), weights=best_params.get('weights'),\n",
    "#                                metric=best_params.get('metric'))\n",
    "   \n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring=scoring, verbose=2)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print('Accuracy:', round(np.mean(acc),4), '+-', round(np.std(acc),4)) \n",
    "print('F1Score:', round(np.mean(f1),4), '+-', round(np.std(f1),4))\n",
    "'''\n",
    "cm_list = []\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "#conf_mat = np.zeros((2,2))\n",
    "conf_mat = np.zeros((6,6))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "    X_train, X_test = X[train_index], X[test_index] \n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    estimator = KNeighborsClassifier(n_neighbors=best_params.get('n_neighbors'), weights=best_params.get('weights'),\n",
    "                                metric=best_params.get('metric'))\n",
    "    estimator.fit(X_train, y_train) \n",
    "    y_pred = estimator.predict(X_test)\n",
    "    acc = accuracy_score(y_test, y_pred)\n",
    "    acc_list.append(acc)\n",
    "    wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "    macro_f1 = f1_score(y_test, y_pred, average='micro')\n",
    "    wtd_f1_list.append(wtd_f1)\n",
    "    macro_f1_list.append(macro_f1)\n",
    "    #cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "    cm = confusion_matrix(y_test, y_pred, labels=[1,2,3,4,5,6])\n",
    "    cm = cm/cm.sum(axis=1, keepdims=True)\n",
    "    #conf_mat = conf_mat+cm\n",
    "    cm_list.append(cm)\n",
    "\n",
    "# Dump results\n",
    "#conf_mat /= conf_mat.sum(axis=1, keepdims=True) #Normalised values in the CM\n",
    "conf_mat = np.mean(cm_list, axis=0)\n",
    "acc_mean = round(np.mean(acc_list),3)\n",
    "acc_std = round(np.std(acc_list),3)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),3)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),3)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),3)\n",
    "macro_f1_std = round(np.std(macro_f1_list),3)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "\n",
    "# Visualise results\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "#ax.set_yticklabels(['LA','HA'],rotation = 0)\n",
    "#ax.set_xticklabels(['LA','HA'],rotation = 0)\n",
    "ax.set_yticklabels(['Sad', 'Happy', 'Fear', 'Disgust', 'Surprise', 'Anger'],rotation = 0)\n",
    "ax.set_xticklabels(['Sad', 'Happy', 'Fear', 'Disgust', 'Surprise', 'Anger'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename[:-4]+'_conf_mat'+'.png')\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'experiment':experiment, 'conf_mat':conf_mat, 'conf_mat_list': cm_list,\n",
    "                       'best_params':best_params, 'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, \n",
    "                       'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k2MPKHgoQene",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# SVM Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Poly SVM"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Aiktv5L9SiZY",
    "outputId": "2552b1be-0e50-493d-ba40-ba82b992839b",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#Poly SVM\n",
    "\n",
    "scoring = {'accuracy': make_scorer(accuracy_score), 'f1_score': make_scorer(f1_score, average='weighted')}\n",
    "#scoring = ['accuracy', 'f1_score']\n",
    "'''\n",
    "param = {'C':Real(1e-3,1e+3,prior='log-uniform'),\n",
    "         'gamma':Real(1e-3,1e+1,prior='log-uniform')}\n",
    "'''\n",
    "param = {'C':[0.001,0.01,0.1,1,10,100,1000]}\n",
    "\n",
    "# Perform Grid search\n",
    "opt = GridSearchCV(SVC(), param_grid=param, cv=10, verbose=2, n_jobs = -1)\n",
    "\n",
    "svmclf = opt.fit(X, y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:',best_params)\n",
    "\n",
    "\n",
    "# Initialise Stratified K Fold\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "#estimator = SVC(C = best_params.get('C'),kernel='rbf')\n",
    "\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring = scoring, verbose=2, n_jobs = -1)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print(\"Accuracy:\", round(np.mean(acc), 4), \"+-\", round(np.std(acc), 4))\n",
    "print(\"F1 score:\", round(np.mean(f1), 4), \"+-\", round(np.std(f1), 4))\n",
    "'''\n",
    "\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "conf_mat = np.zeros((2,2))\n",
    "#conf_mat = np.zeros((6,6))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "    X_train, X_test = X[train_index], X[test_index] \n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    estimator = SVC(C = best_params.get('C'),kernel='rbf')\n",
    "    estimator.fit(X_train, y_train) \n",
    "    y_pred = estimator.predict(X_test)\n",
    "    acc = accuracy_score(y_test, y_pred)\n",
    "    acc_list.append(acc)\n",
    "    wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "    macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "    wtd_f1_list.append(wtd_f1)\n",
    "    macro_f1_list.append(macro_f1)\n",
    "    cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "    #cm = confusion_matrix(y_test, y_pred, labels=[1,2,3,4,5,6])\n",
    "    cm = cm/cm.sum(axis=1, keepdims=True)\n",
    "    #conf_mat = conf_mat+cm\n",
    "    cm_list.append(cm)\n",
    "\n",
    "# Visualise results\n",
    "#conf_mat /= conf_mat.sum(axis=1, keepdims=True) #Normalised values in the CM\n",
    "conf_mat = np.mean(cm_list, axis=0)\n",
    "acc_mean = round(np.mean(acc_list),3)\n",
    "acc_std = round(np.std(acc_list),3)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),3)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),3)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),3)\n",
    "macro_f1_std = round(np.std(macro_f1_list),3)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "#fig, ax = plt.subplots(figsize=(10,8))\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "ax.set_yticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_xticklabels(['NC','PD'],rotation = 0)\n",
    "#ax.set_yticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "#ax.set_xticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename[:-4]+'_conf_mat'+'.png')\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'conf_mat':conf_mat, 'conf_mat_list': cm_list, 'best_params':best_params,\n",
    "                       'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, 'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Linear SVM"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "jwU3CL9NsWqM",
    "outputId": "162ac978-8383-45f1-f2c0-33fc16d64ae2",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#Linear SVM\n",
    "\n",
    "scoring = {'accuracy': make_scorer(accuracy_score), 'f1_score': make_scorer(f1_score, average='weighted')}\n",
    "#scoring = ['accuracy', 'f1_score']\n",
    "'''\n",
    "param = {'C':Real(1e-3,1e+3,prior='log-uniform'),\n",
    "         'gamma':Real(1e-3,1e+1,prior='log-uniform')}\n",
    "'''\n",
    "param = {'C':[0.001,0.01,0.1,1,10,100,1000]}\n",
    "\n",
    "# Perform Grid search\n",
    "opt = GridSearchCV(LinearSVC(), param_grid=param, cv=10, verbose=1, n_jobs = -1)\n",
    "\n",
    "svmclf = opt.fit(X, y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:',best_params)\n",
    "\n",
    "# Initialise Stratified K Fold\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "#estimator = LinearSVC(C = best_params.get('C'))\n",
    "\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring = scoring, verbose=2, n_jobs = -1)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print(\"Accuracy:\", round(np.mean(acc), 4), \"+-\", round(np.std(acc), 4))\n",
    "print(\"F1 score:\", round(np.mean(f1), 4), \"+-\", round(np.std(f1), 4))\n",
    "'''\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "conf_mat = np.zeros((2,2))\n",
    "#conf_mat = np.zeros((6,6))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "    X_train, X_test = X[train_index], X[test_index] \n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    estimator = LinearSVC(C = best_params.get('C'))\n",
    "    estimator.fit(X_train, y_train) \n",
    "    y_pred = estimator.predict(X_test)\n",
    "    acc = accuracy_score(y_test, y_pred)\n",
    "    acc_list.append(acc)\n",
    "    wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "    macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "    wtd_f1_list.append(wtd_f1)\n",
    "    macro_f1_list.append(macro_f1)\n",
    "    #cm = confusion_matrix(y_test, y_pred, labels=[1,2,3,4,5,6])\n",
    "    #conf_mat = conf_mat+cm\n",
    "    cm_list.append(cm)\n",
    "\n",
    "#conf_mat /= conf_mat.sum(axis=1, keepdims=True) #Normalised values in the CM\n",
    "conf_mat = np.mean(cm_list, axis=0)\n",
    "acc_mean = round(np.mean(acc_list),4)\n",
    "acc_std = round(np.std(acc_list),4)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),4)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),4)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),4)\n",
    "macro_f1_std = round(np.std(macro_f1_list),4)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "\n",
    "# Visualise results\n",
    "\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "#fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "ax.set_yticklabels(['LA','HA'],rotation = 0)\n",
    "ax.set_xticklabels(['LA','HA'],rotation = 0)\n",
    "#ax.set_yticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "#ax.set_xticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename)\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'conf_mat':conf_mat, 'conf_mat_list': cm_list, 'best_params':best_params,\n",
    "                       'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, 'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NG8aJPMSQLw5",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Decision Tree classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "DLf1z_AgwUgW",
    "outputId": "30393b59-57b1-4da2-cac3-9f86f9c28d70",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Decision Tree Classifier\n",
    "\n",
    "scoring = {'accuracy' : make_scorer(accuracy_score), 'f1_score' : make_scorer(f1_score, average='weighted')}\n",
    "\n",
    "# Grid search space\n",
    "depth = np.logspace(0.01,10)\n",
    "param = {'criterion':['gini','entropy'], 'splitter':['best'], 'max_depth':depth,\n",
    "         'max_features': ['auto', 'sqrt', 'log2']}\n",
    "\n",
    "# Perform Grid search\n",
    "opt = GridSearchCV(DecisionTreeClassifier(), param_grid=param, cv=10, verbose=1, n_jobs=-1)\n",
    "\n",
    "clf = opt.fit(X,y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:', best_params)\n",
    "\n",
    "\n",
    "# Initialise Stratified K Fold\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "\n",
    "# Build Classifier\n",
    "#estimator = DecisionTreeClassifier(criterion=best_params.get('criterion'), splitter=best_params.get('splitter'),\n",
    "#                                   max_depth=best_params.get('max_depth'))\n",
    "\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring=scoring, verbose=2)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print('Accuracy:', round(np.mean(acc),4), '+-', round(np.std(acc),4)) \n",
    "print('F1Score:', round(np.mean(f1),4), '+-', round(np.std(f1),4))\n",
    "'''\n",
    "\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "conf_mat = np.zeros((2,2))\n",
    "#conf_mat = np.zeros((6,6))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "  X_train, X_test = X[train_index], X[test_index] \n",
    "  y_train, y_test = y[train_index], y[test_index]\n",
    "  estimator = DecisionTreeClassifier(criterion=best_params.get('criterion'), splitter=best_params.get('splitter'),\n",
    "                                   max_depth=best_params.get('max_depth'))\n",
    "  estimator.fit(X_train, y_train) \n",
    "  y_pred = estimator.predict(X_test)\n",
    "  acc = accuracy_score(y_test, y_pred)\n",
    "  acc_list.append(acc)\n",
    "  wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "  macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "  wtd_f1_list.append(wtd_f1)\n",
    "  macro_f1_list.append(macro_f1)\n",
    "  cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "  #cm = confusion_matrix(y_test, y_pred, labels=[1,2,3,4,5,6])\n",
    "  conf_mat = conf_mat+cm\n",
    "\n",
    "conf_mat /= conf_mat.sum(axis=1, keepdims=True)\n",
    "acc_mean = round(np.mean(acc_list),4)\n",
    "acc_std = round(np.std(acc_list),4)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),4)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),4)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),4)\n",
    "macro_f1_std = round(np.std(macro_f1_list),4)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "\n",
    "# Visualise results\n",
    "\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "#fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "ax.set_yticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_xticklabels(['NC','PD'],rotation = 0)\n",
    "#ax.set_yticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "#ax.set_xticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename)\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'conf_mat':conf_mat, 'conf_mat_list': cm_list, 'best_params':best_params,\n",
    "                       'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, 'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jx7s_6KWQSYc",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Discriminant Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Linear Discriminant Analysis"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4R7EDDaZ2MeM",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Linear Discriminant Analysis:\n",
    "\n",
    "scoring = {'accuracy' : make_scorer(accuracy_score), 'f1_score' : make_scorer(f1_score, average='weighted')}\n",
    "param = {'solver':['svd','lsqr','eigen']}\n",
    "\n",
    "opt = GridSearchCV(LinearDiscriminantAnalysis(), param_grid=param, cv=10, verbose=1, n_jobs=-1)\n",
    "clf = opt.fit(X,y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:', best_params)\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "#estimator = LinearDiscriminantAnalysis(solver=best_params.get('solver'),tol=0.0001)\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring=scoring, verbose=2)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print('Accuracy:', round(np.mean(acc),4), '+-', round(np.std(acc),4)) \n",
    "print('F1Score:', round(np.mean(f1),4), '+-', round(np.std(f1),4))\n",
    "\n",
    "'''\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "#conf_mat = np.zeros((2,2))\n",
    "conf_mat = np.zeros((6,6))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "  X_train, X_test = X[train_index], X[test_index] \n",
    "  y_train, y_test = y[train_index], y[test_index]\n",
    "  estimator = LinearDiscriminantAnalysis(solver=best_params.get('solver'),tol=0.0001)\n",
    "  estimator.fit(X_train, y_train) \n",
    "  y_pred = estimator.predict(X_test)\n",
    "  acc = accuracy_score(y_test, y_pred)\n",
    "  acc_list.append(acc)\n",
    "  wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "  macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "  wtd_f1_list.append(wtd_f1)\n",
    "  macro_f1_list.append(macro_f1)\n",
    "  #cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "  cm = confusion_matrix(y_test, y_pred, labels=[1,2,3,4,5,6])\n",
    "  conf_mat = conf_mat+cm\n",
    "\n",
    "conf_mat /= conf_mat.sum(axis=1, keepdims=True)\n",
    "acc_mean = round(np.mean(acc_list),4)\n",
    "acc_std = round(np.std(acc_list),4)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),4)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),4)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),4)\n",
    "macro_f1_std = round(np.std(macro_f1_list),4)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "#ax.set_yticklabels(['NC','PD'],rotation = 0)\n",
    "#ax.set_xticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_yticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "ax.set_xticklabels(['E1','E2','E3','E4','E5','E6'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename)\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'conf_mat':conf_mat, 'conf_mat_list': cm_list, 'best_params':best_params,\n",
    "                       'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, 'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Quadratic Discriminant"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hoauXF0--C9R",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#Quadratic Discriminant\n",
    "\n",
    "scoring = {'accuracy' : make_scorer(accuracy_score), 'f1_score' : make_scorer(f1_score, average='weighted')}\n",
    "param = {'reg_param':[0.001,0.01,0.1,1,10,100,1000]}\n",
    "\n",
    "opt = GridSearchCV(QuadraticDiscriminantAnalysis(), param_grid=param, cv=10, verbose=1, n_jobs=-1)\n",
    "clf = opt.fit(X,y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:', best_params)\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "#estimator = QuadraticDiscriminantAnalysis(reg_param=best_params.get('reg_param'),tol=0.0001)\n",
    "\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring=scoring, verbose=2)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print('Accuracy:', round(np.mean(acc),4), '+-', round(np.std(acc),4)) \n",
    "print('F1Score:', round(np.mean(f1),4), '+-', round(np.std(f1),4))\n",
    "'''\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "conf_mat = np.zeros((2,2))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "  X_train, X_test = X[train_index], X[test_index] \n",
    "  y_train, y_test = y[train_index], y[test_index]\n",
    "  estimator = QuadraticDiscriminantAnalysis(reg_param=best_params.get('reg_param')\n",
    "  estimator.fit(X_train, y_train) \n",
    "  y_pred = estimator.predict(X_test)\n",
    "  acc = accuracy_score(y_test, y_pred)\n",
    "  acc_list.append(acc)\n",
    "  wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "  macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "  wtd_f1_list.append(wtd_f1)\n",
    "  macro_f1_list.append(macro_f1)\n",
    "  cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "  conf_mat = conf_mat+cm\n",
    "\n",
    "conf_mat /= conf_mat.sum(axis=1, keepdims=True)\n",
    "acc_mean = round(np.mean(acc_list),4)\n",
    "acc_std = round(np.std(acc_list),4)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),4)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),4)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),4)\n",
    "macro_f1_std = round(np.std(macro_f1_list),4)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "#fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "ax.set_yticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_xticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename)\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'conf_mat':conf_mat, 'conf_mat_list': cm_list, 'best_params':best_params,\n",
    "                       'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, 'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GbXSkZp1Qnxj",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Naive Bayes CLassifier "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "w1AVnFYsCCvj",
    "outputId": "859e9223-ba65-4496-f6cb-85647a57bb0e",
    "pycharm": {
     "name": "#%%\n",
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "# Naive Bayes\n",
    "scoring = {'accuracy' : make_scorer(accuracy_score), 'f1_score' : make_scorer(f1_score, average='weighted')}\n",
    "param = {'var_smoothing': [0.0001,0.001,0.01,0.1,1,10,100,100,1000]}\n",
    "\n",
    "opt = GridSearchCV(GaussianNB(), param_grid=param, cv=10, verbose=2, n_jobs=-1)\n",
    "clf = opt.fit(X,y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:', best_params)\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "#estimator = GaussianNB(var_smoothing=best_params.get('var_smoothing'))\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring=scoring, verbose=2)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print('Accuracy:', round(np.mean(acc),4), '+-', round(np.std(acc),4)) \n",
    "print('F1Score:', round(np.mean(f1),4), '+-', round(np.std(f1),4))\n",
    "'''\n",
    "cm_list = []\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "#conf_mat = np.zeros((2,2))\n",
    "conf_mat = np.zeros((6,6))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "    X_train, X_test = X[train_index], X[test_index] \n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    estimator = GaussianNB(var_smoothing=best_params.get('var_smoothing'))\n",
    "    estimator.fit(X_train, y_train) \n",
    "    y_pred = estimator.predict(X_test)\n",
    "    acc = accuracy_score(y_test, y_pred)\n",
    "    acc_list.append(acc)\n",
    "    wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "    macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "    wtd_f1_list.append(wtd_f1)\n",
    "    macro_f1_list.append(macro_f1)\n",
    "    #cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "    cm = confusion_matrix(y_test, y_pred, labels=[1,2,3,4,5,6])\n",
    "    cm = cm/cm.sum(axis=1, keepdims=True)\n",
    "    #conf_mat = conf_mat+cm\n",
    "    cm_list.append(cm)\n",
    "\n",
    "#conf_mat /= conf_mat.sum(axis=1, keepdims=True) #Normalised values in the CM\n",
    "conf_mat = np.mean(cm_list, axis=0)\n",
    "acc_mean = round(np.mean(acc_list),3)\n",
    "acc_std = round(np.std(acc_list),3)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),3)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),3)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),3)\n",
    "macro_f1_std = round(np.std(macro_f1_list),3)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "\n",
    "# Visualise results\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "#ax.set_yticklabels(['LA','HA'],rotation = 0)\n",
    "#ax.set_xticklabels(['LA','HA'],rotation = 0)\n",
    "ax.set_yticklabels(['Sad', 'Happy', 'Fear', 'Disgust', 'Surprise', 'Anger'],rotation = 0)\n",
    "ax.set_xticklabels(['Sad', 'Happy', 'Fear', 'Disgust', 'Surprise', 'Anger'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename[:-4]+'_conf_mat'+'.png')\n",
    "plt.show()\n",
    "\n",
    "# Save results\n",
    "sio.savemat(filename, {'conf_mat':conf_mat, 'conf_mat_list': cm_list, 'best_params':best_params,\n",
    "                       'acc_list':acc_list, 'wtd_f1_list':wtd_f1_list, 'macro_f1_list':macro_f1_list})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gMIyAgGpQg9F",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "ZvgmcqpVs37f",
    "outputId": "7ba46d00-187f-4048-ac16-36a6f709ce80",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#Logistic Regression\n",
    "\n",
    "scoring = {'accuracy': make_scorer(accuracy_score), 'f1_score': make_scorer(f1_score, average='weighted')}\n",
    "#scoring = ['accuracy', 'f1_score']\n",
    "\n",
    "param = {'penalty':['l2','none'],'C':[0.001,0.01,0.1,1,10,100,1000]}\n",
    "\n",
    "opt = GridSearchCV(LogisticRegression(), param_grid=param, cv=10, verbose=2, n_jobs = -1)\n",
    "\n",
    "logclf = opt.fit(X, y)\n",
    "best_params = opt.best_params_\n",
    "print('Best parameters:',best_params)\n",
    "\n",
    "kfold = StratifiedKFold(n_splits=10)\n",
    "#estimator = LogisticRegression(C = best_params.get('C'), penalty=best_params.get('penalty'))\n",
    "\n",
    "'''\n",
    "crossval = cross_validate(estimator, X, y, cv=kfold, scoring = scoring, verbose=2, n_jobs = -1)\n",
    "print(crossval)\n",
    "acc = crossval.get('test_accuracy')\n",
    "f1 = crossval.get('test_f1_score')\n",
    "print(\"Accuracy:\", round(np.mean(acc), 4), \"+-\", \n",
    "      round(np.std(crossval.get('test_accuracy')), 4))\n",
    "print(\"F1 score:\", round(np.mean(f1), 4), \"+-\",\n",
    "      round(np.std(crossval.get('test_f1_score')), 4))\n",
    "'''\n",
    "acc_list = [] \n",
    "wtd_f1_list = []\n",
    "macro_f1_list = []\n",
    "conf_mat = np.zeros((2,2))\n",
    "for train_index, test_index in kfold.split(X, y): \n",
    "  X_train, X_test = X[train_index], X[test_index] \n",
    "  y_train, y_test = y[train_index], y[test_index]\n",
    "  estimator = LogisticRegression(C = best_params.get('C'), penalty=best_params.get('penalty'))\n",
    "  estimator.fit(X_train, y_train) \n",
    "  y_pred = estimator.predict(X_test)\n",
    "  acc = accuracy_score(y_test, y_pred)\n",
    "  acc_list.append(acc)\n",
    "  wtd_f1 = f1_score(y_test, y_pred, average='weighted')\n",
    "  macro_f1 = f1_score(y_test, y_pred, average='macro')\n",
    "  wtd_f1_list.append(wtd_f1)\n",
    "  macro_f1_list.append(macro_f1)\n",
    "  cm = confusion_matrix(y_test, y_pred, labels=[0,1])\n",
    "  conf_mat = conf_mat+cm\n",
    "\n",
    "conf_mat /= conf_mat.sum(axis=1, keepdims=True)\n",
    "acc_mean = round(np.mean(acc_list),4)\n",
    "acc_std = round(np.std(acc_list),4)\n",
    "wtd_f1_mean = round(np.mean(wtd_f1_list),4)\n",
    "wtd_f1_std = round(np.std(wtd_f1_list),4)\n",
    "macro_f1_mean = round(np.mean(macro_f1_list),4)\n",
    "macro_f1_std = round(np.std(macro_f1_list),4)\n",
    "print('Accuracy:',acc_mean,'+-',acc_std)\n",
    "print('Weighted FScore:',wtd_f1_mean,'+-',wtd_f1_std)\n",
    "print('Macro FScore:',macro_f1_mean,'+-',macro_f1_std)\n",
    "#conf_mat = plot_confusion_matrix(estimator, X_test, y_test, cmap=plt.cm.Blues)   \n",
    "#conf_mat.ax_.set_title('Conf_mat - Tree - HV vs LV - Full data')\n",
    "#fig, ax = plt.subplots(figsize=(10,8))\n",
    "ax = sns.heatmap(conf_mat, cmap=\"YlGnBu\", annot=True, fmt='.4f', vmin=0, \n",
    "                 vmax=1, annot_kws={\"fontsize\":12})\n",
    "ax.set_yticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_xticklabels(['NC','PD'],rotation = 0)\n",
    "ax.set_title(cm_title)\n",
    "ax.get_figure().savefig(filename)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "EEG_new.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}