exercise-feature-selection.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"**This notebook is an exercise in the [Feature Engineering](https://www.kaggle.com/learn/feature-engineering) course.  You can reference the tutorial at [this link](https://www.kaggle.com/matleonard/feature-selection).**\n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nIn this exercise you'll use some feature selection algorithms to improve your model. Some methods take a while to run, so you'll write functions and verify they work on small samples.\n\nTo begin, run the code cell below to set up the exercise."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set up code checking\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.feature_engineering.ex4 import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then run the following cell. It takes a minute or so to run."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing, metrics\nimport lightgbm as lgb\n\nimport os\n\nclicks = pd.read_parquet('../input/feature-engineering-data/baseline_data.pqt')\ndata_files = ['count_encodings.pqt',\n              'catboost_encodings.pqt',\n              'interactions.pqt',\n              'past_6hr_events.pqt',\n              'downloads.pqt',\n              'time_deltas.pqt',\n              'svd_encodings.pqt']\ndata_root = '../input/feature-engineering-data'\nfor file in data_files:\n    features = pd.read_parquet(os.path.join(data_root, file))\n    clicks = clicks.join(features)\n\ndef get_data_splits(dataframe, valid_fraction=0.1):\n\n    dataframe = dataframe.sort_values('click_time')\n    valid_rows = int(len(dataframe) * valid_fraction)\n    train = dataframe[:-valid_rows * 2]\n    # valid size == test size, last two sections of the data\n    valid = dataframe[-valid_rows * 2:-valid_rows]\n    test = dataframe[-valid_rows:]\n    \n    return train, valid, test\n\ndef train_model(train, valid, test=None, feature_cols=None):\n    if feature_cols is None:\n        feature_cols = train.columns.drop(['click_time', 'attributed_time',\n                                           'is_attributed'])\n    dtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])\n    dvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])\n    \n    param = {'num_leaves': 64, 'objective': 'binary', \n             'metric': 'auc', 'seed': 7}\n    num_round = 1000\n    print(\"Training model!\")\n    bst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], \n                    early_stopping_rounds=20, verbose_eval=False)\n    \n    valid_pred = bst.predict(valid[feature_cols])\n    valid_score = metrics.roc_auc_score(valid['is_attributed'], valid_pred)\n    print(f\"Validation AUC score: {valid_score}\")\n    \n    if test is not None: \n        test_pred = bst.predict(test[feature_cols])\n        test_score = metrics.roc_auc_score(test['is_attributed'], test_pred)\n        return bst, valid_score, test_score\n    else:\n        return bst, valid_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Baseline Score\n\nLet's look at the baseline score for all the features we've made so far."},{"metadata":{"trusted":true},"cell_type":"code","source":"train, valid, test = get_data_splits(clicks)\n_, baseline_score = train_model(train, valid)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1) Which data to use for feature selection?\n\nSince many feature selection methods require calculating statistics from the dataset, should you use all the data for feature selection?\n\nRun the following line after you've decided your answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_1.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we have 91 features we're using for predictions. With all these features, there is a good chance the model is overfitting the data. We might be able to reduce the overfitting by removing some features. Of course, the model's performance might decrease. But at least we'd be making the model smaller and faster without losing much performance."},{"metadata":{},"cell_type":"markdown","source":"### 2) Univariate Feature Selection\n\nBelow, use `SelectKBest` with the `f_classif` scoring function to choose 40 features from the 91 features in the data. "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.feature_selection import SelectKBest, f_classif\nfeature_cols = clicks.columns.drop(['click_time', 'attributed_time','is_attributed'])\ntrain, valid, test = get_data_splits(clicks)\n\n# Create the selector, keeping 40 features\nselector = SelectKBest(f_classif,k=40)\n\n# Use the selector to retrieve the best features\nX_new = selector.fit_transform(train[feature_cols],train['is_attributed'])\n\n# Get back the kept features as a DataFrame with dropped columns as all 0s\nselected_features = pd.DataFrame(selector.inverse_transform(X_new),index=train.index,columns=feature_cols)\n\n# Find the columns that were dropped\ndropped_columns = selected_features.columns[selected_features.var()==0]\n\n# Check your answer\nq_2.check()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Uncomment these lines if you need some guidance\nq_2.hint()\nq_2.solution()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"_ = train_model(train.drop(dropped_columns, axis=1), \n                valid.drop(dropped_columns, axis=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3) The best value of K\n\nWith this method we can choose the best K features, but we still have to choose K ourselves. How would you find the \"best\" value of K? That is, you want it to be small so you're keeping the best features, but not so small that it's degrading the model's performance.\n\nRun the following line after you've decided your answer."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_3.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4) Use L1 regularization for feature selection\n\nNow try a more powerful approach using L1 regularization. Implement a function `select_features_l1` that returns a list of features to keep.\n\nUse a `LogisticRegression` classifier model with an L1 penalty to select the features. For the model, set:\n- the random state to 7,\n- the regularization parameter to 0.1,\n- and the solver to `'liblinear'`.\n\nFit the model then use `SelectFromModel` to return a model with the selected features.\n\nThe checking code will run your function on a sample from the dataset to provide more immediate feedback."},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nfrom sklearn.feature_selection import SelectFromModel\n\ndef select_features_l1(X, y):\n    \"\"\"Return selected features using logistic regression with an L1 penalty.\"\"\"\n    logistic = LogisticRegression(penalty='l1',solver='liblinear',random_state=7,C=0.1).fit(X,y)\n    model = SelectFromModel(logistic,prefit=True)\n    X_new = model.transform(X)\n    selected_features = pd.DataFrame(model.inverse_transform(X_new), \n                                 index=X.index,\n                                 columns=X.columns)\n    selected_columns = selected_features.columns[selected_features.var() != 0]\n    return selected_columns\n\n# Check your answer\nq_4.check()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Uncomment these if you're feeling stuck\n#q_4.hint()\n#q_4.solution()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"n_samples = 10000\nX, y = train[feature_cols][:n_samples], train['is_attributed'][:n_samples]\nselected = select_features_l1(X, y)\n\ndropped_columns = feature_cols.drop(selected)\n_ = train_model(train.drop(dropped_columns, axis=1), \n                valid.drop(dropped_columns, axis=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5) Feature Selection with Trees\n\nSince we're using a tree-based model, using another tree-based model for feature selection might produce better results. What would you do different to select the features using a trees classifier?\n\nRun the following line after you've decided your answer."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_5.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6) Top K features with L1 regularization\n\nHere you've set the regularization parameter `C=0.1` which led to some number of features being dropped. However, by setting `C` you aren't able to choose a certain number of features to keep. What would you do to keep the top K important features using L1 regularization?\n\nRun the following line after you've decided your answer."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_6.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Congratulations on finishing this course! To keep learning, check out the rest of [our courses](https://www.kaggle.com/learn/overview). The machine learning explainability and deep learning courses are great next skills to learn!"},{"metadata":{},"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161443) to chat with other Learners.*"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}