02_engenharia_atributos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Análise, Limpeza e Engenharia de Dados"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "! pip install mlxtend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Bibliotecas úteis\n",
    "\n",
    "import pandas as pd ## Para manipulação dos conjuntos de dados\n",
    "import numpy as np ## Para \n",
    "import chardet ## Biblioteca para lidar com a codificação de caracteres\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "# para escala\n",
    "from mlxtend.preprocessing import minmax_scaling\n",
    "# Para transformação Box-Cox\n",
    "from scipy import stats\n",
    "\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder, MinMaxScaler\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Abrir o conjunto de dados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df = pd.read_csv('bases/summary_of_weather.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizar o conjunto de dados"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tem alguns dados faltando..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limpar o conjunto de dados\n",
    "    --Lidar com dados faltantes, como: __NaN__ ou __None__\n",
    "    --Remover colunas sujas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### -- Lidar com dados faltantes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verificando o número de dados faltantes por coluna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df.isnull().sum().plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verificando a porcentagem de dados faltantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Vamos checar a quantidade de linhas e colunas\n",
    "weather_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Agora, o número total de células no dataset\n",
    "num_total_celulas_weather_df = np.product(weather_df.shape)\n",
    "num_total_celulas_weather_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## E o numero de células com números faltantes\n",
    "num_celulas_faltantes = weather_df.isnull().sum().sum()\n",
    "num_celulas_faltantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Porcentagem de dados faltantes:',  100 * num_celulas_faltantes / num_total_celulas_weather_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remover dados faltantes\n",
    "\n",
    "Vamos começar removendo todas as __linhas__ que contém algum valor faltante"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Eita! Não sobrou nenhuma linha!\n",
    "Vamos tentar remover todas as __colunas__ que têm algum valor faltante:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df_sem_nan = weather_df.dropna(axis=1)\n",
    "weather_df_sem_nan.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos ver a quantidade de dados perdidos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Quantidade de colunas no dataset original:', weather_df.shape[1])\n",
    "print('Quantidade de colunas sem valores faltantes:', weather_df_sem_nan.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sobraram 9 colunas!\n",
    "###### Conseguimos remover dados e continuar com uma parcela do dataset!\n",
    "##### Mas quanto?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Agora, o número total de células no dataset\n",
    "num_total_celulas_weather_df = np.product(weather_df.shape)\n",
    "print('num_total_celulas_weather_df', num_total_celulas_weather_df)\n",
    "\n",
    "## E o numero de células com números faltantes\n",
    "num_celulas_weather_df_sem_nan = np.product(weather_df_sem_nan.shape)\n",
    "print('num_celulas_weather_df_sem_nan', num_celulas_weather_df_sem_nan)\n",
    "\n",
    "print('Porcentagem de dados resultantes:',  100 *  num_celulas_weather_df_sem_nan / num_total_celulas_weather_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Perdemos MUITO dado!\n",
    "\n",
    "###### Vamos olhar mais atentamente para o conjunto de dados.\n",
    "###### Quais dados não existem e quais não foram coletados?\n",
    "\n",
    "-- Se o dado não existe, não faz sentido tentar adivinhar qual valor deveria estar ali\n",
    "\n",
    "-- Se o dado não foi coletado, vale a pena tentar colocar algo na célula desfalcada"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Mostrando 5 amostras do dataset\n",
    "weather_df.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Escolha uma coluna e analise seus dados faltosos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exemplo: coluna 'Snowfall'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "### Ver quantos e quais dados não-nulos\n",
    "pd.value_counts(weather_df['Snowfall'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df.sort_values(by='Date', inplace=True)\n",
    "weather_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df['Snowfall'] = weather_df['Snowfall'].fillna(method='bfill', axis=0).fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.value_counts(weather_df['Snowfall'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Tratar os valores que não são numéricos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df.loc[:, 'Snowfall'].replace(0.0, 0, inplace=True)\n",
    "weather_df.loc[:, 'Snowfall'].replace('0', 0, inplace=True)\n",
    "weather_df.loc[:, 'Snowfall'].replace('#VALUE!', method='bfill', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.value_counts(weather_df['Snowfall'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### -- Remover colunas com todos os valores nulos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df.dropna(how='all', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### -- Aplicar fillna para todo o dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df = weather_df.fillna(method='ffill', axis=0).fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### -- Coluna Precip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.value_counts(weather_df['Precip'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Limpar dados não-numéricos da coluna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df.loc[:,'Precip'].replace('T',0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Mudar tipo da coluna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df.Precip = weather_df.Precip.astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Engenharia de Dados"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Tratamento de Datas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Sua vez! Trate a coluna Date\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Sua vez! monte uma coluna nova com o mês \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Sua vez! monte uma coluna nova com o valor da precipitação do mês anterior\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Escala e Padronização"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Escala"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df_MaxTemp = weather_df.MaxTemp\n",
    "\n",
    "# Colocar em escala de 0 a 1\n",
    "scaled_data = minmax_scaling(weather_df_MaxTemp, columns = [0])\n",
    "\n",
    "# plotar os dados originais e em escala\n",
    "fig, ax=plt.subplots(1,2)\n",
    "sns.distplot(weather_df_MaxTemp, ax=ax[0])\n",
    "ax[0].set_title(\"Dados Originais\")\n",
    "sns.distplot(scaled_data, ax=ax[1])\n",
    "ax[1].set_title(\"Dados em Escala\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sua vez! \n",
    "# Aplique a função scale para a coluna \"MinTemp\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Sua vez! \n",
    "## Identifique as colunas de dados numéricos utilizando a biblioteca pandas e numpy\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Sua vez! Aplique a função scale para todas as colunas de dados numéricos\n",
    "\n",
    "numeric_columns = list(weather_df.select_dtypes(include=[np.number]).columns)\n",
    "numeric_columns.remove('STA')\n",
    "numeric_columns.remove('YR')\n",
    "numeric_columns.remove('MO')\n",
    "numeric_columns.remove('DA')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scaled_columns = minmax_scaling(weather_df[numeric_columns], numeric_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_columns.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalização"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaled_columns['MaxTemp'].hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize the exponential data with boxcox\n",
    "\n",
    "scaled_maxtemp = scaled_columns['MaxTemp']\n",
    "\n",
    "normalized_data = stats.boxcox(scaled_maxtemp[scaled_maxtemp>0])\n",
    "\n",
    "# plot both together to compare\n",
    "fig, ax=plt.subplots(1,2)\n",
    "sns.distplot(scaled_maxtemp, ax=ax[0])\n",
    "ax[0].set_title(\"Dados Originais\")\n",
    "sns.distplot(normalized_data[0], ax=ax[1])\n",
    "ax[1].set_title(\"Dados Normalizados\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sua vez! \n",
    "# Normalize a coluna \"MinTemp\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Sua vez! Normalize todas as colunas de dados numéricos\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Sem usar biblioteca:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize the exponential data with boxcox\n",
    "\n",
    "scaled_maxtemp = scaled_columns['MaxTemp']\n",
    "\n",
    "normalized_data = ( scaled_maxtemp - scaled_maxtemp.mean() ) / scaled_maxtemp.std()\n",
    "\n",
    "# plot both together to compare\n",
    "fig, ax=plt.subplots(1,2)\n",
    "sns.distplot(scaled_maxtemp, ax=ax[0])\n",
    "ax[0].set_title(\"Dados Originais\")\n",
    "sns.distplot(normalized_data, ax=ax[1])\n",
    "ax[1].set_title(\"Dados Normalizados\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One Hot Encoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Via sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df['Month'] = pd.to_datetime(weather_df['Date'], format=\"%Y-%m-%d\").dt.strftime(\"%b\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_hot_encoder = OneHotEncoder(categories='auto')\n",
    "ohe_month = one_hot_encoder.fit_transform(weather_df['Month'].values.reshape(-1,1)).toarray()\n",
    "ohe_month_df = pd.DataFrame(ohe_month, columns=['month_' + str(i) for i in range(ohe_month.shape[1])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe_month_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df_month_ohe = pd.concat([weather_df, ohe_month_df],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df_month_ohe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Via pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe_month = pd.get_dummies(weather_df['Month'], dummy_na=True)\n",
    "weather_df_month_ohe = pd.concat([weather_df, ohe_month],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "weather_df_month_ohe.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Faça OHE para a coluna YR. Escolha um método de sua preferência.\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}