wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wrangling report\n",
    "\n",
    "## Gathering data\n",
    "\n",
    "In this report I have wrangled multiples files with data in common. \n",
    "\n",
    "The three files that I have generated in the **gathering of the data** are:\n",
    "\n",
    "1. **twitter-archive-enhanced.csv**: Data in a CSV format that contains some metadata of the each tweet like the datetime and also the type of dog.\n",
    "2. **image_predictions.tsv**: Data in a TSV format that contains the prediccions of breed of dog in the images of the tweets\n",
    "3. **tweet_json.txt**: Data that comes from the API of Twitter with the favorites and retweets of each tweet\n",
    "\n",
    "## Assessing data\n",
    "\n",
    "After gathering all the data I have assessed the data with some commands like head(), info(), etc. and then I defined 8 quality issues and 2 tidiness issues. This issues are the following:\n",
    "\n",
    "### Quality Issues\n",
    "\n",
    "1. Q1 - Tweets without images\n",
    "    - In the dataframe `twitter_data` there are some tweets that they don't have images, so we don't have predicted data for them.\n",
    "\n",
    "<br>\n",
    "\n",
    "2. Q2 - Wrong expanded urls\n",
    "    - In the dataframe `twitter_data` in some cases the url is wrong and also there in some cases there are multiple URL's.\n",
    "    \n",
    "<br>\n",
    "\n",
    "3. Q3 - Wrong names of dogs\n",
    "    - In the dataframe `twitter_data` in some cases the name of the dog is wrong (like \"a\", \"an\", \"by\", etc.) and also there are some that are missing or are defined like \"None\".\n",
    "\n",
    "<br>\n",
    "\n",
    "4. Q4 - Remove retweets\n",
    "    - In the dataframe `twitter_data` there are some tweets that they are retweets from other tweets. We only want the original ratings.\n",
    "\n",
    "<br>\n",
    "\n",
    "5. Q5 - Wrong breed of dogs\n",
    "    - In the dataframe `img_pred` some of the predictions are not a breed of a dog (like for example \"shopping_cart\").\n",
    "\n",
    "<br>\n",
    "\n",
    "6. Q6 - Wrong data type for the timestamp\n",
    "    - In the dataframe `twitter_data` the column `timestamp` has the data type `object` and it should be `datetime64`.\n",
    "\n",
    "<br>\n",
    "\n",
    "7. Q7 - Wrong data type for the tweet id\n",
    "    - In all 3 dataframes the `tweet_id` has the data type `int`, the id's should be a `string`.\n",
    "\n",
    "<br>\n",
    "\n",
    "8. Q8 - Missing values\n",
    "    - In the dataframe `twitter_data` the columns `in_reply_to_status_id`, `in_reply_to_user_id`, `retweeted_status_id`, `retweeted_status_user_id` and `retweeted_status_timestamp` they have a lot of missing values.\n",
    "\n",
    "<br>\n",
    "\n",
    "### Tidiness Issues\n",
    "\n",
    "1. T1 - Multiple columns for the dogo stage\n",
    "    - In the dataframe `twitter_data` the dogo stage is splitted in 4 columns (doggo, floofer, pupper, puppo).\n",
    "\n",
    "<br>\n",
    "\n",
    "2. T2 - Merge all the data\n",
    "    - Merge all 3 dataframes using the `tweet_id` column in a unique dataframe.\n",
    "\n",
    "## Cleaning data\n",
    "\n",
    "In the cleaning part I solved all the quality and tidiness issues that I defined in the assessing part. Even I solved a lot of quality problems, it doesn't mean that all the data is 100% cleaned.\n",
    "\n",
    "In this part I put in practice new technics in how to clean the data that I found googling on Internet and most of the times in StackOverflow. In this cases, I have documented the link in where I found the help in the report."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}