user_itineraries_inefficiency.Rmd

---
title: "User itineraries inefficiency analysis"
output: html_notebook
---


```{r, echo=FALSE, message=FALSE}
library(tidyverse)
library(lubridate)

all_itineraries <- read_csv("data/all_itineraries_metrics-v2.csv",
                            col_types = cols(
                              .default = col_double(),
                              date = col_date(format = ""),
                              user_trip_id = col_character(),
                              itinerary_id = col_integer(),
                              planned_start_time = col_datetime(format = ""),
                              actual_start_time = col_datetime(format = ""),
                              exec_start_time = col_datetime(format = ""),
                              trip_length_bucket = col_character(),
                              hour_of_day = col_integer(),
                              period_of_day = col_character(),
                              weekday = col_character(),
                              day_type = col_character(),
                              itinerary_id_fo = col_integer(),
                              itinerary_id_fs = col_integer()
                            )) %>%
  filter(date != ymd("2017-05-10")) %>%
  mutate(period_of_day = factor(period_of_day, ordered = TRUE, levels = c("early_morning", "morning", "midday", "afternoon","evening", "night")))

user_itineraries_complete <- all_itineraries %>%
  group_by(date, user_trip_id) %>%
  slice(1)

user_itineraries_with_plans <- all_itineraries %>%
  filter(itinerary_id == "0", !is.na(planned_duration_mins))

```
```{r}
# Return the desired percentiles plus the geometric mean
bp.vals <- function(x, probs=c(0, 0.1, 0.5, 0.9, 1), width=0.8) {
  r <- quantile(x, probs=probs , na.rm=TRUE)
  r <- c(r, width)
  names(r) <- c("ymin", "lower", "middle", "upper", "ymax", "width")
  r
}
```

### Ditribuição de $I_o$

```{r}
summary(user_itineraries_complete$io)

ggplot(data = user_itineraries_complete, aes(x = io)) +
  geom_histogram()
```

### Distribuição de $I_s$

```{r}
summary(user_itineraries_complete$is)

ggplot(data = user_itineraries_complete, aes(x = is)) +
  geom_histogram()
```

### Distribuição de $I_c$

```{r}
summary(user_itineraries_with_plans$ic)

ggplot(data = user_itineraries_with_plans, aes(x = ic)) +
  geom_histogram()
```

## Some analysis


We can use $I_c$ as grouping variable. When it values is greter than 0 it means that the user choice was not optimal. If it's 0 we might assume that the user choice was optimal given all sources of information normaly available. Therefore, if $I_c$ is 0 and $I_o$ isn't, one could infer that the inefficiency of the user observed itinerary happend because of some system inefficiency.

We can see that everytime a user chooses the best planned itinerary, user observerd itinerary is the fastest. This represents the normal functioning of the system. User chooses based on scheduled times, the system perfoms as scheduled leading to a fulfilled user choice. There isn't much to look in this case.

```{r}
user_choose_best <- user_itineraries_with_plans %>%
  filter(ic == 0)

user_choose_best_go_wrong <- user_choose_best %>% 
  filter(io != 1)

user_choose_best.long <- user_choose_best %>% gather("metric", "value", io:is)
user_choose_best_go_wrong.long <- user_choose_best_go_wrong %>% gather("metric", "value", io:is)

```
```{r}
ggplot(data = user_choose_best %>% filter(io == 1) , aes(x = is)) +
  geom_histogram()
```

The other situation is when user chooses the best planned itinerary but because of system inefficiency, the user observed itinerary is not the fastest one (room for improvment). 

```{r}
summary(user_choose_best_go_wrong$is)
```

```{r}
is_by_bucket <- ggplot(data = user_choose_best_go_wrong, aes(x = trip_length_bucket, y = io, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  xlab("Trip length (minutes)") +
  ylab("User choice inefficiency") + 
  guides(fill=FALSE) +
  theme_bw()

plot(is_by_bucket)
```
```{r}
is_by_day <- ggplot(data = user_choose_best_go_wrong, aes(x = day_type, y = io, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  xlab("Day type") +
  ylab("User choice inefficiency") + 
  guides(fill=FALSE) +
  theme_bw()

plot(is_by_day)
```

```{r}

ggplot(data = user_choose_best_go_wrong.long, aes(x = trip_length_bucket, y = value, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  facet_grid(~metric) +
  xlab("Trip length (minutes)") +
  ylab("Ineffciency") + 
  guides(fill=FALSE) +
  theme_bw()

is_by_hour <- ggplot(data = user_choose_best_go_wrong, aes(x = factor(hour_of_day), y = is)) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5)

is_by_period <- ggplot(data = user_choose_best_go_wrong, aes(x = period_of_day, y = is)) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5)

multiplot(is_by_bucket, is_by_hour, is_by_period, cols = 2)
```
#### Scatter plot of Is x Io

```{r}
ggplot(data = user_choose_best, aes(x = is, y = io)) +
  geom_point()

ggplot(data = user_choose_best_go_wrong, aes(x = is, y = io)) +
  geom_point()
```


### When the user does not choose the best suggestion

```{r}
user_choose_not_best <- user_itineraries_with_plans %>%
  filter(ic != 0)

user_choose_not_best_go_right <- user_choose_not_best %>%
  filter(io == 1)

user_choose_not_best.long <- user_choose_not_best %>% gather("metric", "value", io:is)
user_choose_not_best_go_right.long <- user_choose_not_best_go_right %>% gather("metric", "value", io:is)
```
```{r}

is_io_by_bucket_ic1 <- ggplot(data = user_choose_not_best.long, aes(x = trip_length_bucket, y = value, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  facet_grid(~metric) +
  xlab("Trip length (minutes)") +
  ylab("Ineffciency") + 
  guides(fill=FALSE) +
  theme_bw()

ggplot(data = user_choose_not_best.long, aes(x = factor(hour_of_day), y = value, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  facet_grid(~metric) +
  xlab("Hour") +
  ylab("Ineffciency") + 
  guides(fill=FALSE) +
  theme_bw()

ggplot(data = user_choose_not_best.long, aes(x = period_of_day, y = value, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip()+
  facet_grid(~metric) +
  xlab("Period of day") +
  ylab("Ineffciency") + 
  guides(fill=FALSE) +
  theme_bw()
```
```{r}
ggplot(data = user_choose_not_best_go_right, aes(x = trip_length_bucket, y = is, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  xlab("Trip length (minutes)") +
  ylab("System ineffciency (Is)") + 
  guides(fill=FALSE) +
  theme_bw()

ggplot(data = user_choose_not_best_go_right, aes(x = factor(hour_of_day), y = is, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  xlab("Hour") +
  ylab("System ineffciency (Is)") + 
  guides(fill=FALSE) +
  theme_bw()

ggplot(data = user_choose_not_best_go_right, aes(x = period_of_day, y = is, fill="#fc8d62")) +
  stat_summary(fun.data=bp.vals, geom="boxplot") +
  scale_y_log10(breaks = 1:5) +
  coord_flip() +
  xlab("Period of day") +
  ylab("System ineffciency (Is)") + 
  guides(fill=FALSE) +
  theme_bw()

```
```{r}
sp_is_io <- ggplot(data = user_choose_not_best, aes(x = is, y = io)) +
  geom_point() +
  scale_x_log10(breaks = 0:5, name = "log(is)") +
  scale_y_log10(breaks = 0:5, name = "log(io)")
sp_is_io

ggplot(data = user_choose_not_best_go_right, aes(x = is, y = io)) +
  geom_point()
```