fable vs. forecast ETS model estimation performance

[Stochastic-Man]

I'm seeing a large difference in the performance of the forecast ets function and the fable ETS function when estimating multiple models, surprisingly in favor of forecast. Here's a simple example using the "tourism" data set from fpp3. The inelegant loop in the forecast section is intentional. Apologies for the rest of the code in advance:

# libraries ---------------------------------------------------------------

library(fpp3)
library(forecast)

# fable functions ---------------------------------------------------------

start_time <- Sys.time()

fable_fit <- tourism %>%
  model(
    ets = ETS(Trips)
  )

fable_out <- fable_fit %>%
  forecast(h=4) %>%
  hilo(c(95,80)) %>%
  unpack_hilo(c('95%','80%')) %>%
  as_tibble() %>%
  unite(Key_FC,Region, State, Purpose, Quarter, sep = '@', remove = FALSE)

end_time <- Sys.time()

fable_time <- end_time - start_time
cat("Time for `fable` estimation & forecast:", round(as.numeric(fable_time,units = "secs"),2), "sec\n")


# forecast functions ------------------------------------------------------
tourism_aug <- tourism %>%
    unite(Key,Region, State, Purpose, sep = '@', remove = TRUE)

keys <- unique(tourism_aug$Key)

forecast_out <- tibble()

start_time <- Sys.time()
for(i in 1:length(keys)) {
  ts_single <- tourism_aug %>% 
                filter(Key == keys[i]) %>% 
                select(-Key) %>%
                ts(frequency=4, start=c(1998,1))
  
  forecast_new <- forecast(ets(ts_single[,2]),h=4) %>%
    as_tibble(rownames="index") %>%
    mutate(Key = keys[i])
  
  forecast_out <- forecast_out %>% bind_rows(forecast_new)
}
end_time <- Sys.time()

forecast_time <- end_time - start_time
cat("Time for `forecast` estimation & forecast:", round(as.numeric(forecast_time,units = "secs"),2), "sec\n")


# Comparison --------------------------------------------------------------

comparison_data <- forecast_out %>% 
  unite(Key_FC, Key, index, sep="@", remove = FALSE) %>%
    left_join(fable_out,
            by = c("Key_FC" = "Key_FC"),
            suffix = c(".forecast",".fable")) %>%
  mutate(mean_2 = (`Point Forecast` - .mean)^2) %>%
  mutate(`80%_lower_2` = (`80%_lower`-`Lo 80`)^2) %>%
  mutate(`80%_upper_2` = (`80%_upper`-`Hi 80`)^2) %>%
  mutate(`95%_lower_2` = (`95%_lower`-`Lo 95`)^2) %>%
  mutate(`95%_upper_2` = (`95%_upper`-`Hi 95`)^2) %>%
  arrange(desc(mean_2))

comparison_summary <- comparison_data %>%
  select(Key, Quarter,
                mean_2, `80%_lower_2`, `80%_upper_2`,
                `95%_lower_2`,`95%_upper_2`) %>%
  group_by(Key) %>%
  summarise(MSE_mean = mean(mean_2),
            `MSE_80%_lower` = mean(`80%_lower_2`),
            `MSE_80%_upper` = mean(`80%_upper_2`),
            `MSE_95%_lower` = mean(`95%_lower_2`),
            `MSE_95%_upper` = mean(`95%_upper_2`),
            max_err = max(MSE_mean,
                          `MSE_80%_lower`,
                          `MSE_80%_upper`,
                          `MSE_95%_lower`,
                          `MSE_95%_upper`
            )
  ) %>%
  arrange(desc(max_err))

On my machine, the output is

Time for `fable` estimation & forecast: 106.93 sec
Time for `forecast` estimation & forecast: 37.03 sec

The comparison_summary table seems to indicate that the models are giving the same forecasts, only fable is taking almost 3 times as long as using forecast in a simple loop.

Am I using either of the functions incorrectly? My first thought was that fable ETS is searching a much larger set of models, but on default settings the search space for both algorithms should be the same.

I've cross-posted this to StackOverflow in case this is just a simple error or misunderstanding on my part. If I get an answer there I will close the issue.

Thanks for the great library!

[mitchelloharawild]

I'm able to reproduce this.
The relative performance shouldn't be this bad. I expect fable to be slightly slower than forecast when the model code is identical (due to some added overhead of working with tibbles rather than directly with vectors, and accepting more general time series inputs), but there is something problematic here.

[FinYang]

I had this issue yesterday.
I suspect it has more problem with the forecast (the method in fable, not the package) than the fitting because ETS took 2 hours to estimate on my data set with 4 future workers, and 5 hours to forecast with a sequential process (Memory explodes with more than one process which is worrying on its own).
But running the example above, fable took 53 seconds to estimate, and 55 to forecast, whereas forecast only took 16 seconds for everything. Now I'm not sure.

[AngelPone]

The outputs on one of my machine.

Time for `fable` estimation & forecast: 43.97 sec
Time for `forecast` estimation & forecast: 30.63 sec

By injecting some timing codes in the fabletools package, I found that fabletools (model function) takes 1s to prepare the estimating, 38s to fit the models (check the data, train the ets model) and 2s to prepare the output.

Stepping deeper, I found that ets training in fable (compare_ets and etsmodel function) is 20% slower than the implementation in forecast. I cannot figure out why, because the implementation seems very similar.

BTW, the time difference on my another machine is around 4s, which seems acceptable.