Unexpectedly different behavior for factors/dummy variables between parsnip and workflows

[juliasilge]

When training what seems like the same model (same model specification, same formula, same data) using parsnip vs. using workflows, it is surprising to see different results. I found this behavior quite unexpected, especially what workflows did.

Some options to reduce user surprise 😮 would be more clarity in the functions either in parsnip, in workflows, or both.

lm(Sepal.Length ~ ., iris)
#> 
#> Call:
#> lm(formula = Sepal.Length ~ ., data = iris)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            2.1713             0.4959             0.8292            -0.3152  
#> Speciesversicolor   Speciesvirginica  
#>           -0.7236            -1.0235

library(parsnip)
lm_spec <- linear_reg() %>%
  set_engine(engine = "lm") 

## parsnip version looks the same as lm
lm_spec %>%
  fit(Sepal.Length ~ ., data = iris)
#> parsnip model object
#> 
#> Fit time:  2ms 
#> 
#> Call:
#> stats::lm(formula = formula, data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            2.1713             0.4959             0.8292            -0.3152  
#> Speciesversicolor   Speciesvirginica  
#>           -0.7236            -1.0235

## workflows version has made a different choice about dummy variables
library(workflows)
workflow() %>%
  add_model(lm_spec) %>%
  add_formula(Sepal.Length ~ .) %>%
  fit(data = iris)
#> ══ Workflow [trained] ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════
#> Preprocessor: Formula
#> Model: linear_reg()
#> 
#> ── Preprocessor ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Sepal.Length ~ .
#> 
#> ── Model ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> 
#> Call:
#> stats::lm(formula = formula, data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            1.1478             0.4959             0.8292            -0.3152  
#>     Speciessetosa  Speciesversicolor   Speciesvirginica  
#>            1.0235             0.2999                 NA

^{Created on 2020-02-06 by the reprex package (v0.3.0)}

[LaugeGregers]

This behaviour isn't a part of parsnip, but rather a part of workflows and hardhat.

When adding a formula to a workflow with workflows::add_formula it is possible to provide a blueprint from hardhat.

The documentation of workflows::add_formula states that if blueprint = NULL, then hardhat::default_formula_blueprint is called, which has the default arguments intercept = FALSE, allow_novel_levels = FALSE, indicators = TRUE.

The documentation for hardhat::default_formula_blueprint explains that the indicators-argument is what makes factors into dummy variables.

Below some ugly and some nicer code shows the behaviour.

library(parsnip)
library(workflows)
lm_spec <- linear_reg() %>%
  set_engine(engine = "lm") 
tmp <- workflow() %>%
  add_model(lm_spec) %>%
  add_formula(Sepal.Length ~ .) 

# Uses dummy variables 
tmp$pre$actions$formula$blueprint$indicators <- TRUE # Default value
fit(tmp, data = iris) 
#> ══ Workflow [trained] ════════════════════════════════════════════════════════════════════════
#> Preprocessor: Formula
#> Model: linear_reg()
#> 
#> ── Preprocessor ──────────────────────────────────────────────────────────────────────────────
#> Sepal.Length ~ .
#> 
#> ── Model ─────────────────────────────────────────────────────────────────────────────────────
#> 
#> Call:
#> stats::lm(formula = formula, data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            1.1478             0.4959             0.8292            -0.3152  
#>     Speciessetosa  Speciesversicolor   Speciesvirginica  
#>            1.0235             0.2999                 NA

# Uses factors
tmp$pre$actions$formula$blueprint$indicators <- FALSE
fit(tmp, data = iris) 
#> ══ Workflow [trained] ════════════════════════════════════════════════════════════════════════
#> Preprocessor: Formula
#> Model: linear_reg()
#> 
#> ── Preprocessor ──────────────────────────────────────────────────────────────────────────────
#> Sepal.Length ~ .
#> 
#> ── Model ─────────────────────────────────────────────────────────────────────────────────────
#> 
#> Call:
#> stats::lm(formula = formula, data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            2.1713             0.4959             0.8292            -0.3152  
#> Speciesversicolor   Speciesvirginica  
#>           -0.7236            -1.0235

# An easier way to use factors instead of dummy variables
tmp2 <- workflow() %>%
  add_model(lm_spec) %>%
  add_formula(Sepal.Length ~ ., 
              blueprint = hardhat::default_formula_blueprint(indicators = FALSE)) 
fit(tmp2, data = iris)
#> ══ Workflow [trained] ════════════════════════════════════════════════════════════════════════
#> Preprocessor: Formula
#> Model: linear_reg()
#> 
#> ── Preprocessor ──────────────────────────────────────────────────────────────────────────────
#> Sepal.Length ~ .
#> 
#> ── Model ─────────────────────────────────────────────────────────────────────────────────────
#> 
#> Call:
#> stats::lm(formula = formula, data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            2.1713             0.4959             0.8292            -0.3152  
#> Speciesversicolor   Speciesvirginica  
#>           -0.7236            -1.0235

^{Created on 2020-05-16 by the reprex package (v0.3.0)}

[juliasilge]

Yep, we are aware of where the difference arises. Current thoughts on resolving the unexpected differences are in #290.

[juliasilge]

We addressed the issue with indicator/dummy variables in #319 and tidymodels/workflows#51. Next, we need to address the difference in one-hot encoding vs. indicator/dummy variables (i.e. the intercept handling).

[juliasilge]

This has been closed in #332 and tidymodels/workflows#53.

library(parsnip)

lm_spec <- linear_reg() %>%
  set_engine(engine = "lm") 

lm_spec %>%
  fit(Sepal.Length ~ ., data = iris)
#> parsnip model object
#> 
#> Fit time:  5ms 
#> 
#> Call:
#> stats::lm(formula = Sepal.Length ~ ., data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            2.1713             0.4959             0.8292            -0.3152  
#> Speciesversicolor   Speciesvirginica  
#>           -0.7236            -1.0235

library(workflows)

workflow() %>%
  add_model(lm_spec) %>%
  add_formula(Sepal.Length ~ .) %>%
  fit(data = iris)
#> ══ Workflow [trained] ═══════════════════════════════════════
#> Preprocessor: Formula
#> Model: linear_reg()
#> 
#> ── Preprocessor ─────────────────────────────────────────────
#> Sepal.Length ~ .
#> 
#> ── Model ────────────────────────────────────────────────────
#> 
#> Call:
#> stats::lm(formula = ..y ~ ., data = data)
#> 
#> Coefficients:
#>       (Intercept)        Sepal.Width       Petal.Length        Petal.Width  
#>            2.1713             0.4959             0.8292            -0.3152  
#> Speciesversicolor   Speciesvirginica  
#>           -0.7236            -1.0235

^{Created on 2020-07-02 by the reprex package (v0.3.0.9001)}

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.