check() does not set NOT_CRAN by default

[JBGruber]

I knew about this for a while but was surprised to learn that it is apparently not on purpose. As mentioned here, devtools::check() does not seem to automatically set Sys.setenv(NOT_CRAN = "true") while Build -> Check Package in RStudio does.

I created a minimal test package with a test that is supposed to fail locally but not on CRAN:

test_that("multiplication works", {
  skip_on_cran()
  expect_equal(2 * 3, 4)
})

However when I run this via devtools::check(), it runs without error, presumably because NOT_CRAN isn't set. If I use Sys.setenv(NOT_CRAN = "true") before devtools::check(), everything works as intended.

Tested on Ubuntu 18.04 and Win 10, R 3.6.1 with devtools 2.2.0 and 2.2.1.9000.

Console output (slightly redacted for readability):

> devtools::check()
 Building  test 
Setting env vars:
● CFLAGS    : -Wall -pedantic -fdiagnostics-color=always
● CXXFLAGS  : -Wall -pedantic -fdiagnostics-color=always
● CXX11FLAGS: -Wall -pedantic -fdiagnostics-color=always

✔  checking for file ‘/home/johannes/Documents/Github/test/DESCRIPTION’ ...
  preparing ‘test’:
✔  checking DESCRIPTION meta-information ...
  checking for LF line-endings in source and make files and shell scripts
  checking for empty or unneeded directories
  building ‘test_0.1.0.tar.gz’
   
 Checking  test 
Setting env vars:
● _R_CHECK_CRAN_INCOMING_USE_ASPELL_: TRUE
● _R_CHECK_CRAN_INCOMING_REMOTE_    : FALSE
● _R_CHECK_CRAN_INCOMING_           : FALSE
● _R_CHECK_FORCE_SUGGESTS_          : FALSE
 R CMD check 
  using log directory ‘/tmp/RtmpD5vNsW/test.Rcheck’ (372ms)
  using R version 3.6.1 (2019-07-05)
  using platform: x86_64-pc-linux-gnu (64-bit)
  using session charset: UTF-8
  using options ‘--no-manual --as-cran’
✔  checking for file ‘test/DESCRIPTION’
  checking extension type ... Package
  this is package ‘test’ version ‘0.1.0’
  package encoding: UTF-8
✔  checking package namespace information ...
✔  checking package dependencies (1.3s)
✔  checking if this is a source package
✔  checking if there is a namespace
✔  checking for executable files ...
✔  checking for hidden files and directories
✔  checking for portable file names
✔  checking for sufficient/correct file permissions
✔  checking serialization versions
✔  checking whether package ‘test’ can be installed (1.1s)
✔  checking installed package size ...
✔  checking package directory ...
✔  checking for future file timestamps (661ms)
✔  checking DESCRIPTION meta-information (359ms)
✔  checking top-level files
✔  checking for left-over files
✔  checking index information
✔  checking package subdirectories ...
✔  checking R files for non-ASCII characters ...
✔  checking R files for syntax errors ...
✔  checking whether the package can be loaded ...
✔  checking whether the package can be loaded with stated dependencies ...
✔  checking whether the package can be unloaded cleanly ...
✔  checking whether the namespace can be loaded with stated dependencies ...
✔  checking whether the namespace can be unloaded cleanly ...
✔  checking loading without being on the library search path ...
✔  checking dependencies in R code ...
✔  checking S3 generic/method consistency (495ms)
✔  checking replacement functions ...
✔  checking foreign function calls ...
✔  checking R code for possible problems (2s)
✔  checking Rd files ...
✔  checking Rd metadata ...
✔  checking Rd line widths ...
✔  checking Rd cross-references ...
✔  checking for missing documentation entries ...
✔  checking for code/documentation mismatches (423ms)
✔  checking Rd \usage sections (582ms)
✔  checking Rd contents ...
✔  checking for unstated dependencies in examples ...
✔  checking examples (577ms)
✔  checking for unstated dependencies in ‘tests’ ...
  checking tests ...
✔  Running ‘testthat.R’ (353ms)
   
   
 R CMD check results  test 0.1.0 
Duration: 11.2s

0 errors ✔ | 0 warnings ✔ | 0 notes ✔
> 
> Sys.setenv(NOT_CRAN = "true")
> 
> devtools::check()
 Building  test 
Setting env vars:
● CFLAGS    : -Wall -pedantic -fdiagnostics-color=always
● CXXFLAGS  : -Wall -pedantic -fdiagnostics-color=always
● CXX11FLAGS: -Wall -pedantic -fdiagnostics-color=always

✔  checking for file ‘/home/johannes/Documents/Github/test/DESCRIPTION’ ...
  preparing ‘test’:
✔  checking DESCRIPTION meta-information ...
  checking for LF line-endings in source and make files and shell scripts
  checking for empty or unneeded directories
  building ‘test_0.1.0.tar.gz’
   
 Checking  test 
Setting env vars:
● _R_CHECK_CRAN_INCOMING_USE_ASPELL_: TRUE
● _R_CHECK_CRAN_INCOMING_REMOTE_    : FALSE
● _R_CHECK_CRAN_INCOMING_           : FALSE
● _R_CHECK_FORCE_SUGGESTS_          : FALSE
 R CMD check 
  using log directory ‘/tmp/RtmpD5vNsW/test.Rcheck’
  using R version 3.6.1 (2019-07-05)
  using platform: x86_64-pc-linux-gnu (64-bit)
  using session charset: UTF-8
  using options ‘--no-manual --as-cran’
✔  checking for file ‘test/DESCRIPTION’
  checking extension type ... Package
  this is package ‘test’ version ‘0.1.0’
  package encoding: UTF-8
✔  checking package namespace information ...
✔  checking package dependencies (996ms)
✔  checking if this is a source package
✔  checking if there is a namespace
✔  checking for executable files ...
✔  checking for hidden files and directories
✔  checking for portable file names
✔  checking for sufficient/correct file permissions ...
✔  checking serialization versions
✔  checking whether package ‘test’ can be installed (1.1s)
✔  checking installed package size ... 
✔  checking package directory ...
✔  checking for future file timestamps (664ms)
✔  checking DESCRIPTION meta-information (540ms)
✔  checking top-level files
✔  checking for left-over files
✔  checking index information
✔  checking package subdirectories ...
✔  checking R files for non-ASCII characters ...
✔  checking R files for syntax errors ...
✔  checking whether the package can be loaded ...
✔  checking whether the package can be loaded with stated dependencies ...
✔  checking whether the package can be unloaded cleanly ...
✔  checking whether the namespace can be loaded with stated dependencies ...
✔  checking whether the namespace can be unloaded cleanly ...
✔  checking loading without being on the library search path ...
✔  checking dependencies in R code ...
✔  checking S3 generic/method consistency (437ms)
✔  checking replacement functions ...
✔  checking foreign function calls ...
✔  checking R code for possible problems (1.7s)
✔  checking Rd files ...
✔  checking Rd metadata ...
✔  checking Rd line widths ...
✔  checking Rd cross-references ...
✔  checking for missing documentation entries ...
✔  checking for code/documentation mismatches ...
✔  checking Rd \usage sections (590ms)
✔  checking Rd contents ...
✔  checking for unstated dependencies in examples ...
✔  checking examples (519ms)
✔  checking for unstated dependencies in ‘tests’ ...
  checking tests ...
E  Running ‘testthat.R’
   Running the tests in ‘tests/testthat.R’ failed.
   Last 13 lines of output:
     > library(testthat)
     > library(test)
     > 
     > test_check("test")
      1. Failure: multiplication works (@test-name.R#3)  
     2 * 3 not equal to 4.
     1/1 mismatches
     [1] 6 - 4 == 2
     
      testthat results  
     [ OK: 0 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 1 ]
     1. Failure: multiplication works (@test-name.R#3) 
     
     Error: testthat unit tests failed
     Execution halted
   
   See
     ‘/tmp/RtmpD5vNsW/test.Rcheck/00check.log’
   for details.
   
 R CMD check results  test 0.1.0 
Duration: 10.3s

❯ checking tests ...
  See below...

 Test failures  testthat 

> library(testthat)
> library(test)
> 
> test_check("test")
 1. Failure: multiplication works (@test-name.R#3)  
2 * 3 not equal to 4.
1/1 mismatches
[1] 6 - 4 == 2

 testthat results  
[ OK: 0 | SKIPPED: 0 | WARNINGS: 0 | FAILED: 1 ]
1. Failure: multiplication works (@test-name.R#3) 

Error: testthat unit tests failed
Execution halted

1 error ✖ | 0 warnings ✔ | 0 notes ✔

[jimmyday12]

Not sure if relevant in here but in Hadley's R packages book it mentions the following:

In contrast to R CMD check, devtools::check():
...

• Sets the NOT_CRAN environment variable to TRUE. This allows you to selectively skip tests on CRAN. (See ?testthat::skip_on_cran for details.)

I only found out this wasn't the case after a fair bit of searching and finding this issue so assuming there might be others who think the same. It may be worth asking Hadley to update that part of the book since it is a pretty common resource?

[jennybc]

I agree we need a definitive ruling on this (I think devtools::check() should set NOT_CRAN to TRUE) and, regardless of the ruling, I will make sure R Packages reflects reality.

[JBGruber]

I think it makes more sense if devtools::check() would set NOT_CRAN to TRUE and it seems the current behaviour confuses other users as well. This twitter conversation might be interesting as well.

[timcdlucas]

This is kind of related but not related. I think travis runs check() and therefore assumes you're on CRAN. Although I might be wrong (we got very confused trying to work all this out). Either way, I think travis should be running tests that are in a skip_on_cran block by default.

[jimhester]

travis already sets NOT_CRAN=true irrespective of devtools, as does the RStudio build pane. The only times it won't be set are when you are manually running devtools::check() from the R command line.

[hadley]

Why don't we set it in devtools::check()?

[JBGruber]

Thank you @jimhester, 4663004 solves this for me by setting env_vars = c(NOT_CRAN = "true") by default. I just checked it using devtools::check() on the test package I set up for this (linked above).

I noticed though that the documentation doesn't reflect the change yet (See PR, I just roxygenised, nothing else).

[jimhester]

We do now set it in devtools::check(). I think the intention was always to do so, but as far as I can tell from looking back through the code revisions it was never actually done.