tests are run twice when R CMD check

[romainfrancois]

This is a ridiculous consequence of having the unit test vignette.

* checking tests ...
  Running ‘doRUnit.R’

* checking running R code from vignettes ...
   ‘Rcpp-attributes.Rnw’ ... OK
   ‘Rcpp-extending.Rnw’ ... failed
   ‘Rcpp-FAQ.Rnw’ ... failed
   ‘Rcpp-modules.Rnw’ ... OK
   ‘Rcpp-package.Rnw’ ... OK
   ‘Rcpp-quickref.Rnw’ ... OK
   ‘Rcpp-sugar.Rnw’ ... OK
   ‘Rcpp-introduction.Rnw’ ... OK
   ‘Rcpp-unitTests.Rnw’ ...

We need to dump Rcpp-unitTests.Rnw.

[eddelbuettel]

As I recall, the "ridiculous" vignette was your idea. But I digress...

With inst/doc/, we got by not rebuilding each pdf from each Rnw (via our Makefile trick). So this (and other) vignettes were not rebuilt.

With vignettes/, everything gets rebuilt each time. We could add another test / variable to skip the test, or some other trick -- it is after all just a code layer, and it is now straight in the Rnw file.

I still think it's rather nice to ship with a record of tests at time of build.

[romainfrancois]

My idea. Bad idea still.

You for some reason want to keep it. Fine; I've assigned this task to you.

[eddelbuettel]

So tell me what is tolerable or not? "Shorter" test vignette during test builds, and full during release? None ever? None if Sys.getenv("USER")=="romain"?

I like the vignette, but am open to doing things differently. While I'd rather avoid painting the bike shed (ie swapping RUnit for testthat for no gain in functionality) we could think about where / how we want us and / or other developers to run the tests. During build? From the installed package?

[romainfrancois]

Whatever that makes R CMD check not run the tests twice. Tests should run because they are triggered by tests/doRUnit.R not because R CMD check wants to run the code that lives in a vignette.

My preferred option would be to eliminate the vignette entirely. I bet nobody is reading this document. But you insist on keeping it. Then please do whatever environment variable thing.

As for testthat I have not requested that we switched to that.

[eddelbuettel]

Let me expand my question. Unit tests, as we once set them up, run

• during R CMD check via tests/
• during R CMD build via vignettes
• via standard RUnit interfaces including from inst/tests/ in the sources
• from the installed package via the undocumented Rcpp:::test() [ seems broken right now even with Sys.setenv("RunAllRcppTests"="yes") set ]
• possibly other ways I have forgotten

Are you saying that just R CMD check should be suppressed in the vignette build? Maybe we can test whether were triggered by check rather than build.

[romainfrancois]

I'm saying R CMD check should not run the unit tests as part of checking the vignette code.

The simplest way to deal with that is to eliminate the vignette code and therefore the vignette. The only thing this vignette shows is: Look, all the tests passed on Dirk's machine when he built the package.

[eddelbuettel]

Nope. Runs for whoever run 'R CMD build' on the sources, eg whoever builds a tarball from whichever fork or copy. Because that's how the vignette is designed.

Anyway I don't have time right now for your beloved issue ticket ping pong. Maybe after I get back from U Kansas.

[eddelbuettel]

By not forcing the complete run of tests ie running in CRAN mode with just a few tests, we get (and Version: is now 0.10.6.1 here) we get

• checking tests ...
  Running ‘doRUnit.R’ [198s/188s]
  [198s/189s] OK
• checking for unstated dependencies in vignettes ... OK
• checking package vignettes in ‘inst/doc’ ... OK
• checking running R code from vignettes ...
  ‘Rcpp-FAQ.Rnw’ ... [18s/16s] OK
  ‘Rcpp-attributes.Rnw’ ... OK
  ‘Rcpp-extending.Rnw’ ... OK
  ‘Rcpp-modules.Rnw’ ... OK
  ‘Rcpp-package.Rnw’ ... OK
  ‘Rcpp-quickref.Rnw’ ... OK
  ‘Rcpp-sugar.Rnw’ ... OK
  ‘Rcpp-introduction.Rnw’ ... OK
  ‘Rcpp-unitTests.Rnw’ ... OK
  [32s/28s] OK
• checking re-building of vignette outputs ... [38s/40s] OK
• checking PDF version of manual ... OK

NOTE: There was 1 note.
See
‘/home/edd/git/Rcpp.Rcheck/00check.log’
for details.

edd@max:~/git$

Is that acceptable to you?

[romainfrancois]

I have no idea what you mean with cran mode ...

What is acceptable is that when we run R CMD check all tests are run as part of tests/doRUnit.R and no tests are run as part of checking the vignette code.

For example, on travis the line:

‘Rcpp-unitTests.Rnw’ ... OK

should not takes ages as it does now.

[eddelbuettel]

Try to re-read what I quoted above.

The key is the difference between '198s/189s' (unit tests via tests/) and '32s/28s' (via vignettes/).

That difference is 'CRAN mode': When our package version is four digits, tests are run in 'full mode' by setting an environment variable checked in each R file in the tests/ directory. The vignette currently forces the same full mode. Where in 'CRAN mode', as enforced by the CRAN Policy, we are not allowed to run tests for longer than about one minute. As I recall we set all this up well over a year ago and it has been like this for a number of releases. Ok?

So here I changed the code of the unit test to not enforce the full set of tests and hence run in 'limited time CRAN mode'. Its 32 seconds are roughly on par with the code we run in the Rcpp-FAQ vignette.

[romainfrancois]

So now you build a vignette with only a subset of the tests ? I really don't find this useful at all.

[eddelbuettel]

You have a toggle: just set the environment variable and you get the full tests.

Having choices is a good thing. By default it now runs quicker (good) and still allows to run full tests when in release mode (good) by using the same interface we already had for this.

Committed in e95678a