R CMD check --flavor=<flavor>: customized add-on package validation

[HenrikBengtsson]

(adopted from existing Wiki entry)

Background

By default

R CMD check MyPkg

validates the package MyPkg using a (continuously growing) set of tests/checks. In addition to these, we can run

R CMD check --as-cran MyPkg

which will run an additional set of validation tests to assert that the package meets CRAN's requirements before package submission.

When submitting a package to Bioconductor, they require that in addition to R CMD check you also run:

R CMD BiocCheck MyPkg

to run a "set of tests that encompass Bioconductor Best Practices".

Problems / Issues

Although we currently are provided with great mechanisms for validating R packages, we don't have an easy way to extend and/or customize it (beyond hard-coded command-line options and environment variables). This in turn makes it hard for non-R core developers to implement and share additional package validation tests.

Slightly related: The current R CMD check --as-cran hard codes CRAN into the R base distribution. We already have Bioconductor as an additional large-scale package repository. One could imagine others in the future. In other words, there might be a need to decouple CRAN for R base at some point, which would be in line we what we often hear when users ask questions about CRAN and/or R in the wrong forum; "CRAN is not R" and "R is not CRAN".

Wish / Suggestion

One approach could be to extend R CMD check with an option for running add-on validation tests;

R CMD check --flavor=<flavors> MyPkg

For example,

R CMD check --flavor=CRAN MyPkg
R CMD check --flavor=Bioconductor MyPkg
R CMD check --flavor=CRAN,Bioconductor MyPkg
R CMD check --flavor=CRAN,covr,lintr MyPkg

Alternative option names

R CMD check --as=<...> MyPkg
R CMD check --by=<...> MyPkg
R CMD check --flavor=<...> MyPkg
R CMD check --suite=<...> MyPkg
R CMD check --with=<...> MyPkg

How?

A natural place to implement add-on package validation tests is in an R package itself. The most straightforward could be to name the package the same as the flavor, i.e. we would have packages CRAN and Bioconductor. These packages would then need to export a R_CMD_check_flavor() function that can be called by R CMD check --flavor=<flavor> MyPkg.

Things to consider

• Should / could there be a way for an add-on validation package to register or tag itself that it is a validation package. Is R_CMD_check_flavor() good enough?
• Should the flavor be bound to the package name (as suggested above)? An alternative is that a package has an R_CMD_check_get_flavors() function which returns supported "flavors", cf. a vignette build package can provide/register one or more vignette engines.
• What about sub-flavors, e.g. imagine that Bioconductor wish to provide different types of tests depending on package type (software, annotation data, ...) or "current" versus "upcoming" tests. Do we need a special syntax for non-default validation sets, e.g. R CMD check --flavor=CRAN,Bioconductor::upcoming?

[leeper]

Why not have the argument be --as, then it's a simple change from current
--as-cran to --as=cran?
On Mar 8, 2016 6:15 PM, "Henrik Bengtsson" notifications@github.com wrote:

(adopted from existing Wiki
https://github.com/HenrikBengtsson/Wishlist-for-R/wiki entry)
Background

By default

R CMD check MyPkg

validates the package MyPkg using a (continuously growing) set of
tests/checks. In addition to these, we can run

R CMD check --as-cran MyPkg

which will run an additional set of validation tests to assert that the
package meets CRAN's requirements
https://cran.r-project.org/web/packages/policies.html before package
submission.

When submitting a package to Bioconductor https://bioconductor.org/, they
require
https://www.bioconductor.org/developers/package-guidelines/#correctness
that in addition to R CMD check you also run:

R CMD BiocCheck MyPkg

to run a "set of tests that encompass Bioconductor Best Practices".
Problems / Issues

Although we currently are provided with great mechanisms for validating R
packages, we don't have an easy way to extend and/or customize it (beyond
hard-coded command-line options and environment variables). This in turn
makes it hard for non-R core developers to implement and share additional
package validation tests.

Slightly related: The current R CMD check --as-cran hard codes CRAN into
the R base distribution. We already have Bioconductor as an additional
large-scale package repository. One could imagine others in the future. In
other words, there might be a need to decouple CRAN for R base at some
point, which would be in line we what we often hear when users ask
questions about CRAN and/or R in the wrong forum; "CRAN is not R" and "R is
not CRAN".
Wish / Suggestion

One approach could be to extend R CMD check with an option for running
add-on validation tests;

R CMD check --flavor= MyPkg

For example,

R CMD check --flavor=CRAN MyPkg
R CMD check --flavor=Bioconductor MyPkg
R CMD check --flavor=CRAN,Bioconductor MyPkg
R CMD check --flavor=CRAN,covr,lintr MyPkg

How?

A natural place to implement add-on package validation tests is in an R
package itself. The most straightforward could be to name the package the
same as the flavor, i.e. we would have packages CRAN and Bioconductor.
These packages would then need to export a R_CMD_check_flavor() function
that can be called by R CMD check --flavor= MyPkg.
Things to consider

• Should / could there be a way for an add-on validation package to
  register or tag itself that it is a validation package. Is
  R_CMD_check_flavor() good enough?
• Should the flavor be bound to the package name (as suggested above)?
  An alternative is that a package has an R_CMD_check_get_flavors()
  function which returns supported "flavors", cf. a vignette build package
  can provide/register one or more vignette engines.
• What about sub-flavors, e.g. imagine that Bioconductor wish to
  provide different types of tests depending on package type (software,
  annotation data, ...) or "current" versus "upcoming" tests. Do we need a
  special syntax for non-default validation sets, e.g. R CMD check
  --flavor=CRAN,Bioconductor::upcoming?

—
Reply to this email directly or view it on GitHub
#16.

[HenrikBengtsson]

@leeper, using --as could certainly be an option (pun intended). Though, doesn't that imply "as someone else does it" and somewhat also that "package is about to submitted somewhere else"? It makes sense for --as=CRAN and --as=Bioconductor, but can't we imagine other scenarios too that doesn't imply we're "imitating someone else"? Not saying --flavor is much better.

I've update top post with your proposal (and some others).

[gmbecker]

What about --suite. It is a little murkier with respect to settings, but
clearer that different sets of tests are being run. Could even be separated
from environment settings.

We could imagine a config file format/mechanism to define these suites,
allowing customization and testing moving forward.

On Tue, Mar 8, 2016 at 10:28 AM, Henrik Bengtsson notifications@github.com
wrote:

@leeper https://github.com/leeper, using --as could certainly be an
option (pun intended). Though, doesn't that imply "as someone else does it"
and somewhat also that "package is about to submitted somewhere else"? It
makes sense for --as=CRAN and --as=Bioconductor, but can't imagine other
scenarios too that doesn't imply we're "imitating someone else"? Not saying
--flavor is much better.

I've update top post with your proposal (and some others).

—
Reply to this email directly or view it on GitHub
#16 (comment)
.

Gabriel Becker, PhD
Associate Scientist (Bioinformatics)
Genentech Research