R package modules have priority over user-installed packages

[zao]

R has multiple variables that control the library path, R_LIBS, R_LIBS_USER and R_LIBS_SITE.

The path can be inspected with .libPaths() and is made up of in order of the entries in R_LIBS, R_LIBS_USER and R_LIBS_SITE and last the R installation itself.

Our R package modules currently prepend their path to R_LIBS similar to how PYTHONPATH is handled, but this means that the user's directory (typically specified in ~/.Renviron or the default) is not at the front of the library path.

This means that users cannot override system-installed packages and more importantly, install.packages only ever considers the first entry in the library path unless you explicitly specify a lib= to install to. Some installation functions like install_github doesn't even have that customization point.

I suggest that we alter our R packages to prepend to R_LIBS_SITE instead, it seems to yield the preferred order for an end-user.

Our R installations do not have a bundled site-library directory so the environment variable is currently unused.

Upstream documentation on how the paths are constructed: https://stat.ethz.ch/R-manual/R-devel/library/base/html/libPaths.html

[zao]

Current state:

b-an02 [~]$ R --quiet --no-save --no-restore -e '.libPaths()'
> .libPaths()
[1] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/R-bundle-Bioconductor/3.11-R-4.0.0"
[2] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/ncdf4/1.17-R-4.0.0"                
[3] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/rgdal/1.5-16-R-4.0.0"              
[4] "/afs/hpc2n.umu.se/home/l/lars/Public/R-packages-4.0.0"                                                                
[5] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/R/4.0.0/lib/R/library"             

Swapping the variables to use R_LIBS_SITE instead:

b-an02 [~]$ env R_LIBS_SITE=$R_LIBS R_LIBS= R --quiet --no-save --no-restore -e '.libPaths()'
> .libPaths()
[1] "/afs/hpc2n.umu.se/home/l/lars/Public/R-packages-4.0.0"                                                                
[2] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/R-bundle-Bioconductor/3.11-R-4.0.0"
[3] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/ncdf4/1.17-R-4.0.0"                
[4] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/rgdal/1.5-16-R-4.0.0"              
[5] "/cvmfs/ebsw.hpc2n.umu.se/amd64_ubuntu1604_bdw/software/MPI/GCC/9.3.0/OpenMPI/4.0.3/R/4.0.0/lib/R/library"             

[ocaisa]

Has the behaviour always been like this? We could maintain backwards compatibility by putting in a version check on which envvar to use (but I wonder if that is overkill).

@boegel I can't see a reason why we wouldn't do this.

[akesandgren]

This is clearly something that should be fixed. Does anyone have any objections to changing this?

[akesandgren]

And at least R/3.5.1 and forward has this behaviour.