functions for working with Docker

[benmarwick]

Docker is all the rage amongst DevOps types and cloud service providers and seems to offer some concrete advantages for R users in managing dependencies, reproducibility and isolation of the computational environment (@cboettig has a nice paper on this: http://arxiv.org/pdf/1410.0846.pdf). And no doubt you've seen rocker-org by @eddelbuettel and @cboettig, which is a huge contribution to using R and RStudio with docker.

What do you think about these functions being part of devtools:

• add_dockerfile() writes a basic dockerfile to build a docker image that includes the R package and all its dependencies, and maybe edits other files in the package if necessary (maybe .Rbuildignore, cf. https://github.com/benmarwick/Steele_et_al_VR003_MSA_Pigments/blob/master/vignettes/Dockerfile and https://github.com/cboettig/nonparametric-bayes/blob/master/manuscripts/Dockerfile).
• add_circle() writes a circle.yml file (analogous to travis.yml) for circleci's continuous integration service of the docker image that the dockerfile generates (cf. https://github.com/benmarwick/Steele_et_al_VR003_MSA_Pigments/blob/master/circle.yml this also pushes the image to the docker hub when the tests pass)
• build_docker() builds the docker image from the dockerfile made by add_dockerfile(). This would require docker to be installed locally. Analogous to R package build tools in devtools and RStudio.
• push_docker() pushes the docker image to an online repository, the Docker Hub is the big one at the moment, but Google have just announced one also. This would require docker to be installed locally. Analogous to git push in RStudio.

This might be a bit of mission creep for devtools and overlap with the goals of packrat, but it would simplify the reproducible research workflow nicely (for Carl and I at least, not sure that Dirk is a devtools user ;) No doubt there are others using R and Docker that would benefit also. Anyway, what do you reckon?

[rmflight]

Given the already huge amount of stuff in devtools, why wouldn't one package these functions up into their own package?

[gaborcsardi]

Especially that these are only loosely coupled to devtools, and should probably be in the same package together. A dockerdev package or something like that.

[benmarwick]

Yes, fair point. In favour of squeezing them into devtools:

• I'm imagining these docker functions being a logical part of package development, just like travisci has become for many people.
• So when the user thinks 'right, I'm doing package development... where are all the handy functions to save me time and pain?' they don't have to think beyond devtools

But perhaps I'm wrong in assuming others see docker as part of the R package building and testing workflow...

[benmarwick]

On the other other hand, a package with these functions could probably be whipped up pretty quickly by @sckott, @karthik, @cboettig and others at @ropensci for their collection.

[sckott]

@benmarwick @wch is working on harbor https://github.com/wch/harbor - spawned at least in part so we could use it in analogsea - is that what you had in mind?

[benmarwick]

Yes, thanks, I didn't know about that one, looks very relevant. Perhaps that's where the functions I'm thinking of should go.

[hadley]

I think these functions would be better off in another package too

[nuest]

Is anyone here aware of developments in direction since this thread closed?

I want to develop a package for automatically creating Dockerfiles based on R sessions, scripts, or "workspaces" in a folder. I'll rely on sysreqs, harbor. I just want to make sure I'm not redoing something that exists!

I'm happy to take this discussion elsewhere. Feedback on a first draft how I imagine the package to be used is highly welcome: https://github.com/o2r-project/containerit/blob/master/vignettes/basic.Rmd

[gaborcsardi]

I am doing sg somewhat similar right now here:
https://github.com/r-hub/localbuilder (there is an rpackage branch, which will have the scripts as an R package).

The idea of localbuilder is that it will create an environment for your R code, and then it will run it. More specifically, this currently means:

1. take a source R package tarball, and
2. take a docker image (e.g. one of the R-hub images),
3. install all sysreqs of the package on the R-hub image,
4. install the package dependencies of the package on the image,
5. run an arbitrary (R?) command in the new image, currently R CMD check.

I am not using harbor, though.

[benmarwick]

@nuest I'm not aware of anything beyond Gabor's work with r-hub. Your containerit package looks great, seems to be right on what I was discussing above.

[nuest]

@gaborcsardi Thanks for the pointer! I would like to use sysreqs to create the respective distribution's install statements for the libraries within the Dockerfile. Afaics you're using bash scripts, and your use case is quite different.
Therefore I'm not worrying about overlaps anymore and we will continue our developments. We will start with extending functions of dockertest.

@hadley apologies for stealing this issue!

[nuest]

Just a short follow up: We've completed a first prototype, see this blog post. I will also be at useR Brussels and it would be awesome to chat with people here about containers.

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/