improved devcontainer support?

[cboettig]

Looking at the amazing work @eitsupi and @revodavid have done with devcontainer standard (particularly for deploy on codespaces, which looks rather promising in some contexts), I'm wondering if we should support more of that boilerplate here. Not sure if that would b best as an additional image in the stack or (probably better) an additional bash script in rocker_scripts that can bundle up at least some of the additional things that get added to the devcontainer Dockerfile and debian script.

(There's also a lot in there I haven't quite wrapped my head around or am unsure about. e.g. see revodavid/devcontainers-rstudio#1, I'm not sure that we want to use separate users vscode and rstudio on the same VM? seems to create a minor issue for git too?) Thoughts / ideas / objections welcome!

[eitsupi]

First of all, I think it is necessary for you to understand the status of devcontainer (microsoft/vscode-dev-containers#1589).

For a long time, devcontainer could only be handled from the proprietary vscode extension available only on Visual Studio Code, but Microsoft has recently moved this to an open source implementation.
https://github.com/devcontainers/spec
https://github.com/devcontainers/cli

In parallel, they are working on implementing a feature called devcontainer features, which will add functionality on the containers.
https://github.com/devcontainers/features

This is effectively a more generalized version of the Rocker Scripts that exist in this repository.
In other words, I believe that almost all of the contents of this repository could be migrated to the devcontainer features.
(devcontainer features spec still a work in progress and subject to destructive changes / I was going to try this in the near future in my personal repository)
If such a thing could be done, it would more generally accomplish what I was trying to do in #483.

For example, the contents of the script can be imported into any debian-based container by building devcontainer from the following json file.

"name": "my-project-devcontainer",
"image": "rocker/tidyverse",  // Any generic, debian-based image.
"features": {
    "ghcr.io/devcontainers/features/common-utils:1": {
        "version": "latest"
    }
}

As devcontainer featuers spread, shell scripts for installing any software should be able to be shared among arbitrary Debian base images, eliminating the need to add shell scripts in the repository for building containers on a dare.

Also, in relation to devcontainer's move to OSS, it would make sense to move the R definition to the Rocker project, since https://github.com/microsoft/vscode-dev-containers is scheduled to be archived in the near future.

I'm not sure that we want to use separate users vscode and rstudio on the same VM?

Yes, that is not necessary, see revodavid/devcontainers-rstudio#2.
(@revodavid contacted me personally a few months ago and I have had time to look through that repository before)

[cboettig]

Thanks @eitsupi , this is super as always. Definitely helps to understand a bit more of the big picture goals Microsoft has with devcontainers, and glad to see they have open-sourced the project. (Though is the devcontainer.json yet consumed by any third-party tooling (i.e. outside of vscode / codespaces?) I think I understand and like their distinction between 'development' and 'production' containers in the overview page, but still not really sure I see quite how devcontainers fits within the larger ecosystem of building & orchestrating container-based workflows. Is a devcontainer.json file operating like our *.json files in stack/? How does it compare to existing orchestration tools, like kubernetes YAML files? (to me, I see the need to write a devcontainer.json to deploy on Microsoft systems, i.e. VSCode and now codespaces, but imagine that one would accomplish something quite similar on a different provider's platform with some different orchestration metadata, like some k8s yaml.... From your comments I'm encouraged that devcontainer.json might be something both more platform agnostic and solving some additional needs beyond deploy, but I haven't wrapped my head around precisely what that is....)

More immediately, I'd be happy to see the devcontainer R-definition move here if the existing container is to be archived. Thanks for linking that PR too, agree 💯 on that. I like @revodavid's example repo, but it feels like that repo has a size-able code base in it that could be moved upstream (e.g. here / part of the "R definition"), rather than be stuff every single repo needs to duplicate, right?

[eitsupi]

If you want to know how to use devcontainer.json in general, it is also used in the RStudio IDE development.
https://github.com/rstudio/rstudio/tree/main/.devcontainer

pandas has also included devcontainer.json in its repository for several years, making it available for use in development.
https://github.com/pandas-dev/pandas/blob/573e7eaffd801ee5bd1f7685697b51eef5b8ed85/.devcontainer.json
https://pandas.pydata.org/docs/dev/development/contributing_environment.html#creating-an-environment-using-docker

is the devcontainer.json yet consumed by any third-party tooling (i.e. outside of vscode / codespaces?)

The Dev Container CLI is a stand-alone CLI tool.
Of course, this is first party, so I don't think any third party tools have appeared so far.

Is a devcontainer.json file operating like our *.json files in stack/?

This is very similar to compose file, compose.yaml (known as docker-compose.yaml).

How does it compare to existing orchestration tools, like kubernetes YAML files? (to me, I see the need to write a devcontainer.json to deploy on Microsoft systems, i.e. VSCode and now codespaces, but imagine that one would accomplish something quite similar on a different provider's platform with some different orchestration metadata, like some k8s yaml....

I haven't followed much discussion on this either (since I only use docker and docker compose), but if you look at https://github.com/microsoft/vscode-remote-release and https://github.com/devcontainers/spec you will find some discussions.
e.g. microsoft/vscode-remote-release#12, devcontainers/spec#10

At this stage it can only be combined with docker compose.
If you use devcontainer.json with compose.yaml, very few properties can be described in devcontainer.json because of the existence of duplicated functionality.
(Sorry for a Japanese post, but here is my post about VSCode Remote-Containers use with docker compose and rocker/rstudio. https://qiita.com/eitsupi/items/ae0f89266b560b4e7096)

it feels like that repo has a size-able code base in it that could be moved upstream (e.g. here / part of the "R definition"), rather than be stuff every single repo needs to duplicate, right?

Yes, the files in that repository were copied and modified from https://github.com/microsoft/vscode-dev-containers (as is common usage).

More immediately, I'd be happy to see the devcontainer R-definition move here if the existing container is to be archived.

Thanks!
Currently these containers need to be built on local, and takes about 5-10 minutes to build (unlike the pre-built devcontainers distributed by Microsoft).
It would be great to be able to deploy pre-built images in the Rocker project.

[cboettig]

Thanks, this is all helpful.

Right, I've noticed the delay in deploy, and also that each of these repos is building a 50+ line Dockerfile that, among other things, involves running a ~ 450 line debian-common.sh bash script. It seems that it would be desirable to have much of that install / config prebuilt into an upstream rocker container, both to achieve faster deploys and also to avoid duplicating code that could then diverge. Individual repos would I guess still contain a Dockerfile as well as the devcontainer.json file, but it could be one that was considerably more concise then, right?

I guess the next step would be to bring the devcontainer R-definition in over here?

[eitsupi]

I guess the next step would be to bring the devcontainer R-definition in over here?

Yes! But, I believe it is better to create a new repository.
This is because Microsoft's new build system uses the devcontainer CLI to build images. (Previously it was the docker buildx build command)
When using the devcontainer CLI, the common-debian.sh script can be installed as a devcontainer feathure, so we no longer need to make copy of this script when building images.
https://github.com/devcontainers/images/blob/e2558fdcb7554c07b0381db5d66e2806c77395e4/src/debian/.devcontainer.json#L6-L13

The downside of using the devcontainer CLI is that we can't use docker-bake.json files to build images, which makes tag processing very cumbersome......

If we do not use the devcontainer CLI, we may need to copy this script directly from the GitHub repository for synchronization and use it for builds.

In any case, I would be happy to try my own repository on how to do this and bring it to rocker-org after I have successfully done so.

[eitsupi]

Here is a very early prototype.
https://github.com/eitsupi/r-devcontainer-images/blob/7e06974cb7c36f80051490c415c2744d635318a8/src/r-ver/Dockerfile
https://github.com/eitsupi/r-devcontainer-images/blob/7e06974cb7c36f80051490c415c2744d635318a8/src/r-ver/.devcontainer.json

This can be used as a pre-built devcontainer by building and pushing it with the devcontainer cli.

[eitsupi]

@cboettig We can now manage container configuration with json files, as in this repository, and GitHub Actions can build and push.
(Reporting capabilities and automatic updates can be added later if needed, but I believe it is not a priority for now.)
https://github.com/eitsupi/r-devcontainer-images/tree/a2c35937cc063bf107ffb557bcf75e81f15990cc

Future, bspm based containers can be easily added.

I am wondering if this can be brought to rocker-org.
Of course, the devcontainer cli is not yet stable, so this repo and containers should be considered experimental (I can add a badge later)

[cboettig]

Thanks @eitsupi , this is great. All right with you to transfer the repo to rocker-org and see what we can do with it?

Would it be possible to change the configuration such that the RStudio instance opens in the working directory of the cloned repository as an 'open project', thus allowing RStudio's git panel to also accurately reflect the github repo from which the devcontainer was launched? Currently I think that issue is the biggest ergonomic barrier to making the devcontainer practical for RStudio-based training events etc.

[eitsupi]

All right with you to transfer the repo to rocker-org and see what we can do with it?

👍

Would it be possible to change the configuration such that the RStudio instance opens in the working directory of the cloned repository as an 'open project'

It appears that the default working directory can be changed by editing /home/rstudio/.config/rstudio/rstudio-prefs.json.

{
    "initial_working_directory": "/workspaces"
}

If this folder does not exist, it will fall back to /home/rstudio, so there seems to be no harm in this setting even if the path to bind mount is changed.

However, the current rocker/rstudio uses an old configuration file, so /home/rstudio/.config/rstudio/rstudio-prefs.json is automatically generated when RStudio is first started and cannot be edited before RStudio is started.

rocker-versioned2/scripts/default_user.sh

Lines 16 to 20 in 9423d18

	cat <<EOF >"/home/${DEFAULT_USER}/.rstudio/monitored/user-settings/user-settings"
	alwaysSaveHistory="0"
	loadRData="0"
	saveAction="0"
	EOF

Fortunately, RStudio started using the new configuration file in the rocker/rstudio:4.0.0 era (rstudio/rstudio#5069), so we can edit the above line to configure it to use /home/rstudio/.config/rstudio/rstudio-prefs.json instead of /home/rstudio/.rstudio/monitored/user-settings.
I will send a PullRequest later.

[thohan88]

Fantastic to see this conversation getting started. I have been using devcontainers for R-projects during the last year, and they are a gamechanger for onboarding collaborators with less Docker-experience. Just clone the repo using the remote-extension and you are up and running.

Getting the working directory for RStudio to default to workspaces was a bit of a pain for me. I've resorted to putting it into a postCreateCommand as the following:

"postCreateCommand": "echo \"setwd('${containerWorkspaceFolder}')\" >> /usr/local/lib/R/etc/Rprofile.site"

I struggled with permissions for saving files to this folder from RStudio, so I had to add rstudio as owner:

chown -R rstudio ${containerWorkspaceFolder}

This works fine in RStudio but breaks the git-integration in WSL2 (see microsoft/vscode-remote-release#6683 and devcontainers/cli#98), so I had to add:

git config --global --add safe.directory ${containerWorkspaceFolder}"

So it ends up being alot more ugly than it should be

"postCreateCommand": "echo \"setwd('${containerWorkspaceFolder}')\" >> /usr/local/lib/R/etc/Rprofile.site && chown -R rstudio ${containerWorkspaceFolder} && git config --global --add safe.directory ${containerWorkspaceFolder}"

Would love to hear if you have any experience with other approaches, @eitsupi

My current .devcontainer.json-file for a shiny server looks like this:

{
  "name": "name",
  "dockerFile": "Dockerfile",
  "overrideCommand": false,
  "postCreateCommand": "echo \"setwd('${containerWorkspaceFolder}')\" >> /usr/local/lib/R/etc/Rprofile.site && chown -R rstudio ${containerWorkspaceFolder} && git config --global --add safe.directory ${containerWorkspaceFolder}",
  "build": { "args": { "LOCAL_DEVCONTAINER": "true"}},
  "containerEnv": {
	"DISABLE_AUTH": "true",
	"ROOT": "true"
  },
  "extensions": [
    "REditorSupport.r"
   ],
  "appPort": [
      "127.0.0.1:8787:8787",
      "127.0.0.1:3838:3838"
      ]
} 

I prefer to have RStudio available in the devcontainter, but not when running deploying to the cloud. Hence, I add the LOCAL_DEVCONTAINER-argument in the Dockerfile to only install RStudio when running locally:

ARG LOCAL_DEVCONTAINER
RUN if [ -n "$LOCAL_DEVCONTAINER" ] ; then /rocker_scripts/install_rstudio.sh ; fi

Works great with the exception that it does not carry forward my local git credentials due to the issue above.

[eitsupi]

Hi @thohan88, thanks to sharing that!

Would love to hear if you have any experience with other approaches, @eitsupi

I was taking the approach of using docker compose as I wrote in this Japanese article.
https://qiita.com/eitsupi/items/ae0f89266b560b4e7096

The config files look like these:

{
  "name": "rocker_rstudio",
  "dockerComposeFile": [],
  "service": "rstudio",
  "workspaceFolder": "/home/rstudio/works",
  "settings": {
    "r.rterm.linux": "/usr/local/bin/radian",
    "r.bracketedPaste": true,
    "r.plot.useHttpgd": true,
    "[r]": {
      "editor.wordSeparators": "`~!@#%$^&*()-=+[{]}\\|;:'\",<>/?"
    }
  },
  "extensions": ["REditorSupport.r"],
  "shutdownAction": "none",
  "remoteUser": "rstudio"
}

services:
  rstudio:
    build:
      context: ./.devcontainer
    container_name: rocker_rstudio
    ports:
      - "8787:8787"
    volumes:
      - ./.rstudio_config:/home/rstudio/.config
      - ./works:/home/rstudio/works

Instead of changing RStudio's settings, we can set workspaceFolder to match the directory where VSCode opens to RStudio.

The downside of this approach is more configuration files and RStudio is always running and consuming resources...

[eitsupi]

Now experimental docker image ghcr.io/rocker-org/devcontainer/tidyverse:4 is available.
https://github.com/rocker-org/devcontainer-images/pkgs/container/devcontainer%2Ftidyverse

This image has radian, languageserver, and httpgd already installed, which are necessary for using vscode-R, and the common utilities are also installed, so we can use it comfortably without local builds.

{
  "image": "ghcr.io/rocker-org/devcontainer/tidyverse:4",
  "customizations": {
    "vscode": {
      "settings": {
        "r.rterm.linux": "/usr/local/bin/radian",
        "r.bracketedPaste": true,
        "r.plot.useHttpgd": true,
        "[r]": {
          "editor.wordSeparators": "`~!@#%$^&*()-=+[{]}\\|;:'\",<>/?"
        }
      },
      "extensions": ["reditorsupport.r", "rdebugger.r-debugger"]
    }
  },
  "remoteUser": "rstudio"
}

@cboettig Do you think you can close this in favor of https://github.com/rocker-org/devcontainer-images?

[cboettig]

definitely!

closing in favor of further development under https://github.com/rocker-org/devcontainer-images