Add devcontainer (#142)

This adds a devcontainer configuration to our repository, and documents
it as the recommended development workflow.

This should clean up quite a few platform and environment setup problems
for development work.

Batch jobs and evaluation should still be done directly in Linux
environments, which are still supported.

This also switches our pre-commit hook to use `dprint` instead of
`taplo`, because it's easier to install dprint on linux-aarch64 (and
having pre-commit hooks depend on executables only inside the Pixi
environment doesn't work very well).

.devcontainer/Dockerfile

@@ -0,0 +1,9 @@
+FROM mcr.microsoft.com/devcontainers/base:bookworm
+ARG PIXI_VERSION=v0.38.0
+
+# derived from https://pixi.sh/latest/ide_integration/devcontainer/
+RUN curl -L -o /usr/local/bin/pixi -fsSL --compressed "https://github.com/prefix-dev/pixi/releases/download/${PIXI_VERSION}/pixi-$(uname -m)-unknown-linux-musl" \
+    && chmod +x /usr/local/bin/pixi \
+    && pixi info
+
+RUN echo 'eval "$(pixi completion -s bash)"' >> /home/vscode/.bashrc

.devcontainer/devcontainer.json

@@ -0,0 +1,35 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/debian
+{
+  "name": "POPROX Recommender",
+  "build": {
+    "dockerfile": "Dockerfile"
+  },
+  "features": {
+    "ghcr.io/devcontainers-extra/features/pre-commit:2": {}
+  },
+  // Use 'forwardPorts' to make a list of ports inside the container available locally.
+  "forwardPorts": [
+    3000
+  ],
+  // Set up mounts for locally-installed software
+  "mounts": [
+    "source=${localWorkspaceFolderBasename}-pixi,target=${containerWorkspaceFolder}/.pixi,type=volume",
+    "source=${localWorkspaceFolderBasename}-node_modules,target=${containerWorkspaceFolder}/node_modules,type=volume"
+  ],
+  "postCreateCommand": "./.devcontainer/setup.sh",
+  // VS Code settings
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "EditorConfig.EditorConfig",
+        "ms-python.python",
+        "charliermarsh.ruff",
+        "redhat.vscode-yaml",
+        "tamasfe.even-better-toml",
+        "ms-toolsai.jupyter",
+        "ms-azuretools.vscode-docker"
+      ]
+    }
+  }
+}

.devcontainer/setup.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+set -xeo pipefail
+
+# set up environment dirs
+sudo chown vscode:vscode .pixi node_modules
+
+# get pre-commit wired up and ready
+pre-commit install
+pre-commit install-hooks
+
+# install the development environment
+pixi install -e dev
+pixi run -e dev install-serverless

.dprint.json

@@ -0,0 +1,17 @@
+{
+  "json": {
+  },
+  "toml": {
+  },
+  "yaml": {
+  },
+  "excludes": [
+    "**/*-lock.json",
+    "**/cloudformation.yml"
+  ],
+  "plugins": [
+    "https://plugins.dprint.dev/json-0.19.4.wasm",
+    "https://plugins.dprint.dev/toml-0.6.3.wasm",
+    "https://plugins.dprint.dev/g-plane/pretty_yaml-v0.5.0.wasm"
+  ]
+}

.gitignore

@@ -86,6 +86,7 @@ celerybeat.pid
 
 # Environments
 .env
+.envrc
 .venv
 env/
 venv/

.pre-commit-config.yaml

@@ -5,7 +5,7 @@ repos:
       - id: check-added-large-files
         exclude: conda-lock.*\.yml|pixi\.lock
       - id: check-json
-        exclude: outputs/.*\.json
+        exclude: (outputs/.*|\.devcontainer/.*)\.json
       - id: check-merge-conflict
       - id: check-shebang-scripts-are-executable
       - id: check-yaml
@@ -18,16 +18,10 @@ repos:
     rev: v8.21.1
     hooks:
       - id: gitleaks
-  - repo: https://github.com/google/yamlfmt
-    rev: v0.13.0
-    hooks:
-      - id: yamlfmt
-        name: format yaml files
-        exclude: (cloudformation|conda-lock.*)\.yml
-  - repo: https://github.com/ComPWA/taplo-pre-commit
-    rev: v0.9.3
-    hooks:
-      - id: taplo-format
+  # - repo: https://github.com/ComPWA/taplo-pre-commit
+  #   rev: v0.9.3
+  #   hooks:
+  #     - id: taplo-format
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
     rev: v0.7.1
@@ -43,3 +37,17 @@ repos:
     rev: v3.10.0-1
     hooks:
       - id: shfmt
+        name: format shell scripts
+  - repo: local
+    hooks:
+      - id: dprint
+        name: format source files
+        entry: dprint fmt
+        language: node
+        types_or:
+          - yaml
+          - toml
+          - json
+        exclude: ^models/
+        additional_dependencies:
+          - dprint

.vscode/settings.json

@@ -2,6 +2,20 @@
   "[python]": {
     "editor.defaultFormatter": "charliermarsh.ruff",
     "editor.formatOnSave": true,
-    "editor.codeActionsOnSave": {"source.organizeImports": "explicit"}
+    "editor.codeActionsOnSave": {
+      "source.organizeImports": "explicit"
+    }
+  },
+  "[json]": {
+    "editor.formatOnSave": true
+  },
+  "[jsonc]": {
+    "editor.formatOnSave": true
+  },
+  "[toml]": {
+    "editor.formatOnSave": true
+  },
+  "[yaml]": {
+    "editor.formatOnSave": true
   }
 }

README.md

@@ -11,53 +11,86 @@ trained models.
 
 ## Installation for Development
 
-Software environments for this repository are managed with [pixi][], and model
-and data files are managed using [dvc][]. The `pixi.lock` file provides a locked
-dependency set for reproducibly running the recommender code with all
-dependencies, on Linux, macOS, and Windows (including with CUDA on Linux).
+This repository includes a devcontainer configuration that we recommend using
+for development and testing of the recommender code.  It is not a good solution
+for running evaluations (see below for non-container setup), but is the easiest
+and most reliable way to set up your development environment across platforms.
 
-See the Pixi [install instructions][pixi] for how to install Pixi in general. On
-macOS, you can also use Homebrew (`brew install pixi`), and on Windows you can
-use WinGet (`winget install prefix-dev.pixi`).
+To use the devcontainer, you need:
 
-> [!NOTE]
->
-> If you are trying to work on poprox-recommender with WSL on Windows, you need
-> to follow the Linux install instructions, and also add the following to the
-> Pixi configuration file (`~/.pixi/config.toml`):
->
-> ```toml
-> detached-environments = true
-> ```
+- VS Code (other editors supporting DevContainer may also work, but this is the
+  best-supported and best-tested).
+- Docker (probably also works with Podman or other container CLIs, but we test
+  with Docker).
+
+With those installed, open the repository in VS Code, and it should prompt you
+to re-open in the dev container; if it does not, open the command palette
+(<kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd>) and choose “Dev Containers:
+Rebuild and Reopen in Container”.
+
+### Installing Docker
+
+On Linux, install the [Docker Engine][engine], and add your user to the `docker`
+group so you can create containers without root.
+
+On Windows, install [Docker Desktop][dd], [Rancher Desktop][rancher], or
+similar.
+
+On MacOS, you can install Docker or Rancher Desktop linked above, or you can use
+[Colima][], which we recommend for simplicity and licensing clarity.  To install
+and use Colima:
+
+```console
+$ brew install colima docker
+$ colima start -m 4
+```
+
+It should also be possible to directly use Lima, but we have not tested or
+documented support for that.
+
+[engine]: https://docs.docker.com/engine/install/
+[dd]: https://www.docker.com/products/docker-desktop/
+[rancher]: https://rancherdesktop.io/
+[Colima]: https://github.com/abiosoft/colima
+
+## Working with the Software
+
+We manage software environments for this repository with [pixi][], and model and
+data files with [dvc][]. The `pixi.lock` file provides a locked dependency set
+for reproducibly running the recommender code with all dependencies on Linux and
+macOS (we use the devcontainer for development support on Windows).
 
 [pixi]: https://pixi.sh
 [dvc]: https://dvc.org
 
-Once Pixi is installed, to install the dependencies needed for development work:
+The devcontainer automatically installs the development Pixi environment; if you
+want to manually install it, you can run:
 
 ```console
 pixi install -e dev
 ```
 
-Once you have installed the dependencies, there are 3 easy ways to run code in the environment:
+VS Code will also usually activate this environment by default when opening a
+terminal; you can also directly run code in in the Pixi environment with any of
+the following methods:
 
 1.  Run a defined task, like `test`, with `pixi run`:
 
     ```console
-    pixi run -e dev test
+    $ pixi run -e dev test
     ```
 
 2.  Run individual commands with `pixi run`, e.g.:
 
     ```console
-    pixi run -e dev pytest tests
+    $ pixi run -e dev pytest tests
     ```
 
 3.  Run a Pixi shell, which activates the environment and adds the appropriate
     Python to your `PATH`:
 
     ```console
-    pixi shell -e dev
+    $ pixi shell -e dev
     ```
 
 > [!NOTE]
@@ -71,49 +104,13 @@ Once you have installed the dependencies, there are 3 easy ways to run code in t
 > you type `exit` in this shell, it will exit the nested shell and return you to
 > you original shell session without the environment active.
 
-Finally, set up `pre-commit` to make sure that code formatting rules are applied
-as you make changes:
-
-```console
-pre-commit install
-```
-
-> [!NOTE]
->
-> If you will be working with `git` outside of the `pixi` shell, you may want to
-> install `pre-commit` separately.  You can do this with Brew or your preferred
-> system or user package manager, or with `pixi global install pre-commit`.
+## Data and Model Access
 
 To get the data and models, there are two steps:
 
 1.  Obtain the credentials for the S3 bucket and put them in `.env` (the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
 2.  `dvc pull`
 
-### Dependency Updates
-
-If you update the dependencies in poprox-recommender, or add code that requires
-a newer version of `poprox-concepts`, you need to regenerate the lock file with
-`pixi update`.  To update just `poprox-concepts`, run:
-
-```console
-pixi update poprox_concepts
-```
-
-To update all dependencies, run:
-
-```console
-pixi update
-```
-
-> [!NOTE]
-> We use [Pixi][] for all dependency management.  If you need to add a new dependency
-> for this code, add it to the appropriate feature(s) in `pixi.toml`.  If it is a
-> dependency of the recommendation components themselves, add it both to the
-> top-level `dependencies` table in `pixi.toml` *and* in `pyproject.toml`.
-
-> [!NOTE]
-> Currently, dependencies can only be updated on Linux.
-
 ## Local Endpoint Development
 
 Local endpoint development also requires some Node-based tools in addition to the tools above — this install is
@@ -238,6 +235,7 @@ You can test this by sending a request with curl:
 $ curl -X POST -H "Content-Type: application/json" -d @tests/request_data/basic-request.json localhost:3000
 ```
 
+
 ## Running the Evaluation
 
 The default setup for this package is CPU-only, which works for basic testing
@@ -265,12 +263,36 @@ Footnotes:
 2. Using 8 worker processes
 3. Estimated based on early progress, not run to completion.
 
-## Editor Setup
 
-If you are using VSCode, you should install the following plugins for best success with this repository:
+## Additional Software Environment Notes
+
+### Non-Container Development Notes
+
+If you are not using the devcontainer, set up `pre-commit` to make sure that
+code formatting rules are applied as you make changes:
+
+```console
+pre-commit install
+```
+
+### Dependency Updates
+
+If you update the dependencies in poprox-recommender, or add code that requires
+a newer version of `poprox-concepts`, you need to regenerate the lock file with
+`pixi update`.  To update just `poprox-concepts`, run:
+
+```console
+pixi update poprox_concepts
+```
+
+To update all dependencies, run:
+
+```console
+pixi update
+```
 
-- [EditorConfig](https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig)
-- [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
-- [Ruff](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff)
+### Editor Setup
 
-When you open the repository, they should be automatically recommended.
+The devcontainer automatically configures several VS Code extensions and
+settings; we also provide an `extensions.json` listing recommended extensions
+for this repository.