Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docker-compose support for easier and more portable environment set-up #519

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
4f245d8
Add first revision of the `docker-compose.yaml`
apockill Nov 23, 2024
64e368f
Add first revision of the `Dockerfile`
apockill Nov 23, 2024
0b32337
Add docker instructions
apockill Nov 23, 2024
f7e3c3c
Clean up docs
apockill Nov 23, 2024
80e75ad
Add `tty: true`
apockill Nov 24, 2024
634d15a
Improve handling of poetry extras
apockill Nov 24, 2024
69a3b9d
Improve handling of poetry extras
apockill Nov 24, 2024
a0940e0
Add required ports for visualization scripts
apockill Nov 24, 2024
6631247
Add libsm6
apockill Nov 24, 2024
59c1a78
Remove the need to do `xhost` commands
apockill Nov 25, 2024
26dcaff
Use `poetry run` as the entrypoint
apockill Nov 25, 2024
c6d9ef2
Use consistent strings in docker-compose
apockill Nov 28, 2024
bca43d1
Bump poetry lockfile
apockill Nov 28, 2024
9314f87
Successfully built pyav, and was able to use `cv2.imshow`!
apockill Nov 28, 2024
7514f59
Add `python3-pip` to build
apockill Nov 28, 2024
7d9d05d
Don't isolate python dependencies in container (this plays better wit…
apockill Nov 28, 2024
3a35fd6
Remove spd-say so it doesn't crash the system
apockill Nov 28, 2024
e473a3d
Use a pre-built ffmpeg (this seems to also work with pyav!)
apockill Nov 29, 2024
23aa4c3
Merge in `main`
apockill Nov 29, 2024
e901cc2
Simplify `mv` command to not reference directories too specifically
apockill Nov 29, 2024
3e3eaa6
Fix pyav issues after the `Dataset 2.0` merge
apockill Nov 30, 2024
016d9a0
Add credentials management so `wandb` and `huggingface` work properly
apockill Dec 2, 2024
e4073f7
Merge branch 'main' into feature/docker-compose
apockill Dec 4, 2024
d0729c5
Merge branch 'main' into feature/docker-compose
apockill Dec 10, 2024
3b5748a
Add torch cache to compose file
apockill Dec 12, 2024
23ffaff
Merge remote-tracking branch 'origin/feature/docker-compose' into fea…
apockill Dec 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,15 @@

## Installation

Download our source code:
You can install lerobot either locally, or in a docker container.

Start by downloading our source code:
```bash
git clone https://github.com/huggingface/lerobot.git
cd lerobot
```

### Local Installation
Create a virtual environment with Python 3.10 and activate it, e.g. with [`miniconda`](https://docs.anaconda.com/free/miniconda/index.html):
```bash
conda create -y -n lerobot python=3.10
Expand Down Expand Up @@ -115,6 +118,26 @@ wandb login

(note: you will also need to enable WandB in the configuration. See below.)

### Docker Installation

If using docker, the only dependency you need is [Docker](https://docs.docker.com/get-docker/). For GPU support, you can also install the [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).

Build your environment with:
```
docker compose build
```

If you want to install an environment with `--extras`, you can add a build argument like so:

```
docker compose build --build-arg POETRY_EXTRAS="aloha"
```

Then enter the container with the following, and run lerobot scripts as you would locally:
```
docker compose run --interactive lerobot
```

## Walkthrough

```
Expand Down
26 changes: 26 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: lerobot

services:
lerobot:
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In future PR's we could split up this service to be lerobot-x86 and lerobot-arm to support multiple platforms (like jetson).

For now, I think let's see how people use this, improve the experience, then tackle multi-arch support in the future.

image: lerobot-env
build:
context: .
dockerfile: docker/lerobot-compose/Dockerfile
volumes:
# Necessary for display passthrough
- "/tmp/.X11-unix:/tmp/.X11-unix:rw"
# Mount the current directory so users are not confused about data saving locations
- ".:/lerobot"
environment:
# Necessary for display passthrough
DISPLAY: $DISPLAY
# Give the container as much access as possible to the host machine.
# Folks using this library might be less familiar with how docker isolates devices
# such as USB or networks, so removing that layer is helpful for the purposes of
# development on this library.
#
# For production, remove the following line and add the necessary devices and networks
privileged: true
network_mode: host
cap_add:
- ALL
79 changes: 79 additions & 0 deletions docker/lerobot-compose/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
FROM nvidia/cuda:12.2.2-devel-ubuntu22.04
apockill marked this conversation as resolved.
Show resolved Hide resolved

# Configure image
ARG PYTHON_VERSION=3.10
ARG DEBIAN_FRONTEND=noninteractive
ARG POETRY_EXTRAS=""
ENV PIP_CACHE=/root/.cache/pip

# Install apt dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential cmake \
git git-lfs openssh-client \
nano vim less util-linux tree \
htop atop nvtop \
sed gawk grep curl wget zip unzip \
tcpdump sysstat screen tmux \
libglib2.0-0 libgl1-mesa-glx libegl1-mesa \
speech-dispatcher \
# python*-dev is required for building certain python dependencies
python${PYTHON_VERSION} python${PYTHON_VERSION}-venv python${PYTHON_VERSION}-dev\
&& apt-get clean && rm -rf /var/lib/apt/lists/*

# Install ffmpeg build dependencies. See:
# https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu
# TODO(aliberts): create image to build dependencies from source instead
RUN apt-get update && apt-get install -y --no-install-recommends \
autoconf automake yasm \
libass-dev \
libfreetype6-dev \
libgnutls28-dev \
libunistring-dev \
libmp3lame-dev \
libtool \
libvorbis-dev \
meson \
ninja-build \
pkg-config \
texinfo \
yasm \
zlib1g-dev \
nasm \
libx264-dev \
libx265-dev libnuma-dev \
libvpx-dev \
libfdk-aac-dev \
libopus-dev \
libsvtav1-dev libsvtav1enc-dev libsvtav1dec-dev \
libdav1d-dev

# Install gh cli tool
RUN (type -p wget >/dev/null || (apt update && apt-get install wget -y)) \
&& mkdir -p -m 755 /etc/apt/keyrings \
&& wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
&& chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& apt update \
&& apt install gh -y \
&& apt clean && rm -rf /var/lib/apt/lists/*

# Setup `python`
RUN ln -s /usr/bin/python3 /usr/bin/python

# Install poetry
RUN curl -sSL https://install.python-poetry.org | python -
ENV PATH="/root/.local/bin:$PATH"
RUN echo 'if [ "$HOME" != "/root" ]; then ln -sf /root/.local/bin/poetry $HOME/.local/bin/poetry; fi' >> /root/.bashrc
RUN poetry config virtualenvs.create false
RUN poetry config virtualenvs.in-project true

# Copy in pyproject
WORKDIR /lerobot
COPY pyproject.toml poetry.lock ./
RUN apt-get update && apt-get install -y python3-dev
# Install dependencies, while using docker builkit cache to speed up future builds
RUN --mount=type=cache,target="${PIP_CACHE}" \
poetry install --extras ${POETRY_EXTRAS}

# Set EGL as the rendering backend for MuJoCo
ENV MUJOCO_GL="egl"