-
Notifications
You must be signed in to change notification settings - Fork 799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docker-compose
support for easier and more portable environment set-up
#519
Open
apockill
wants to merge
26
commits into
huggingface:main
Choose a base branch
from
apockill:feature/docker-compose
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 23 commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
4f245d8
Add first revision of the `docker-compose.yaml`
apockill 64e368f
Add first revision of the `Dockerfile`
apockill 0b32337
Add docker instructions
apockill f7e3c3c
Clean up docs
apockill 80e75ad
Add `tty: true`
apockill 634d15a
Improve handling of poetry extras
apockill 69a3b9d
Improve handling of poetry extras
apockill a0940e0
Add required ports for visualization scripts
apockill 6631247
Add libsm6
apockill 59c1a78
Remove the need to do `xhost` commands
apockill 26dcaff
Use `poetry run` as the entrypoint
apockill c6d9ef2
Use consistent strings in docker-compose
apockill bca43d1
Bump poetry lockfile
apockill 9314f87
Successfully built pyav, and was able to use `cv2.imshow`!
apockill 7514f59
Add `python3-pip` to build
apockill 7d9d05d
Don't isolate python dependencies in container (this plays better wit…
apockill 3a35fd6
Remove spd-say so it doesn't crash the system
apockill e473a3d
Use a pre-built ffmpeg (this seems to also work with pyav!)
apockill 23aa4c3
Merge in `main`
apockill e901cc2
Simplify `mv` command to not reference directories too specifically
apockill 3e3eaa6
Fix pyav issues after the `Dataset 2.0` merge
apockill 016d9a0
Add credentials management so `wandb` and `huggingface` work properly
apockill e4073f7
Merge branch 'main' into feature/docker-compose
apockill d0729c5
Merge branch 'main' into feature/docker-compose
apockill 3b5748a
Add torch cache to compose file
apockill 23ffaff
Merge remote-tracking branch 'origin/feature/docker-compose' into fea…
apockill File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -144,3 +144,6 @@ dmypy.json | |
|
||
# Pyre type checker | ||
.pyre/ | ||
|
||
# Volumes | ||
.huggingface_cache/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
name: lerobot | ||
|
||
services: | ||
lerobot: | ||
image: lerobot-env | ||
build: | ||
context: . | ||
dockerfile: docker/lerobot-compose/Dockerfile | ||
tty: true | ||
volumes: | ||
# Necessary for display passthrough | ||
- "/tmp/.X11-unix:/tmp/.X11-unix" | ||
- "~/.Xauthority:/root/.Xauthority" | ||
# Mount the current directory so users are not confused about data saving locations | ||
- ".:/lerobot" | ||
# Pass through USB devices | ||
- "/dev:/dev" | ||
# Mount wandb credentials into the container, in read-only mode | ||
- "~/.netrc:/root/.netrc:ro" | ||
# Mount Hugging Face credentials into the container, in read-only mode. | ||
- ".huggingface_cache/:/root/.cache/huggingface/" | ||
environment: | ||
# Necessary for display passthrough | ||
DISPLAY: $DISPLAY | ||
XAUTHORITY: $XAUTHORITY | ||
# Give the container as much access as possible to the host machine. | ||
# Folks using this library might be less familiar with how docker isolates devices | ||
# such as USB or networks, so removing that layer is helpful for the purposes of | ||
# development on this library. | ||
# | ||
# For production, remove the following line and add the necessary devices and networks | ||
privileged: true | ||
network_mode: host | ||
cap_add: | ||
- ALL |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
FROM nvidia/cuda:12.2.2-devel-ubuntu22.04 | ||
apockill marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# Configure image | ||
ARG PYTHON_VERSION=3.10 | ||
ARG DEBIAN_FRONTEND=noninteractive | ||
ENV PIP_CACHE=/root/.cache/pip | ||
|
||
# Install apt dependencies | ||
RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
# Basic build and dev tools | ||
build-essential cmake \ | ||
git git-lfs openssh-client \ | ||
nano vim less util-linux tree \ | ||
htop atop nvtop \ | ||
sed gawk grep curl wget zip unzip \ | ||
tcpdump sysstat screen tmux \ | ||
libglib2.0-0 libgl1 libglx-mesa0 \ | ||
# python*-dev is required for building certain python dependencies | ||
python${PYTHON_VERSION} python${PYTHON_VERSION}-venv python${PYTHON_VERSION}-dev \ | ||
# Required for cv2.imshow(...) and other highgui operations | ||
libsm6 \ | ||
# Required for building PyAv \ | ||
pkg-config \ | ||
python3-pip \ | ||
libavcodec-dev \ | ||
libavdevice-dev \ | ||
libavfilter-dev \ | ||
libavformat-dev \ | ||
libavutil-dev \ | ||
libswscale-dev \ | ||
libswresample-dev \ | ||
# Symbolic link to python3 \ | ||
python-is-python3 \ | ||
&& apt-get clean && rm -rf /var/lib/apt/lists/* | ||
|
||
# Install pre-built ffmpeg binaries with support for libsvtav1, which is the lerobot default | ||
WORKDIR /ffmpeg | ||
# To update, select a build from https://github.com/BtbN/FFmpeg-Builds/releases | ||
ARG FFMPEG_BUILD=autobuild-2024-10-31-12-59/ffmpeg-N-117676-g87068b9600-linux64-gpl.tar.xz | ||
RUN wget https://github.com/BtbN/FFmpeg-Builds/releases/download/${FFMPEG_BUILD} && \ | ||
tar -xvf ffmpeg*.tar.xz && \ | ||
mv ffmpeg*/bin/* /usr/local/bin/ && \ | ||
# Verify this ffmpeg build succeeded and has support for libsvtav1, the default \ | ||
# lerobot video decoder \ | ||
ffmpeg -encoders | grep libsvtav1 && \ | ||
# Clean up \ | ||
rm -rf /ffmpeg | ||
|
||
# Install poetry | ||
RUN curl -sSL https://install.python-poetry.org | python - | ||
ENV PATH="/root/.local/bin:$PATH" | ||
RUN echo 'if [ "$HOME" != "/root" ]; then ln -sf /root/.local/bin/poetry $HOME/.local/bin/poetry; fi' >> /root/.bashrc | ||
RUN poetry config virtualenvs.create false | ||
RUN poetry config virtualenvs.in-project true | ||
|
||
# Install python dependencies, including `POETRY_EXTRAS` if specified as a --build-arg | ||
WORKDIR /lerobot | ||
COPY pyproject.toml poetry.lock ./ | ||
ARG POETRY_EXTRAS="none" | ||
RUN --mount=type=cache,target="${PIP_CACHE}" \ | ||
poetry install --extras ${POETRY_EXTRAS} | ||
ENV PYTHONPATH="/lerobot:${PYTHONPATH}" | ||
|
||
# Forcefully reinstall pyav using `--no-binary`, so it has to build itself using the | ||
# systems ffmpeg. This resolves a lockup bug that can happen when trying to open windows | ||
# with OpenCV, with `av` is imported and targets a different version of ffmpeg. | ||
# Tracking issue: https://github.com/opencv/opencv/issues/21952 | ||
RUN pip3 install --no-binary pyav --force-reinstall pyav==12.3.0 | ||
|
||
# Set the entrypoint such that users enter the container with the poetry environment | ||
ENTRYPOINT ["poetry", "run"] | ||
CMD ["bash"] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In future PR's we could split up this service to be
lerobot-x86
andlerobot-arm
to support multiple platforms (like jetson).For now, I think let's see how people use this, improve the experience, then tackle multi-arch support in the future.