Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial docker setup in core ODC repo #382

Merged
merged 16 commits into from
Mar 14, 2018
Merged

Initial docker setup in core ODC repo #382

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
827202c
change psychpg2 for psycopg2-binary
Mar 7, 2018
96cd444
add dockerignore
Mar 7, 2018
67da346
Add Dockerfile
Mar 7, 2018
f8d5bec
add docker-compose for dev
Mar 7, 2018
8c19f0b
Merge branch 'develop' into docker-setup
alexgleith Mar 13, 2018
b73e72d
Remove docker-compose file
alexgleith Mar 13, 2018
0907453
Change the Psycopg2 install to a manual special case
alexgleith Mar 13, 2018
82d657e
Comment the Dockerfile
alexgleith Mar 13, 2018
91b9fa6
Add note to the whats new file in docs
alexgleith Mar 13, 2018
48c0782
Add an entrypoint to set up the database credentials
alexgleith Mar 14, 2018
7729996
update readme with docker info
alexgleith Mar 14, 2018
903386e
fix readme formatting
alexgleith Mar 14, 2018
8209a46
fix readme formatting again
alexgleith Mar 14, 2018
1a82936
further readme formatting
alexgleith Mar 14, 2018
01a9a16
yet more formatting
alexgleith Mar 14, 2018
81feb1a
Merge branch 'develop' into docker-setup
alexgleith Mar 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
.appveyor
.circleci
.github
.travis
contrib
docs
examples
integration_tests
utils
33 changes: 33 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
FROM ubuntu:16.04
# This Dockerfile should follow the Travis configuration process
# available here: https://github.com/opendatacube/datacube-core/blob/develop/.travis.yml

# First add the NextGIS repo
RUN apt-get update && apt-get install -y \
python-software-properties \
software-properties-common

RUN add-apt-repository ppa:nextgis/ppa

# And now install apt dependencies, including a few of the heavy Python projects
RUN apt-get update && apt-get install -y \
gdal-bin libgdal-dev libgdal20 libudunits2-0 \
python3 python3-gdal python3-setuptools python3-dev python3-numpy python3-netcdf4 \
python3-pip

# Get the code, and put it in /opt
ENV APPDIR=/code
RUN mkdir -p $APPDIR
COPY . $APPDIR
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not too sure about "getting code from working copy strategy".
I'd rather fetch directly from github with a hash/tag/branch supplied as build argument

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concept here is that the Dockefile is in with the code, so that it can be automatically built using Docker Cloud. We can configure tagged builds that are in sync with this repo automatically that way.

At the moment, I'm thinking we have a latest tag for the docker image coming out of the develop branch (that should always build and work, right?) and then tagged builds from releases.

It was suggested that we do the install from PIP, since that's the 'point of truth'. But if we have the code local to the build, it's not necessary to go to GitHub and fetch it again.


# Install the dependencies
WORKDIR $APPDIR

# Set the locale, this is required for some of the Python packages
ENV LC_ALL C.UTF-8
RUN pip3 install '.[test,analytics,celery,s3]' --upgrade
RUN pip3 install ./tests/drivers/fail_drivers --no-deps --upgrade

RUN python3 setup.py develop

CMD datacube --help
9 changes: 9 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
version: '3'
services:
datacube:
build: .
volumes:
- ./:/code
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So are we using a volume for code or are we copying code during build?
Seem to be doing both.

Copy link
Contributor Author

@alexgleith alexgleith Mar 8, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Kirill, the docker-compose file uses the Dockerfile as a base, so that dependencies are already installed, and then mounts in the volume so that it has your local changes. This means you can change your code locally, and run it in Docker with the changes without rebuilding the image and creating a new container.

Note that it's using the path to the cli_app.py so that it is not using the installed version of the Open Data Cube, it's using the one from your local directory.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, but won't it trigger rebuild anyway, since you changed folder content and that get's copied into the docker, so from COPY down things will need to be redone anyway?

Also we use python "entry points" (https://amir.rachum.com/blog/2017/07/28/python-entry-points/), how this will interact with the proposed setup?

Copy link
Contributor Author

@alexgleith alexgleith Mar 8, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Docker Compose, the first build is mandatory, but later builds are manually triggered (with docker-compose build). So no, it won't trigger a new build.

I'm aware of the entry points in the setup.py file, and that's where I copied the main one from. I think a Docker Compose file can only have one in it, so unless we set up a whole new way of doing it, the other commands will need to be referenced by their full path. That'll probably need to be documented.

Just FYI, if you use this docker-compose file, the entrypoint commands (like datacube and pixeldrill) will still work, but if you use them, they will point to the installed version, and not the version mounted by the volume. That's a bit of a gotcha. To run code that you have changed, you need to use the path, like /code/datacube/scripts/cli_app.py.

entrypoint:
python3 /code/datacube/scripts/cli_app.py
command: --help
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
'jsonschema',
'netcdf4',
'numpy',
'psycopg2',
'psycopg2-binary',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should still depend on psycopg2. The binary package is available as an installation option, but, not a package that should be depended on.

By recent discussions at psycopg2 issue 674, it also doesn't sound to be completely resolved.

It would be totally fine to pip install psycopg2-binary in the Dockerfile, but leave the dependency alone.

'pypeg2',
'python-dateutil',
'pyyaml',
Expand Down