Support for the Docker Python SDK #2158

apierleoni · 2017-06-15T15:48:43Z

Description

Hi there, we did some work to run a data processing pipeline using Docker containers with Luigi using the Docker Python SDK and we would like to contribute it back.

Motivation and Context

These PR allows to use the Docker Python SDK to run Docker containers instead of launching them from a subprocess.
The main advantage over calling an external process is that all the communications done with the Docker daemon are through the API and not a subprocess, being able to handle exceptions properly and potentially connect to remote APIs (we did not test this scenario)
It also solve non trivial problems with the docker SDK like mounting a single file, and makes sure that there is enough space in the container to run the job by mounting a temporary directory from the host , avoiding the container size limit of 10Gb.

Have you tested this? If so, how?

Unit tests are included. But they require a Docker daemon to run. Not sure it is available in Travis.
We are currently running this in our staging environment and we are ready to run production pipelines with this contrib.

… low-level API

…ed amongst all tasks. also some linting

dlstadther · 2017-09-11T11:04:31Z

@apierleoni It would be great if you could separate "companies who use luigi" into another PR - that one can be merged quickly.

Also, looks like you've had some Travis errors; could you take a look and resolve? Thanks!

elipapa · 2017-09-11T11:43:23Z

@dlstadther the travis errors are not ours and apply to every PR. @Tarrasch has already acknowledged that they have yet to be fixed (see comment above: june 23)

on separating the PR: we have no problem waiting for this one to be merged for our "companies with luigi" link to show up. We'd rather keep them in the same PR for simplicity and just wait for the review for it to be merged.

dlstadther · 2017-09-11T11:49:50Z

@elipapa Those errors were resolved elsewhere. Your current errors are flake8.

apierleoni · 2017-09-11T12:58:24Z

flake8 errors fixed now, all tests passed on travis

dlstadther · 2017-09-11T17:55:08Z

tox.ini

@@ -10,6 +10,7 @@ deps=
  moto<1.0
  HTTPretty==0.8.10
  nose<2.0
+  docker


Do you want to specify a specific version?

yes thanks, well spotted!

dlstadther

If there are no objections, I'm ok to merge this.

drobakowski

Hi @apierleoni thanks a lot for all this work! I'm using this DockerTask for quite some time and it is working fine overall so far. Even though I had to adapt some parts to make it work properly for our use case. I will start with this first small review because I think the test are not working correctly when they are fixed. See my comments (volumes to binds) in the tests. After that I will add some additional comments and suggestions, for some later improvements or extensions.

drobakowski · 2017-09-11T18:40:40Z

test/contrib/docker_runner_test.py

+    image = "busybox"
+    name = "MountLocalFileAsVolume"
+    # volumes= {'/tmp/local_file_test': {'bind': local_file.name, 'mode': 'rw'}}
+    volumes = [local_file.name + ':/tmp/local_file_test']


Shouldn't this be renamed to binds?

drobakowski · 2017-09-11T18:40:56Z

test/contrib/docker_runner_test.py

+    dummyopt = luigi.Parameter()
+    image = "busybox"
+    name = "MountLocalFileAsVolumeWithParam"
+    volumes = [local_file.name + ':/tmp/local_file_test']


Shouldn't this be renamed to binds?

drobakowski · 2017-09-11T18:41:41Z

test/contrib/docker_runner_test.py

+    name = "MountLocalFileAsVolumeWithParamRedef"
+
+    @property
+    def volumes(self):


Shouldn't this be renamed to binds?

drobakowski · 2017-09-11T19:18:28Z

luigi/contrib/docker_runner.py

+        # derive volumes (ie. list of container destination paths) from
+        # specified binds
+        self._volumes = [b.split(':')[1] for b in self._binds]
+


One addition I had to make to get this properly working in our scenario was to include a custom instance attribute self._complete = False and set this to True when the container exited.

drobakowski · 2017-09-11T19:54:25Z

luigi/contrib/docker_runner.py

+                                                      **self.container_options)
+            self._client.start(container['Id'])
+
+            exit_status = self._client.wait(container['Id'])


+self._complete = True

I'm not sure anymore if I included this simply to True on purpose or if this should be set in subject to the exit_status.

In our use case we're chaining multiple DockerTask consecutive so that each task depend on the previous one and its output. As I'm quite new to working with luigi I don't have that much deep knowledge and experience so far, but I think I had some problems with the luigi scheduler while the scheduler was trying to resolve and start the DockerTask process chain properly. After including the previous changes and overwriting the complete methods, the chain worked fine for me. As far as I remember correctly, I've even got the tests to run fine after renaming volumes to binds. Perhaps this is use case specific, but maybe someone with better knowledge of luigi can overlook this better.

class ImportProcess(DockerTask): .... def complete(self): return self.output().exists() or self._complete ...

Of course this could also be implemented in the DockerTask class directly with maybe some modification like this:

class DockerTask(luigi.Task): .... def complete(self): return super(DockerTask, self).complete() or self._complete

drobakowski · 2017-09-11T20:12:52Z

I do have two more smaller additions, but in order to keep this PR simple, I will create a PR after this one will be merged. Thanks again for DockerTask, this helps us a lot! 👍

dlstadther · 2017-09-19T12:39:27Z

@apierleoni Where does this PR stand given @drobakowski 's review and comments?

apierleoni · 2017-09-25T14:28:39Z

@drobakowski
I am glad DockerTask was useful for you, and than you for the review.
Yes I have renamed volume to binds, and changed default behaviour for tmp directory mounting. Tests are now passing in my system.
I have added docker to the services in travis, so now the test should run in CI

Regarding the second issue, I am not 100% sure I understood what you want to do with the self._complete attribute. The way we use DockerTask is to make sure that the Target is labeled as completed, so that Luigi will know if he needs to rerun the task or not.
Would you prefer to trigger an error in case the container quit status 0? how do you know it is not completed at that stage?

@dlstadther
tests should be improved now, maybe you can help here on what is the preferred way to store a task to be completed.

apierleoni · 2017-09-25T14:51:55Z

some tests are failing for an error that looks related to boto non importing the google cloud platform module. this might not be limited to this PR
https://travis-ci.org/spotify/luigi/jobs/279538792#L565

apierleoni · 2017-09-26T11:21:56Z

merged with current master, still failing for boto

apierleoni · 2017-10-14T23:46:33Z

@dlstadther
I have fixed the errors with boto in the tests and resolved conflicts with master.
Should be clear to merge if you want.

apierleoni · 2017-10-14T23:51:38Z

I guess the codecov issue is related with this https://github.com/codecov/support/issues/180

dlstadther

LGTM! I'm not concerned about that codecov thing. The tests pass, so let's merge and allow others the goodness of Luigi v1 DockerTask support!

dlstadther · 2018-02-16T19:52:18Z

Hey @apierleoni , do you have any insight into why two of the Docker tests started failing recently?

It'd be much appreciated if you could check it out! Thanks

apierleoni · 2018-04-11T00:02:06Z

@dlstadther sorry for the late reply, is it still an issue? I cannot find the failing tests.

dlstadther · 2018-04-11T10:59:35Z

@apierleoni They've since been fixed!

Andrea Pierleoni and others added 30 commits February 26, 2017 00:30

added stub for minimal docker wrapper

8a4d331

don't set a name by default

8481b8a

handle None name

83e6b48

changed to use the docker low level api

3b67754

removed debugging print

3cd4d11

refractored names

849e59c

added a test for single file mount

55f0528

merged with tempdir support and image download

bb8122c

directory and file mounting working

f1fb534

fixed tests and using busybox when needed

ac55d9e

always use busybox

9321e4a

fallback if exception has no message

07fcb84

lowered pull image to debug level

09f254e

do not declare volumes unless they are passed as extra params

5258346

remove name requirement from tmpdir

0f1547c

linting

a4643a1

fix tests for tox

65b1a46

tag must be specified to get the right number of images back from the…

e0a4f28

… low-level API

test to trigger duplicate mount pt issue

a7da033

fix pid lock in tests

7273d3d

use a wrapper task in test

5750452

another test to see if the issue is in properties

8c8580f

created an _init_ method to avoid the mutable volumes list to be shar…

6ce5586

…ed amongst all tasks. also some linting

need args on the init to allow parameters

8452bf4

solves pid lock

89ce5c0

linting

a625c32

remove bad logging lines

caedbcd

fix handling of multiple volumes

fa3127c

must be passing kwargs when using super!

f7c4197

old gitignore

caf0070

flake8 linting

e997426

dlstadther reviewed Sep 11, 2017

View reviewed changes

dlstadther approved these changes Sep 11, 2017

View reviewed changes

added version to docker dependency

c8d007f

drobakowski suggested changes Sep 11, 2017

View reviewed changes

drobakowski reviewed Sep 11, 2017

View reviewed changes

Andrea Pierleoni added 2 commits September 25, 2017 15:19

mount tmp dir by default, and rename volumes to binds

76757f4

enable docker service in travis ci to allow DockerTask tests to run

6ad3094

Merge remote-tracking branch 'remotes/upstream/master'

f877ca3

apierleoni added 2 commits October 15, 2017 00:29

Merge branch 'master' into master

fe5636d

fix boto problem in tests

4e1fb1f

dlstadther approved these changes Oct 15, 2017

View reviewed changes

dlstadther merged commit 4801fec into spotify:master Oct 15, 2017

This was referenced Jun 29, 2022

no mo enum 34 #3180

Closed

enum34 be gone #3181

Closed

mdragilev mentioned this pull request Jun 28, 2024

for S3 contrib package move to boto3 Affirm/luigi#26

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for the Docker Python SDK #2158

Support for the Docker Python SDK #2158

apierleoni commented Jun 15, 2017 •

edited

Loading

dlstadther commented Sep 11, 2017

elipapa commented Sep 11, 2017

dlstadther commented Sep 11, 2017

apierleoni commented Sep 11, 2017

dlstadther Sep 11, 2017

apierleoni Sep 11, 2017

dlstadther left a comment

drobakowski left a comment

drobakowski Sep 11, 2017

drobakowski Sep 11, 2017

drobakowski Sep 11, 2017

drobakowski Sep 11, 2017

drobakowski Sep 11, 2017

drobakowski commented Sep 11, 2017

dlstadther commented Sep 19, 2017

apierleoni commented Sep 25, 2017

apierleoni commented Sep 25, 2017

apierleoni commented Sep 26, 2017

apierleoni commented Oct 14, 2017

apierleoni commented Oct 14, 2017

dlstadther left a comment

dlstadther commented Feb 16, 2018

apierleoni commented Apr 11, 2018

dlstadther commented Apr 11, 2018

Support for the Docker Python SDK #2158

Support for the Docker Python SDK #2158

Conversation

apierleoni commented Jun 15, 2017 • edited Loading

Description

Motivation and Context

Have you tested this? If so, how?

dlstadther commented Sep 11, 2017

elipapa commented Sep 11, 2017

dlstadther commented Sep 11, 2017

apierleoni commented Sep 11, 2017

dlstadther Sep 11, 2017

Choose a reason for hiding this comment

apierleoni Sep 11, 2017

Choose a reason for hiding this comment

dlstadther left a comment

Choose a reason for hiding this comment

drobakowski left a comment

Choose a reason for hiding this comment

drobakowski Sep 11, 2017

Choose a reason for hiding this comment

drobakowski Sep 11, 2017

Choose a reason for hiding this comment

drobakowski Sep 11, 2017

Choose a reason for hiding this comment

drobakowski Sep 11, 2017

Choose a reason for hiding this comment

drobakowski Sep 11, 2017

Choose a reason for hiding this comment

drobakowski commented Sep 11, 2017

dlstadther commented Sep 19, 2017

apierleoni commented Sep 25, 2017

apierleoni commented Sep 25, 2017

apierleoni commented Sep 26, 2017

apierleoni commented Oct 14, 2017

apierleoni commented Oct 14, 2017

dlstadther left a comment

Choose a reason for hiding this comment

dlstadther commented Feb 16, 2018

apierleoni commented Apr 11, 2018

dlstadther commented Apr 11, 2018

apierleoni commented Jun 15, 2017 •

edited

Loading