Look at migrating docker plugin to use Docker SDK for Python

[cjh1]

Move away from popen to https://github.com/docker/docker-py

[zachmullen]

👍

Looks like this communicates directly with the service over HTTP rather than using a subprocess, which is cool. We'd have to investigate how well its API supports streaming IO.

[cjh1]

Yes, talking directly to the service seems like a nice step up from using subprocess. When you say "streaming IO" are you taking about the support we currently have for pipes?

[zachmullen]

Yeah, the ability to do non-blocking reads from pipes (a la select).

[kotfic]

was just poking around a little dockerpty may be relevant.

[cjh1]

Recording quote from @zachmullen "we should see about using more modern tooling for docker garbage collection in lieu of the current docker-gc script, which is a third party tool that was the industry standard for a while." Was thinking the same ...

[cjh1]

Some possible alternative to docker-gc script:

https://github.com/Yelp/docker-custodian (uses docker-py )

[manthey]

Docker-py has just recently release a version 2.0. Prior to version 2.0, it is docker-py on pip, now it is is just docker. There are substantial breaking changes between the two. We use version 1.9 in the slicer_cli_web and HistomicsTK projects, so if we use version 2, we may want to update those as well.

It almost seems like less work to use the command line interface than docker-py, especially for volumes. Version 2 is a little cleaner than version 1.9. The transition wouldn't be hard.

And, our GC task should run out-of-band; it seems wasteful to delay sending job results until the GC has run, especially when that GC can take a long time.

[cjh1]

Docker-py has just recently release a version 2.0. Prior to version 2.0, it is docker-py on pip, now it is is just docker. There are substantial breaking changes between the two. We use version 1.9 in the slicer_cli_web and HistomicsTK projects, so if we use version 2, we may want to update those as well.

It almost seems like less work to use the command line interface than docker-py, especially for volumes. Version 2 is a little cleaner than version 1.9. The transition wouldn't be hard.

@manthey Are you saying that the move docker-py isn't worth it despite being able to talk directly to the docker daemon rather than through a sub process?

And, our GC task should run out-of-band; it seems wasteful to delay sending job results until the GC has run, especially when that GC can take a long time.

I agree a celery periodic task could be used for this

[manthey]

@cjh1 I don't think switching to docker-py will reduce code complexity. Version 2 won't increase it much, but version 1.x will reduce readability. If it gains us robustness or maintainability, then switch can still be worth while.

[cjh1]

It would reduce the number of processes used to run a docker task by one, which might help with scalability.

[zachmullen]

As part of this, we should try to mitigate the busy loop behavior while we wait on the docker container to finish.

[jcfr]

And if could we have an option to disable GC ? That would avoid cleanup when setting up the system locally on a development workstation.

…

On Mon, Jan 23, 2017 at 9:54 AM, Zach Mullen ***@***.***> wrote: As part of this, we should try to mitigate the busy loop behavior while we wait on the docker container to finish. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANXo500cb6HcOe4xsqt05IcTDxzgoyfks5rVL8YgaJpZM4LjKgZ> .

-- +1 919 869 8849

[cjh1]

Fixed by #96