add port range support to docker compose #1191

yuval-k · 2015-03-26T14:22:27Z

fixes #1102
tests passed

aanand · 2015-03-26T17:24:57Z

Thanks for contributing!

Could you add some integration tests?
Could any of this logic be moved into docker-py?

dnephin · 2015-03-26T17:38:07Z

compose/service.py

+        return ["%s%s" % (port, protocol)]
+
+    if len(parts) == 2:
+        return ["%s%s" % (p, protocol) for p in range(int(parts[0]), int(parts[1]) + 1)]


Can anyone confirm this needs to be handled by the client?

It seems like it should be handled by dockerd. I can't find any reference to it in the remote api docs.

When I looked at the docker API, the data structure that I saw there only contained one port, that's why I chose to implement it on the client.
see ExposedPorts at https://docs.docker.com/reference/api/docker_remote_api_v1.17/

Also looking at the docker client source code, you can also see that the ports are used individually.
see runconfig/parse.go line 229-238 (the line with ParsePortRange)

As for integration tests - I can add those later tonight, but let's first decide if the general idea is acceptable. i.e. it should be done here, vs the docker server and docker-py

Looks like Docker does indeed do it client-side, so that's where we'll need to do it too. Moving it to the server is a discussion to be had on docker/docker.

Paging @shin-: does processing of port ranges belong in docker-py, in your opinion?

Hmm, possibly. But probably as a function in the tools module rather than inside the API methods.

@shin- Thanks, I agree - I think we can move build_port_bindings and its dependencies (to_port_range, split_port, add_port_mapping) all over to docker.utils, perhaps as a submodule (docker.utils.ports).

@uvgroovy Could you submit a PR to docker-py?

@aanand So basically move build_port_bindings and deps to docker.utils.ports and then import it in compose/service.py?

I can do that, just one question -
doing that will temporary break the code, unless requirments-dev is updated.
so basically, i think we need to:

add build_port_bindings to docker-py

release new docker-py

remove build_port_bindings from compose/service.py, import it from docker-py and update requirements-dev.txt to the new docker-py version

sounds good? any comments\feedback?

I think it should go:

Submit PR to docker-py

Update this PR with the code changes and set the docker-py requirement in requirements.txt to point at your repo and branch with a git URL (see the pyinstaller requirement in requirements-dev.txt for an example)

When the docker-py PR is merged, update the git URL to point at the docker-py merge commit

When the next docker-py is released, update requirements.txt to the stable version

Merge this PR

sounds good.

dougborg · 2015-04-03T23:14:23Z

Hey @uvgroovy, just wanted to check in with you to see if this was still moving forward. Do you need any help with testing, etc.? Let me know. I'd be glad to help any way I can.

yuval-k · 2015-04-07T16:35:38Z

Hi, Sorry for the delay, jewish holidays are slowing me down...
I got the changes to my branch in the docker-py repository, and now just trying to verify that it works with compose, and then i'll submit a PR for docker-py

yuval-k · 2015-04-11T12:57:51Z

I updated the version of requests to version 2.5.2 to match the one required from docker-py;

but now, I encounter this error:
ImportError: No module named 'requests.packages.chardet.sys'
see:
http://stackoverflow.com/questions/28775276/pyinstaller-error-importerror-no-module-named-requests-packages-chardet-sys

@dougborg \ @aanand would love for some advice :)

dougborg · 2015-04-13T04:18:30Z

I tried to reproduce this error on my local development environment with your branch, but was unable to. Can you provide any more details as to what your setup looks like and how you are invoking your tests?

I am running:

OSX Yosemite 10.10.2
docker-machine version 0.2.0 (48bea37)
docker-compose 1.1.0 (ee64e0d)
docker 1.6-rc4

yuval-k · 2015-04-13T12:53:05Z

So basically, When I tried to build compose with the docker-py repo directly. this is my requirements.txt:

PyYAML==3.10
git+https://github.com/uvgroovy/docker-py.git@8028c988ce363345988bc23444f08e27394a7fb5#egg=docker-py
dockerpty==0.3.2
docopt==0.6.1
requests==2.2.1
six==1.7.3
texttable==0.8.2
websocket-client==0.11.0

it complained that requests 2.2.1 is too old:

./script/test
...
Installed /usr/local/lib/python2.7/dist-packages/docker_compose-1.1.0-py2.7.egg
Processing dependencies for docker-compose==1.1.0
error: Installed distribution requests 2.2.1 conflicts with requirement requests>=2.5.2
...

so I changed the requirements.txt for compose in my branch to contain requests 2.5.2 and got the error i mentioned.

 ImportError: No module named 'requests.packages.chardet.sys'

awsmsrc · 2015-04-17T14:18:27Z

hey guys whats happening with this?
Anything I can do to help?

aanand · 2015-04-17T15:14:32Z

Same issue as #1075 and #1269 - we're waiting for an upstream fix in requests (https://github.com/kennethreitz/requests/pull/2533) to make it into a release.

yuval-k · 2015-04-17T15:20:02Z

@aanand , perhaps can you assist with merging the docker-py PR?
docker/docker-py#544

yuval-k · 2015-04-23T21:51:34Z

requests 2.6 was released with a fix to this issue. updating both docker-py and compose to 2.6.2 seems to be working (tried it in my own laptop and the tests pass) - but here jenkins is specifically looking for requests>=2.2.1,<2.6
and as a result uses
Best match: requests 2.5.3

I couldn't find a reference to requests "<2.6" in the repo it self.

Any idea why that happens\how to fix this?

aanand · 2015-05-06T11:13:22Z

Sorry I missed this. Needs rebasing on master, then it should pick up the new requests version range.

yuval-k · 2015-05-16T14:39:28Z

i re-did my commit on the master from upstream compose. all should be well now

aanm · 2015-05-17T14:14:51Z

How will this behave if we scale containers that are greater than the number of ports available?

yuval-k · 2015-05-17T14:39:29Z

@aanm not sure what you are asking. this PR just allows you to specify multiple ports in the configuration. it is as if you wrote them individually (i.e. the code 'translates' the port range to multiple single ports for the rest of the program).

aanm · 2015-05-17T17:27:10Z

@uvgroovy Currently, compose sends a user error if you try to scale a container that has ports defined. For example if you have port: 8888 and you want to scale that service to 2, it won't work because the compose.yml only has one port defined. What I was asking is, if the same occurs if you have port: 8888-8889 and you try to scale that service to 3.

aanand · 2015-05-18T10:14:46Z

@aanm The same will happen, yes.

It would be nice to be able to specify a set of ports where each one is assigned to a single container in the service, but it's not what the Docker CLI's port range syntax means, so we'd need to come up with an alternate way of specifying it.

aanm · 2015-05-18T13:52:16Z

@aanand I think it would also be nice to NOT throw an error on the case I was talking about, either with a range or a single port specification. I think it would be better to just throw a warning. I'm saying this because compose + swarm fail if we try to scale up with a single port defined.

dnephin · 2015-05-18T14:56:56Z

Replacing the error with a warning seems appropriate now that swarm makes that case possible.

aanm · 2015-05-26T22:37:31Z

@dnephin look at #1466

yuval-k · 2015-07-06T16:27:54Z

@aanand re-did the commit to the updated compose master.

aanand · 2015-07-07T10:25:02Z

Thanks!

LGTM

aanand · 2015-07-07T10:25:40Z

Strike that - still needs docs.

Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>

yuval-k · 2015-07-28T21:31:09Z

@aanand sorry for the delay (i missed your last comment) - I added documentation to yml.md - please let me know if that works.

Also, while it seems that tests failed, it actually looks like an IT issue:
"unable to access 'https://github.com/docker/compose.git/': server certificate verification failed"
any way to re-run the tests?

aanand · 2015-07-29T10:19:29Z

docs/yml.md

@@ -102,7 +102,7 @@ An entry with the ip address and hostname will be created in `/etc/hosts` inside
 ### ports

 Expose ports. Either specify both ports (`HOST:CONTAINER`), or just the container
-port (a random host port will be chosen).
+port (a random host port will be chosen). You can specify a port range instead of a single port (`START-END`). If you use a range for the container ports, you may specify a range for the host ports as well. both ranges must be of equal size.


ping @moxiegirl

My take: this seems to suggest on first reading that you can specify a range of container ports but a single host port, which I don't think is correct. Perhaps:

Expose ports. You can specify just a container port (which will be mapped to a random host port), or both (HOST:CONTAINER).

You can optionally pass in ranges of port numbers. If you're specifying host ports as well as container ports, the host and container port ranges must be the same size.

yangxuesong · 2015-07-30T08:45:00Z

the lastest release 1.4.0rc2 still not support port range.
when will it be available?

mnowster · 2015-08-14T09:00:45Z

Merged in #1827

dnephin reviewed Mar 26, 2015
View reviewed changes

yuval-k force-pushed the master branch from afeaf92 to 5cce18a Compare April 10, 2015 10:43

yuval-k mentioned this pull request Apr 11, 2015

parse port ranges docker/docker-py#544

Merged

GordonTheTurtle removed the dco/no label Apr 23, 2015

yuval-k force-pushed the master branch from 8b8c0fd to e2df017 Compare April 23, 2015 20:47

GordonTheTurtle removed dco/no labels Apr 23, 2015

yuval-k force-pushed the master branch from 9c644c2 to d32fce2 Compare April 23, 2015 21:38

GordonTheTurtle removed the dco/no label Apr 23, 2015

yuval-k force-pushed the master branch from 6f6fb73 to ac3cbb4 Compare May 16, 2015 14:31

GordonTheTurtle added the dco/no label May 16, 2015

yuval-k force-pushed the master branch from ac3cbb4 to f3eb3e4 Compare May 16, 2015 14:33

GordonTheTurtle removed the dco/no label May 16, 2015

GordonTheTurtle added the dco/no label May 16, 2015

yuval-k force-pushed the master branch from 49fe16d to 6cd3b00 Compare May 16, 2015 20:44

GordonTheTurtle removed the dco/no label May 16, 2015

GordonTheTurtle added the dco/no label May 18, 2015

yuval-k force-pushed the master branch from e348c41 to 59af3a9 Compare May 18, 2015 20:21

GordonTheTurtle removed the dco/no label May 18, 2015

aanand mentioned this pull request Jul 2, 2015

ports not accepting a range in yml #1102

Closed

yuval-k force-pushed the master branch from 59af3a9 to 481cd44 Compare July 6, 2015 16:17

GordonTheTurtle added the dco/no label Jul 6, 2015

yuval-k force-pushed the master branch from 481cd44 to 08005bd Compare July 6, 2015 16:19

GordonTheTurtle removed the dco/no label Jul 6, 2015

yuval-k added 3 commits July 28, 2015 16:42

rebasing port range changes

46fe867

Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>

fixed merge issue from previous commit

5e64b15

Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>

ports documentation

ea9e865

Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>

yuval-k force-pushed the master branch from 08005bd to ea9e865 Compare July 28, 2015 21:27

aanand reviewed Jul 29, 2015
View reviewed changes

mnowster closed this Aug 14, 2015

add port range support to docker compose #1191

add port range support to docker compose #1191

Conversation

yuval-k commented Mar 26, 2015

aanand commented Mar 26, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dougborg commented Apr 3, 2015

yuval-k commented Apr 7, 2015

yuval-k commented Apr 11, 2015

dougborg commented Apr 13, 2015

yuval-k commented Apr 13, 2015

awsmsrc commented Apr 17, 2015

aanand commented Apr 17, 2015

yuval-k commented Apr 17, 2015

yuval-k commented Apr 23, 2015

aanand commented May 6, 2015

yuval-k commented May 16, 2015

aanm commented May 17, 2015

yuval-k commented May 17, 2015

aanm commented May 17, 2015

aanand commented May 18, 2015

aanm commented May 18, 2015

dnephin commented May 18, 2015

aanm commented May 26, 2015

yuval-k commented Jul 6, 2015

aanand commented Jul 7, 2015

aanand commented Jul 7, 2015

yuval-k commented Jul 28, 2015

Choose a reason for hiding this comment

yangxuesong commented Jul 30, 2015

mnowster commented Aug 14, 2015