Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add port range support to docker compose #1191

Closed
wants to merge 3 commits into from
Closed

Conversation

yuval-k
Copy link

@yuval-k yuval-k commented Mar 26, 2015

fixes #1102
tests passed

@aanand
Copy link

aanand commented Mar 26, 2015

Thanks for contributing!

  • Could you add some integration tests?
  • Could any of this logic be moved into docker-py?

return ["%s%s" % (port, protocol)]

if len(parts) == 2:
return ["%s%s" % (p, protocol) for p in range(int(parts[0]), int(parts[1]) + 1)]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can anyone confirm this needs to be handled by the client?

It seems like it should be handled by dockerd. I can't find any reference to it in the remote api docs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I looked at the docker API, the data structure that I saw there only contained one port, that's why I chose to implement it on the client.
see ExposedPorts at https://docs.docker.com/reference/api/docker_remote_api_v1.17/

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also looking at the docker client source code, you can also see that the ports are used individually.
see runconfig/parse.go line 229-238 (the line with ParsePortRange)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for integration tests - I can add those later tonight, but let's first decide if the general idea is acceptable. i.e. it should be done here, vs the docker server and docker-py

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like Docker does indeed do it client-side, so that's where we'll need to do it too. Moving it to the server is a discussion to be had on docker/docker.

Paging @shin-: does processing of port ranges belong in docker-py, in your opinion?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, possibly. But probably as a function in the tools module rather than inside the API methods.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shin- Thanks, I agree - I think we can move build_port_bindings and its dependencies (to_port_range, split_port, add_port_mapping) all over to docker.utils, perhaps as a submodule (docker.utils.ports).

@uvgroovy Could you submit a PR to docker-py?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aanand So basically move build_port_bindings and deps to docker.utils.ports and then import it in compose/service.py?

I can do that, just one question -
doing that will temporary break the code, unless requirments-dev is updated.
so basically, i think we need to:

  1. add build_port_bindings to docker-py
  2. release new docker-py
  3. remove build_port_bindings from compose/service.py, import it from docker-py and update requirements-dev.txt to the new docker-py version

sounds good? any comments\feedback?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should go:

  1. Submit PR to docker-py
  2. Update this PR with the code changes and set the docker-py requirement in requirements.txt to point at your repo and branch with a git URL (see the pyinstaller requirement in requirements-dev.txt for an example)
  3. When the docker-py PR is merged, update the git URL to point at the docker-py merge commit
  4. When the next docker-py is released, update requirements.txt to the stable version
  5. Merge this PR

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good.

@dougborg
Copy link

dougborg commented Apr 3, 2015

Hey @uvgroovy, just wanted to check in with you to see if this was still moving forward. Do you need any help with testing, etc.? Let me know. I'd be glad to help any way I can.

@yuval-k
Copy link
Author

yuval-k commented Apr 7, 2015

Hi, Sorry for the delay, jewish holidays are slowing me down...
I got the changes to my branch in the docker-py repository, and now just trying to verify that it works with compose, and then i'll submit a PR for docker-py

@yuval-k
Copy link
Author

yuval-k commented Apr 11, 2015

I updated the version of requests to version 2.5.2 to match the one required from docker-py;

but now, I encounter this error:
ImportError: No module named 'requests.packages.chardet.sys'
see:
http://stackoverflow.com/questions/28775276/pyinstaller-error-importerror-no-module-named-requests-packages-chardet-sys

@dougborg \ @aanand would love for some advice :)

@dougborg
Copy link

I tried to reproduce this error on my local development environment with your branch, but was unable to. Can you provide any more details as to what your setup looks like and how you are invoking your tests?

I am running:

OSX Yosemite 10.10.2
docker-machine version 0.2.0 (48bea37)
docker-compose 1.1.0 (ee64e0d)
docker 1.6-rc4

@yuval-k
Copy link
Author

yuval-k commented Apr 13, 2015

So basically, When I tried to build compose with the docker-py repo directly. this is my requirements.txt:

PyYAML==3.10
git+https://github.com/uvgroovy/docker-py.git@8028c988ce363345988bc23444f08e27394a7fb5#egg=docker-py
dockerpty==0.3.2
docopt==0.6.1
requests==2.2.1
six==1.7.3
texttable==0.8.2
websocket-client==0.11.0

it complained that requests 2.2.1 is too old:

./script/test
...
Installed /usr/local/lib/python2.7/dist-packages/docker_compose-1.1.0-py2.7.egg
Processing dependencies for docker-compose==1.1.0
error: Installed distribution requests 2.2.1 conflicts with requirement requests>=2.5.2
...

so I changed the requirements.txt for compose in my branch to contain requests 2.5.2 and got the error i mentioned.

 ImportError: No module named 'requests.packages.chardet.sys'

@awsmsrc
Copy link

awsmsrc commented Apr 17, 2015

hey guys whats happening with this?
Anything I can do to help?

@aanand
Copy link

aanand commented Apr 17, 2015

Same issue as #1075 and #1269 - we're waiting for an upstream fix in requests (https://github.com/kennethreitz/requests/pull/2533) to make it into a release.

@yuval-k
Copy link
Author

yuval-k commented Apr 17, 2015

@aanand , perhaps can you assist with merging the docker-py PR?
docker/docker-py#544

@yuval-k
Copy link
Author

yuval-k commented Apr 23, 2015

requests 2.6 was released with a fix to this issue. updating both docker-py and compose to 2.6.2 seems to be working (tried it in my own laptop and the tests pass) - but here jenkins is specifically looking for requests>=2.2.1,<2.6
and as a result uses
Best match: requests 2.5.3

I couldn't find a reference to requests "<2.6" in the repo it self.

Any idea why that happens\how to fix this?

@aanand
Copy link

aanand commented May 6, 2015

Sorry I missed this. Needs rebasing on master, then it should pick up the new requests version range.

@yuval-k
Copy link
Author

yuval-k commented May 16, 2015

i re-did my commit on the master from upstream compose. all should be well now

@aanm
Copy link

aanm commented May 17, 2015

How will this behave if we scale containers that are greater than the number of ports available?

@yuval-k
Copy link
Author

yuval-k commented May 17, 2015

@aanm not sure what you are asking. this PR just allows you to specify multiple ports in the configuration. it is as if you wrote them individually (i.e. the code 'translates' the port range to multiple single ports for the rest of the program).

@aanm
Copy link

aanm commented May 17, 2015

@uvgroovy Currently, compose sends a user error if you try to scale a container that has ports defined. For example if you have port: 8888 and you want to scale that service to 2, it won't work because the compose.yml only has one port defined. What I was asking is, if the same occurs if you have port: 8888-8889 and you try to scale that service to 3.

@aanand
Copy link

aanand commented May 18, 2015

@aanm The same will happen, yes.

It would be nice to be able to specify a set of ports where each one is assigned to a single container in the service, but it's not what the Docker CLI's port range syntax means, so we'd need to come up with an alternate way of specifying it.

@aanm
Copy link

aanm commented May 18, 2015

@aanand I think it would also be nice to NOT throw an error on the case I was talking about, either with a range or a single port specification. I think it would be better to just throw a warning. I'm saying this because compose + swarm fail if we try to scale up with a single port defined.

@dnephin
Copy link

dnephin commented May 18, 2015

Replacing the error with a warning seems appropriate now that swarm makes that case possible.

@aanm
Copy link

aanm commented May 26, 2015

@dnephin look at #1466

@yuval-k
Copy link
Author

yuval-k commented Jul 6, 2015

@aanand re-did the commit to the updated compose master.

@aanand
Copy link

aanand commented Jul 7, 2015

Thanks!

LGTM

@aanand
Copy link

aanand commented Jul 7, 2015

Strike that - still needs docs.

yuval-k added 3 commits July 28, 2015 16:42
Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
@yuval-k
Copy link
Author

yuval-k commented Jul 28, 2015

@aanand sorry for the delay (i missed your last comment) - I added documentation to yml.md - please let me know if that works.

Also, while it seems that tests failed, it actually looks like an IT issue:
"unable to access 'https://github.com/docker/compose.git/': server certificate verification failed"
any way to re-run the tests?

@@ -102,7 +102,7 @@ An entry with the ip address and hostname will be created in `/etc/hosts` inside
### ports

Expose ports. Either specify both ports (`HOST:CONTAINER`), or just the container
port (a random host port will be chosen).
port (a random host port will be chosen). You can specify a port range instead of a single port (`START-END`). If you use a range for the container ports, you may specify a range for the host ports as well. both ranges must be of equal size.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @moxiegirl

My take: this seems to suggest on first reading that you can specify a range of container ports but a single host port, which I don't think is correct. Perhaps:


Expose ports. You can specify just a container port (which will be mapped to a random host port), or both (HOST:CONTAINER).

You can optionally pass in ranges of port numbers. If you're specifying host ports as well as container ports, the host and container port ranges must be the same size.

@yangxuesong
Copy link

the lastest release 1.4.0rc2 still not support port range.
when will it be available?

@mnowster
Copy link

Merged in #1827

@mnowster mnowster closed this Aug 14, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ports not accepting a range in yml