run weave on remote docker

[tcolar]

Say I want to run weave on two remote hosts that I'm connected to via there docker daemon, could I invoke the weave router on them via docker (without running commands via SSH)

something like
DOCKER_HOST=tcp://someremote:2375 docker run weaveimg weave launch
DOCKER_HOST=tcp://otherremote:2375 docker run weaveimg weave launch someremote

I assume if it can work at all, i would have to user --privileged and -host and maybe other things ..
In others words can weave itself be run within a container ?

Is that possible at all ? is that documented anywhere ?

Thank you.

[rade]

Is that possible at all?

It should be.

Is that documented anywhere?

Would be great if you took a stab at getting this to work.

1. build an image that contains the weave script an all required dependencies, e.g. docker, ip, iptables. It will be easiest to derive this from an ubuntu or debian base image and then simply use apt to install the dependencies. Ultimately one would want to construct a much smaller image, but that can wait.
2. make the weave script the entry point of the image.
3. modify the weave script to access /hostproc instead of /proc. See don't wrap docker #230 (comment)
4. run weave like this: DOCKER_HOST=tcp://someremote:2375 docker run --rm --net=host --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc weaveimg <weavecmd> <weavearg> ...

[rade]

I was bored...

The Dockerfile

FROM ubuntu
MAINTAINER Weaveworks Inc <help@weave.works>
WORKDIR /usr/local/bin
RUN apt-get -y update
RUN apt-get -y install curl iptables
RUN curl -o docker https://get.docker.com/builds/Linux/x86_64/docker-latest && chmod +x docker
# The following should be changed to point to https://github.com/zettio/weave/releases/download/latest_release/weave once https://github.com/zettio/weave/commit/cce5d2417a75c096546fb3bfbfd975d0a3a24723 has made it into a release
RUN curl -o weave https://raw.githubusercontent.com/zettio/weave/master/weave && chmod +x weave
ENTRYPOINT ["/usr/local/bin/weave"]

will construct a zettio/weaveimg image which runs the weave script in a container. Run it with

docker run --rm --net=host --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /proc:/hostproc -e PROCFS=/hostproc zettio/weaveimg <weave_cmd> <weave_arg> ...

I haven't done much testing with this, but so far everything appears to work just fine. Remotely too, though one obviously needs to get the zettio/weaveimg into the remote docker by either building it there or publishing it to the dockerhub or docker saveing it locally and then docker loading it into the remote docker.

[tcolar]

Awesome, thanks.
On Jan 4, 2015 5:17 AM, "rade" notifications@github.com wrote:

Closed #312 #312.

—
Reply to this email directly or view it on GitHub
#312 (comment).

[rade]

I am re-opening this since conceivably this is a feature weave should support more directly.

E.g. when DOCKER_HOST is set, the weave script could automatically delegate the command to that host via the mechanism described above. Or we could go one step further and always do that, i.e. weave would become a tiny script that simply runs a container executing the 'real' weave script.

[rade]

Some thoughts on the image we need to construct for this...

• The script doesn't have a huge number of dependencies, but still a few - ip, iptabes, grep, sed, rm, ln, mkdir, cat, cut, tr and of course a shell (e.g. dash). Building a minimal image by packaging these ourselves is a pain. The stock busybox image has everything we need and weighs in at just 2.6M, so I reckon that is the way to go.
• Instead of constructing yet another image, we could extend the weavetools image. Which, if we go the busybox or similar route, could also allow us to switch to dynamic linking, against libc at least, thus reducing image size.

[binocarlos]

I've just come across comments by @progrium pointing to Alpine Linux - might this be a useful base image because it has a more complete package manager? (to fetch all the little bits and pieces)

[squaremo]

As an approximation,

FROM gliderlabs/alpine
WORKDIR /home/weave
RUN ["apk-install", "ethtool", "conntrack-tools", "curl", "iptables"]
ADD ./weave /home/weave/

gives an image

squaremo/weavetools    latest  6a1c064ee3c3   27 seconds ago 11.44 MB

Adding the proxy is another 6MB or so; the docker client is another 14MB.

The docker exe can be bind-mounted from the host, but another idea struck me: perhaps the script can be split into those parts that deal with dockerating (to live in the "remote" script), and those parts that enact networking config (to live in the inner, container-borne script).

[squaremo]

weave would become a tiny script that simply runs a container executing the 'real' weave script.

https://github.com/squaremo/weave/tree/weave_in_weavetools is a PoC of putting the weave script in a container and having a "thin client" script that drives it. It's enough to be able to run weave launch at least.

[squaremo]

Notes on using busybox:

• Busybox includes wget, but it's not featureful enough to be usable (it doesn't appear to support --method for instance). curl adds about 2MB to the image.
• sed has different options (it needs -r to enable extended regexes) and different regex syntax (no escaping of parens, plus, etc.)
• It needs iproute2 installed to support the uses of ip in the driver script

[rade]

• sed has different options (it needs -r to enable extended regexes) and different regex syntax (no escaping of parens, plus, etc.)

Does that indicate our use of sed is not Posix? It's supposed to be!

• It needs iproute2 installed to support the uses of ip in the driver script

root@xps:~# docker run -ti busybox
/ # ip -V
ip utility, iproute2-ss131122

What's the problem?

[rade]

Or perhaps your notes were in relation to alpine.

Btw, conflating build and runtime in one image, and consequently shipping an image that contains a package manager and init.d, is a dreadful idea. Though I am prepared to hold my nose on that one.

[progrium]

I, too, prefer separate build and runtime images when you need to build.
Though with some exceptions, such as dynamic languages.

On Tue, Jan 27, 2015 at 11:33 AM, rade notifications@github.com wrote:

Or perhaps your notes were in relation to alpine.

Btw, conflating build and runtime in one image, and consequently shipping
an image that contains a package manager and init.d, is a dreadful idea.
Though I am prepared to hold my nose on that one.

—
Reply to this email directly or view it on GitHub
#312 (comment).

Jeff Lindsay
http://progrium.com

[dpw]

sed has different options (it needs -r to enable extended regexes) and different regex syntax (no escaping of parens, plus, etc.)

The "no escaping of parens, plus, etc." is the extended regex syntax. What led to the conclusion that basic regular expressions are not sufficient? If there are sed scripts where we go outside of BREs, it is probably best to try to adapt them to conform before switching to EREs.

[squaremo]

The "no escaping of parens, plus, etc." is the extended regex syntax.

Right yes, I wasn't being very scientific, by which I mean accurate.

If there are sed scripts where we go outside of BREs, it is probably best to try to adapt them to
conform before switching to EREs.

Sure. GNU sed supports \+ as an extension, which is presumably how that slipped through.