Feature/env

[SkinyMonkey]

This feature allow you to use environment variables in the fig.yml file.
It simply expand any found environment variable (ex: $HOST) to the value defined in the environment running fig.

useful example :

I'm using Skydock (https://github.com/crosbymichael/skydock) along with skydns.

When i launch a skydns container by hand I need to specify the ip of the docker0 interface so that docker inject a new nameserver into /etc/resolv.conf:

ex:

Here the ip is "172.17.42.1"

docker run -d -dns 172.17.42.1 -p 172.17.42.1:53:53/udp -name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker

A nice way to be more flexible, compute what is the address automatically for example, is to add an environment variable in the equation :

env DNS_IP=172.17.42.1 docker run -d -dns $DNS_IP -p $DNS_IP:53:53/udp -name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker

Wouldn't it be nice to be able to do the same thing in a fig.yml file?

ex:

env DNS_IP=172.17.42.1 fig up

with this fig.yml:

skydns:
    image: crosbymichael/skydns
    ports:
        - $DNS_IP:53:53/udp
        - 8080:8080
    dns:
        - $DNS_IP
    command: "-nameserver 8.8.8.8:53 -domain docker"

I wanted to add a flag to make this expansion optional, just in case. What do you think?

[bfirsh]

Thanks! Will take a look over this properly in a bit. You might also be interested in #64. /cc @sherzberg

[SkinyMonkey]

It's an extremly quick/simple feature to implement.
But some problem might rise from it, that's why i wanted to discuss about it : to be sure that i would not unfortunately forget something.

Yes, i read about #64, and i have some comments to elaborate before i post.

[hugorodgerbrown]

+1 for this. I want to be able to inject file paths in the volumes section of fig.yml so that we can commit our project fig.yml, but allow devs to put their project directories wherever they please.

web:
  build: .
  ports:
   - 5000:5000
  volumes:
   - .:/app
  links:
   - redis
   - db

becomes:

web:
  build: $DOCKER_ROOT
  ports:
   - 5000:5000
  volumes:
   - $APP_ROOT:/app
  links:
   - redis
   - db

[mzdaniel]

+1 for implementing os.path.expandvars . Although we understand in theory why it might not be advisable to have envvars in the fig.yml, in practice we don't want to have credentials in our repositories, and the fig file gives very important structural and actionable information to be left out of the repos.

[aanand]

I agree that this would solve a pressing problem (secrets in version control), but I'm also a fan of #64, and I'm not yet sure how we'd get both features to coexist nicely.

One option would be to use $ for one type of environment variable expansion, and another glyph, say %, for the other. For example:

volumes:
  - ${APP_ROOT}:/app
environment:
  - DATABASE_URL=mysql://user:password@%{DB_1_PORT_3306_TCP_ADDR}/db

I haven't fully thought through the implications and complexity of implementation, though.

[thestonefox]

Would it be possible to extend this feature to having fig read a settings file (e.g. .env)

Then you could have something like

production.env
$dbname=this
$dbpassword=that

staging.env
$dbname=this
$dbpassword=that

fig.yml
environment:

• dbname=$dbname
• dbpassword=$dbpassword

then do something like fig up production and this would load in the variables from the production.env file.

[Jud]

One of the things that was added to maestro-ng (signalfx/maestro-ng#8) is simply preprocessing the yaml with Jinja so the yaml looks like:

environment:
  - DB_PASSWORD: {{ env.DB_PASSWORD }}

I can't say I thought of the idea, SaltStack and Rails both preprocess their yaml files with Jinja and erb respectively.

To achieve something like @thestonefox wants, you could just use envdir and do envdir production.env fig up (though it would be a directory instead of a file).

[bbradbury]

+1 for this.

[zachlatta]

+1 as well

[SkinyMonkey]

@Jud : Jinja could be great, but i think it's a huge dependency to add.
I'll let you all decide if you want to be able to add some logic in the fig.yml file or not.

I see basic cases, like default values, where logic in the yaml file could be seen as useful:

example:

environment:
      - DB_HOST={{ env.DB_HOST if env.DB_HOST else "localhost" }}

But this kind of cases should be handle at the application configuration level instead.

example:

app.config['DB_HOST'] = getenv('DB_HOST') or 'localhost'

Right now I don't see any case where Jinja could be really useful.

Do you?

[hunterloftis]

@SkinyMonkey could you explain a little about how you've got skydock working with fig? There are a lot of things in the skydock install that I wasn't sure how to use with fig, like udp ports and providing the docker0 interface IP.

[hugorodgerbrown]

(slight tangent alert) I just found out about Skydock last night, and although it doesn't solve the ENV VAR expansion issue (this issue), it does seem to offer a solution to a lot of the issues to the resolution of dynamic IPs. I'd recommend anyone who hasn't played around with it to take a look.

[SkinyMonkey]

@hunterloftis : http://pastebin.com/89PeJ1sC

Sorry for polluting this thread, but might be useful to other people.

[analytically]

+1 for Jinja. Check out envtpl too. https://github.com/andreasjansson/envtpl

[rweng]

looking so forward to this

[robbiev]

+1

[djmaze]

+1

[bajohns]

+1

[drupalmodule]

+1
I currently have to use a calling shell script and see the yml file. It would be great to eliminate it.

[webwurst]

Maybe Jinja2 could be a some kind of an optional dependency? So when fig is starting up it looks for a file like fig.yml.jinja2. It creates a fig.yml from that if jinja2 is installed and goes on as usual.

[mozz100]

Because fig uses yaml.load, you can already pass environment variables from the machine where you're running 'fig' to the containers. If you're on a Mac and you wanted your containers to know the IP address of the boot2docker VM, for instance:

  environment:
    SOME_VALUE: 123
    DOCKER_HOST: !!python/object/apply:os.getenv ["DOCKER_HOST"]

This feels like a hack, and it feels a bit wrong that fig doesn't use yaml.safe_load. Makes all kinds of potentially nefarious things possible, I think. I'm about to submit a pull request which, if accepted, would deliberately stop this from working - I guess it's up to the community whether this is desirable or not.

[d11wtq]

@mozz100 that's interesting, though yeah, I don't think it's a good idea to have end-users doing that. In particular, fig may not always be written in Python (there's a current effort underway by a few devs to do it in Go).

[dmitriy-kiriyenko]

Any news on such a feature? #297 isn't sufficient for me, because I want to supply from host things like volumes, not just environment variables.

Any templating of fig.yml would be great. This one, with expandvars is particularly great, but I would be happy with any way of doing:

volumes:
  - ${something}:/app

and being able to supply something on fig run.

[dmitriy-kiriyenko]

If we don't want loading jinja2, what about just using string.Template with a whitelist of keys – not ultimately substitute it from env vars, but provide a list of substitutions on run:

volumes:
  - ${something}:/app

$ fig run --template something:/var/www/cache

[dnephin]

Volumes already support environment variables in this form, at least for the host portion: https://github.com/docker/fig/blob/master/fig/service.py#L490

I'm not a big fan of "just pass everything to a substitution (I think it's error prone). A lot of fields support environment variables now, it shouldn't be difficult to add more support.

[dnephin]

For your example:

something="/var/www/cache" fig run

[dmitriy-kiriyenko]

Wow. They do. Thank you.

[funkymonkeymonk]

I also ran into a situation where I would like to use this. I would like to tag each container built on our CI server with a build number. having a function like this would allow for us to deploy not only the latest, but also roll back to a previous successful build just buy putting the build number in.

[dnephin]

@funkymonkeymonk I think you can already do this using

FIG_PROJECT_NAME=$BUILD_NUMBER fig build

#457 is also related to this idea

[funkymonkeymonk]

@dnephin Thanks for the response. If I'm not mistaken wouldn't that name the container with the build number not tag it? That would create a large number of repos quickly when I push. Also when I go to deploy the stack I'd like to deploy the stack passing in the build number as an argument. Something to the effect of

FIG_CONTAINER_TAG=$BUILD_NUMBER fig up

This would give us the ability to roll back to a previous build easier.

[dnephin]

Yes, it would be part of the name. What you're looking for sounds a lot like #457

[hugoduncan]

I would like to use environment vars in the command: arguments for a service. In my case I would like to get etcd to bind to a public ip that is specified based on $DOCKER_HOST. Environment expansion would allow the writing of a wrapper script for fig, which sets an environment variable based on $DOCKER_HOST, and use it in fig.xml. A more general templating solution might allow writing this without the need for a wrapper script for fig.

[manolitto]

+1

Trying to solve service lookups via dns and would love to beeing able to use the very same fig.yml for development and production. To solve this I would need something like
dns: ${MYPROJECT_DNS_IP}
By running a local dns container, I could then easily tweak some of my needed services to my local development machine.

[albers]

I would like to use variables in ports:

I have a usecase where several services share the same public default port on the docker host.
Each service is exclusively bound to a public IP of the Docker host using ports like "10.0.10.0:80:8080".
This forces me to hard code the IP of the Docker host into the fig files, which makes the fig files unportable.
I would like to use variables in ports, like "${DOCKER_HOST_BASE}.0:80:8080"

[Stono]

+1

[djschilling]

+1

[hadim]

Any news here ?

[relwell]

Please refer to #845 for the implementation as outlined by some of the repo admins, as discussed in #495

[muddydixon]

+1

[hamiltont]

+1

[ysaotome]

+1

[ajmath]

+1 Would like to use this to pass in AWS credentials into containers

[dnephin]

#1488 will be in the next release, which will let you do something like this if you wanted to:

cat docker-compose.yml | python -c "import os, sys; os.path.expandvars(sys.stdin.read())" | docker-compose up -f -

Which can be aliased as dc or put into a script along with the project.

I think it's still work calling out that this can easily break the yaml syntax, so should be used with caution.

[griwes]

This is a non-solution, @dnephin. Heck, it might even be an unsolution.

When invoking docker run yourself, you have an option to specify important parameters with (shell expanded) env variables... like the IP to expose ports to, which in most cases can't be statically embedded in the .yml file. So, while this might be an extremely specific thing, the one absolutely crucial feature is a way to reference an IP of an interface in Compose's .yml file.

Wrapping an invocation of a tool with something like the above basically means the tool in question sucks majorly.

[albers]

I agree that the need to interpolate IP addresses in a Compose file is a legal request that is insufficiently addressed by this solution.

The suggested solution is only a minor improvement to command substitution which was possible before:

IP=1.2.3.4
docker-compose -f <(sed "s/IP/$IP/" docker-compose.yml) -p projectname ps

[dnephin]

@albers that's true,

#1377 is still tracking a more robust solution for this problem

[jayfk]

I totally agree with @griwes, this looks like an unsolution.

[dnephin]

This is done in #1765