Pass variables inside fig.yml on runtime

[dkapanidis]

Is it possible to configure a fig.yml in such way that you can pass on execution time variables instead of hard-wiring them inside the fig.yml?

I think a generic variable injection during execution would be quite useful for various use cases.

e.g:

jenkins:
  image: aespinosa/jenkins:latest
  ports:
    - "8080"
  hostname: ${HOSTNAME}

HOSTNAME=ci fig up

That could inject the variable HOSTNAME inside the fig.yml during execution and execute a docker run with hostname ci.

ps. This is different than passing environment variables inside docker which is already supported (http://www.fig.sh/yml.html#environment)

[mieciu]

Good idea - i've also found it really handy when passing env. variable as a parameter in automated test run 😄
But does your example imply passing all the variables from current users shell session? I'd rather see this like in docker

fig up -d -e HOSTNAME:ci

[mieciu]

Another thing is: which container should set this inline passed variable? First one, the second one, both?

[dkapanidis]

the format fig up -d -e HOSTNAME:ci sounds good by me.

as for which container, I would say make it global for the entire file. If the author of fig.yml doesn't want to exist conflicts between containers he will use different variables to different containers, or he can share the same variable to both if needed.

[mieciu]

That's great 👍

There are two possible options right now:

• After fig up -d -e HOSTNAME:ci we pass variable to all the containers, and let's say that we can use variable names to detect, which container should use the variable, for instance:
  Having web and db container we can simply run:

fig up -d -e WEBS_ENDPOINT=192.168.0.40 -e DBS_ENDPOINT=192.168.0.50

So that besides the fact, that each container has two defined endpoints it still can use this one defined for him.

• We can also use container names like in scale=? approach, but with syntax:
  fig up -d -e <container_name>:<VARIABLE_NAME>:<VARIABLE>, but does it look good:
  fig up -d -e web:HOSTNAME:ci ??
  Maybe something like:
  fig up -d -c web -e HOSTNAME:web -e ROLE:ci -c db -e PASS:pass
  where web and db are containers names defined in fig.yml

[mieciu]

But the first idea looks definiately best to me

[dkapanidis]

can you show me a mockup of a fig.yml file and the respective CLI command
so I can understand better the first option?

are the variable names arbitrary or they have to match the container name?

2014-09-30 11:47 GMT+02:00 mieciu notifications@github.com:

But the first idea looks definiately best to me

—
Reply to this email directly or view it on GitHub
#495 (comment).

[mieciu]

Well, I thought about just arbitrary variable names, but matching them to container name will be okay as well (as FIG use it 'natively' http://www.fig.sh/env.html). So maybe this is the proper way 👍

I also don't consider modyfing fig.yml as necessary, passing them just inline, like in

fig up -d -e HOSTNAME:ci

is 100% okay to me.

[mieciu]

So let's say that :

fig up -d -e WEB_HOSTNAME:ci

will pass variable to all web containers, and if no prefix is detected (like HOSTNAME:ci ), variables will be passed to all the containers

[dkapanidis]

ok, then we're probably talking about different things.

I understand what you propose as passing variables inside the container,
but during the creation of docker command do nothing with them. My initial
proposal is to pass variables inside fig.yml before parsing it.

In my use case I need to be able to set hostname not as variable inside the
container, but as part of the docker command:

e.g. docker -h '$HOSTNAME' jenkins

which in fig.yml format translates:

jenkins:
  image: aespinosa/jenkins:latest
  hostname: ci

but that ci I'd like to define it in runtime. If I pass my vriable as you
say WEB_HOSTNAME:ci then the container needs to know what to do with it.

what you propose is already done with
http://www.fig.sh/yml.html#environment but
only the white-list of things that are defined in the fig.yml as possible
variable entries, which seems sufficient (the author of fig.yml should
explain what entries are parsable from the container)

Environment variables with only a key are resolved to their values on the machine Fig is running on, which can be helpful for secret or host-specific values
(taken from fig site)

[matleh]

I too would love to have some kind of variable expansion in the fig.yml. As of now, I have to use a template fig.yml and create the real fig.yml with some wrapper script.

My current use-case: Have functional test run agains a docker container, but be able to specify the tag of the image under test.

[sidabs]

+1 on this request!

[mrfelton]

Also looking for this support so that I can let a container know the hosts IP address so that it can connect to it.

[manolitto]

+1

[codeitagile]

+1

[dzmitry-paulenka]

This is really handy, if you store fig.yml in VCS. For example, our team uses the volumes: option, but everybody maps containers' folder to different path on the host machine. So we need a way to modify some parameters before 'fig up' execution, without changing the fig.yml itself.

These are file fragments to clarify:

# run.sh
export DIST_FOLDER=~/work/project/dist
fig up

# fig.yml
tomcat:
  image: internal/tomcat
  volumes:
    - "$DIST_FOLDER:/dist:ro"

[dkapanidis]

This is the exact same use case I have and why I proposed it: have fig.yml
in VCS and avoid changes on file to customize executions.

On Wed, Nov 26, 2014, 2:34 PM Dzmitry Paulenka notifications@github.com
wrote:

This is really handy, if you store fig.yml in VCS. For example, our team
uses the volumes: option, but everybody maps containers' folder to
different path on the host machine. So we need a way to modify some
parameters before 'fig up' execution, without changing the fig.yml itself.

These are file fragments to clarify:

run.shexport DIST_FOLDER=~/work/project/dist

fig up

fig.ymltomcat:

image: internal/tomcat
volumes:
- "$DIST_FOLDER:/dist:ro"

—
Reply to this email directly or view it on GitHub
#495 (comment).

[matleh]

My use-case is similar: I have a fig.yml defining the setup for a suite of functional tests. Now I want to be able to run these tests against different versions.

# fig.yml
app:
  image: mickey/theapp:${APP_VERSION:latest}
test:
  image: mickey/tests
  links:
   - app

# run
APP_VERSION=1.3 fig up

[bushong1]

+1

[baptistedonaux]

+1

[fmahnke]

+1

[byrnedo]

+1

[balokir]

+1

[decoomanj]

+1

Are there any plans to implement this feature?

[dnephin]

I think this feature is already partially implemented. Many fields already support environment variables.

The one example I see here that isn't implemented yet is environment variable support for image tags. That should be a small change, PRs welcomed :)

I believe all it needs is an os.environ() in fig/service.py:_get_image_name()

[decoomanj]

Yep, sorry, I was unclear. I was referring to the environment-resolution inside Fig itself.

[mingfang]

+1

[relwell]

+:100:

[jmreicha]

+1

[jlrigau]

+1

[docteurklein]

isn't it already possible by using the env_file ?
for example:

echo 'VAR=test' > env && docker-compose up -d

Might not, because "Environment variables specified in environment override these values.", which I find a little strange, cause env_file is meant to allow custom (per-install) params IMHO

[dkapanidis]

Compose.yml file does not define just env variables. So that would not work
for setting dynamically the hostname or the exposed port number etc.

On Wed, Apr 15, 2015, 9:39 AM Florian Klein notifications@github.com
wrote:

isn't it already possible by using the env_file
https://docs.docker.com/compose/yml/#env_file ?
for example:

echo 'VAR=test' > env && docker-compose up -d

Might not, because "Environment variables specified in environment
override these values."

—
Reply to this email directly or view it on GitHub
#495 (comment).

[mgasner]

+1

[justinwiley]

What are the security implications of this? I.e. can an unprivileged user can set HOSTNAME or other environment variables referenced in the YAML to something nefarious, that would lead to unintended code execution...either through failed YAML parsing or via its use after parsing?

[dkapanidis]

Well, good question, I suppose if someone has privileges to execute a user-defined compose.yml file, you should assume he 'owns' the machine, since he can activate --privileged mode on a container of his choice.

If the compose.yml file is not user-defined, then the hostname can only be set if the author of the compose.yml file is letting it be defined by variable. In that case, hostname substitution shouldn't execute any arbitrary shell commands to be resolved, otherwise command injection could be possible by unprivileged users.

[jsegura]

+1

[olalonde]

+1

[dlouwers]

👍

[ebuildy]

Come on, even Symfony 2 has variables support in YML!

Or we need something in top of docker-compose.

[goloroden]

+1

[Ovski4]

+1

[cyppan]

+1

[bobrik]

Every time you write your +1, 73 people get useless email. That doesn't make any sense. I bet maintainers got your point already.

[cyppan]

Sorry for this additional spam, but until Github supports votes in another way you're gonna see a lot of ugly, space consuming and useless +1... You have a nice car btw :p

[funkyfuture]

fyi, yesterday i requested automated censorship for '+1'- spamming content on github.

if you want something implemented, make a sponsoring offer!

[aanand]

OK! Sincere thanks for the +1s folks, it's genuinely helpful to know how many people want this feature. I've created a new issue at #1377 to track it. Head over there with your thoughts.

[olalonde]

In case anyone doesn't know, you can unsubscribe from notifications via the Unsubscribe button on the right. /meta

[phazei]

I wanted to share a rather simple temporary solution I had for variable expansion:

#!/bin/bash
apply_shell_expansion() {
    declare file="$1"
    declare data=$(< "$file")
    declare delimiter="__apply_shell_expansion_delimiter__"
    declare command="cat <<$delimiter"$'\n'"$data"$'\n'"$delimiter"
    eval "$command"
}
#in case you want to store the variables in a local file
source ./local.env
if ! [ -n "$DOCKER_DATA_PATH" ] ; then
    echo "DOCKER_DATA_PATH not set, using default value"
fi
DOCKER_DATA_PATH=${DOCKER_DATA_PATH:=//c/projects/dockerdata}

apply_shell_expansion docker-compose.tpl.yml > docker-compose.yml

Then your docker-compose.tpl.yml can be commited, each person can have local.env that's sourced into the script, and it just uses bash's own variable expansion, quick and easy.

Can either put default values for all the vars in the script, or just use the default variable syntax directly inside the tpl.yml

Edit: actually, that script seems to be bash only, since boot2docker only has sh, sh would be:

#!/bin/sh
apply_shell_expansion() {
    local file="$1"
    local data=$(cat "$file")
    local delimiter="__apply_shell_expansion_delimiter__"
    local command="cat <<$delimiter"$'\n'"$data"$'\n'"$delimiter"$'\n'""
    eval "$command"
}
#in case you want to store the variables in a local file
source ./local.env
if ! [ -n "$DOCKER_DATA_PATH" ] ; then
    echo "DOCKER_DATA_PATH not set, using default value"
fi
DOCKER_DATA_PATH=${DOCKER_DATA_PATH:=//c/projects/dockerdata}

apply_shell_expansion docker-compose.tpl.yml > docker-compose.yml

[garystafford]

For a small number of variables ('tokens'), I use a simple shell script along with a templated version of my YAML file. Here's an actual example:

files:

docker-compose-template.yml
docker-compose.yml
compose_replace.sh

run:

sh compose_replace.sh

script:

#!/bin/sh

# variables
base_url_token="{{ base_url }}" # find all these...
base_url="api.foo.com" # replace with url of public rest api
host_ip_token="{{ host_ip }}" # find all these...
host_ip=$(docker-machine ip $(docker-machine active)) # replace with ip of host running NGINX

# output
echo ${base_url_token} = ${base_url}
echo ${host_ip_token} = ${host_ip}

# find and replace
sed -e "s/${base_url_token}/${base_url}/g" \
    -e "s/${host_ip_token}/${host_ip}/g" \
    < docker-compose-template.yml \
    > docker-compose.yml

this in docker-compose-template.yml:

  extra_hosts:
   - "{{ base_url }}:{{ host_ip }}"

becomes this in docker-compose.yml:

  extra_hosts:
   - "api.acme.com:192.168.99.100"

[apobbati]

Github needs to implement an upvote/importance feature for issues.

[aanand]

@apobbati +1

(Environment variable interpolation has been implemented and merged in #1765, if that's what you're getting at.)

[funkyfuture]

i admit there's room for misinterpretations, so refresh your knowledge about bolshevik strategies!

[levivm]

+1