Secure way to handle environment variables?

[myhro]

How can I handle environment variables in a secure way, e.g. not explicitly writing them on Dockerfile or fig.yml? This is needed for settings like Django's SECRET_KEY.

I've tried some approaches, without success:

1. Load them from a file that is on .gitignore
2. Load them from the current env which fig is called, using "_env:VARIABLE_NAME[:DEFAULT]", like docker-registry
3. Specify them as an argument of fig up, to be passed to the docker client as -e

I'll try to implement the second one (guess it needs just a little modification here), but if someone is already doing this in another way, I would really like to know.

Regards,
Tiago.

[aanand]

Yeah - no. 2 is basically #76, though your solution can supply defaults too.

I'd really like to be able to do this. As described here, there are complications regarding how it'd interact with another feature relating to environment variables, but I think that feature's less important than this one, so I'm tempted to say let's ignore that and merge it.

[myhro]

Aanand, thanks for pointing these two issues. Looks like the only problem in merging #76 is its potential for breaking #64 in a future implementation of the last one? In the way I've thought about this, it won't be a problem. Instead of evaluating every environment variable on fig.yml, it would do this only for keys that its values starts with _env:. Are you interested in this?

Update: looks like there are some people willing to apply this to keys also, not only its values.

[d11wtq]

I'm not sure what's in the API itself (it would be easy to implement client-side), but Docker's --env-file flag may be of some help here too. The idea of an env file has certainly long been a way of keeping sensitive data out of version control anyway.

[aanand]

Interesting - I didn't know about --env-file. I also didn't know that the -e flag to docker run can accept a lone variable name without a value, telling the client to get the value from the current environment.

This points toward a solution which is both 100% compliant with the Docker approach and wouldn't conflict with #64:

1. Support an 'environment' array where each entry can optionally specify a value.

web:
  build: .
  environment:
    - DEBUG=true
    - SECRET_API_TOKEN

Fig will pull the value for SECRET_API_TOKEN from the current environment.

2. As a future enhancement, allow environment variable expansion.

web:
  build: .
  environment:
    - DEBUG=true
    - SECRET_API_TOKEN
    - THINGS_URL=https://example.com/api/v1/things.json?apiToken=$SECRET_API_TOKEN

Thoughts?

[myhro]

Interesting - I didn't know about --env-file.

I didn't knew about that either, but anyone knows in what version it was introduced? Docker 0.9.1 (from Ubuntu 14.04 repos) doesn't have this option.

Update: it was added on version 0.10.0.

I also didn't know that the -e flag to docker run can accept a lone variable name without a value, telling the client to get the value from the current environment.

This is interesting. I've only found out about it when reading the cli reference, as this isn't mentioned on the documentation about the run command.

This points toward a solution which is both 100% compliant with the Docker approach and wouldn't conflict with #64:

The way #64 could be implemented was already figured out? The easiest workaround I've found to use variables generated by links in this way is running a shell script that reads them and set new variables, before executing the target command.

Thoughts?

This solution looks very nice for now (until #64 is resolved).

[d11wtq]

The --env-file flag has been in there for ages. Just checked the changelog and it was added in 0.10.0. If you're still using 0.9.1 and you're actively using docker you really should update ;) Docker is kind of a bleeding-edge-only technology at this point, so you're probably safest following the releases and staying up-to-date. The next release of fig will only support docker >= 1.0, because the API itself in docker has changed.

I also didn't know about the -e flag behaviour wihtout a value. That is amazeballs!

[myhro]

If you're still using 0.9.1 and you're actively using docker you really should update ;) Docker is kind of a bleeding-edge-only technology at this point, so you're probably safest following the releases and staying up-to-date. The next release of fig will only support docker >= 1.0, because the API itself in docker has changed.

I guess you're right, Chris, as Docker >= 1.0.0 is now tagged for production, the API is stable, etc. I don't like to handle packages without a manager like apt-get/pip, but Docker makes its update process very easy, as I just have do download the newer binary and point to it at /etc/default/docker.io, so the daemon could be handled by Ubuntu's upstart. Thanks for the tip.

[d11wtq]

Oh, I just assumed Ubuntu would be up-to-date :) Yeah, 1.0 shouldn't be making any breaking changes now, though I do imagine new features will continue to be added (with appropriate version changes).

I use Gentoo which tends to be about as up to date as you can get (docker seems to be updated the day after it is released).

[d11wtq]

Fig should support:

environment:
  - FOO=bar
  - EXAMPLE

Where $EXAMPLE is used from the host. Then you can basically "compose" the other environment variables in the container without magic syntax for interpolation.

[stefanfoulis]

Are there any ideas on how to support host variables on osx?

[d11wtq]

Doesn't it work? I haven't tested it, but it seems like it would be a client-side feature.

[stefanfoulis]

Sorry for not being clear. On OSX docker is running in a VM. So the "host variables" would be the ones in the VM and not the ones from the current user on OSX.

[d11wtq]

I thought only the daemon was on the VM, but the client was still on OS X, just using $DOCKER_HOST. I will have to experiment though, as I don’t run docker on OS X itself (I use a VM).

On 7 Jul 2014, at 9:18 am, Stefan Foulis notifications@github.com wrote:

Sorry for not being clear. On OSX docker is running in a VM. So the "host variables" would be the ones in the VM and not the ones from the current user on OSX.

—
Reply to this email directly or view it on GitHub.

[stefanfoulis]

Ah, you are right. It's the docker client issuing the commands and that is actually running on OSX. So it might not be an issue at all. Sorry for the noise.

[ryanbrainard]

I tested this out, but Fig (or docker-py rather) does not work out of the box to support key-only environment variables like the Docker CLI. I just sent a pull request to docker-py to fix this:

docker/docker-py#263

--env-file support should probably also be added into docker-py and Fig, but until it does, you could use something like Foreman or Forego to load an .env file to provide the environment to Fig.

[d11wtq]

Actually, thinking about it, I'm not sure it should be an implicit measure taken by the client and should be something that fig would do. The reason for that is that the client can't really assume it's running on the command line (or even on the same machine as the end-user). It's safer that it require explicit inputs and outputs. So in that case, fig would need to recognize env vars without an assignment, then look them up from the environment itself and replace them accordingly.

Let's see what the docker-py guys do anyway :)

[d11wtq]

@ryanbrainard I'm pretty sure --env-file is just a convenience implemented on the official docker client and not something that goes in docker-py. You can implement --env-file given only a mechanism for setting individual environment variables. Basically the docker-py client is not supposed to be a drop-in-replacement for the official docker client, but is supposed to be an interface to the REST API.

[ryanbrainard]

@d11wtq I agree that the --env-file support probably belongs in Fig rather than docker-py. As far as the key-only env resolution, I can see it either way, but I do see your point. As you said, let's see what the docker-py guys say. If they don't want it in the library, I'm happy to add it to Fig directly.

[ryanbrainard]

@d11wtq I realized a problem in the way docker-py was flattening the env, which caused problems with my change, so closed my pull request there. Thinking about it more, you are right that this really belongs in Fig (not docker-py), so I opened a new pull request here: #297