Validate Compose file and produce better error messages

[aanand]

There are a lot of ways to confuse Compose with a badly-formed docker-compose.yml. This file should be checked when it is read and good error messages should be produced to help users fix the problem.

This could be done by producing a schema for the file, like was done in #1348.

These sorts of things should be checked:

• Top-level object is a dict
• Each key (a service name) maps to a dict (Improve error when service is not a dict #128)
• String config options (e.g. image) - are numbers allowed? When? Don't accept YAML's eager interpretation of values (e.g. true) which should be quoted (docker-compose does not pass environment variables true/false unchanged #1788)
• List config options (e.g. volumes) either must be a list, or we helpfully wrap single values in a list for you
• String-or-list config options (e.g. command, entrypoint)
• Dictionary config options (e.g. environment)
• Format of various options (e.g. volumes - see Error when specifying permission in volume #260)

References: #117 #127

[wiliamsouza]

fig.yml

mysql:
  image: wiliamsouza/mysql
  ports: "3306:3306"
  volumes:
    - volumes/log:/var/log/mysql
    - volumes/lib:/var/lib/mysql
    - volumes/conf.d:/etc/mysql/conf.d
  environment:
    - MYSQL_ROOT_PASSWORD: rootpass
    - MYSQL_DATABASE: mydb
    - MYSQL_USER: myuser
    - MYSQL_PASSWORD: mypass

The port should be in the format:

  ports: 
   - "3306:3306"

fig up mysql
Recreating dockermysql_mysql_1...
Traceback (most recent call last):
  File "/home/wiliam/.local/bin/fig", line 9, in <module>
    load_entry_point('fig==0.3.2', 'console_scripts', 'fig')()
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/main.py", line 39, in main
    command.sys_dispatch()
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/docopt_command.py", line 21, in sys_dispatch
    self.dispatch(sys.argv[1:], None)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/command.py", line 30, in dispatch
    super(Command, self).dispatch(*args, **kwargs)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/docopt_command.py", line 24, in dispatch
    self.perform_command(*self.parse(argv, global_options))
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/command.py", line 47, in perform_command
    return super(Command, self).perform_command(options, *args, **kwargs)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/docopt_command.py", line 27, in perform_command
    handler(command_options)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/cli/main.py", line 301, in up
    (old, new) = self.project.recreate_containers(service_names=options['SERVICE'])
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/project.py", line 115, in recreate_containers
    (s_old, s_new) = service.recreate_containers()
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/service.py", line 155, in recreate_containers
    (old_container, new_container) = self.recreate_container(c, **override_options)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/service.py", line 178, in recreate_container
    new_container = self.create_container(**options)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/service.py", line 129, in create_container
    container_options = self._get_container_options(override_options, one_off=one_off)
  File "/home/wiliam/.local/lib/python2.7/site-packages/fig/service.py", line 251, in _get_container_options
    all_ports = container_options.get('ports', []) + self.options.get('expose', [])
TypeError: cannot concatenate 'str' and 'list' objects

[jneen]

Have we considered designing a custom syntax that enforces this stuff at a syntactic level? Perhaps something like

web {
  build .
  command python app.py
  link db
  port 8000:8000
}

db {
  image postgres
}

[jneen]

I'm happy to put together a proposed implementation if yall think this is a direction we'd like to go.

[aanand]

Appealing - but personally, I'd much rather maintain a few lines of validation code than a custom language grammar + parser, as flawed as YAML is.

On Sat, Aug 9, 2014 at 11:26 AM, Jeanine Adkisson
notifications@github.com wrote:

I'm happy to put together a proposed implementation if yall think this is a direction we'd like to go.

Reply to this email directly or view it on GitHub:
#129 (comment)

[jneen]

Ok. My feeling is that validation is not going to be anywhere near complete with just a few lines, and parsing is not actually that hard, especially for such a dirt-simple grammar. But it's your call.

[jneen]

Oh, and it'd probably just be a small amount of pyparsing, not a whole flex/bison stack or something, that would indeed be way overkill.

[jneen]

... and in terms of being a tool for humans, we would do well to make the format obvious by design, which yaml doesn't really accomplish. But yes, I will actually stop talking about this now.

[d11wtq]

👎 on introducing a new grammar. It would break backwards compatibility and YAML is a pretty universal format. We actually do things with our fig.yml in our deployment tooling for example. I don't want to have to build a parser into our deployment tooling. YAML is flexible enough that we don't have to update the grammar as we introduce new features. It just requires that we check the data types for the keys we want to use.

[nubs]

As a user of fig, I also would prefer the YAML. In my experience there is very rarely need for something beyond a simple structured configuration using a standard format (JSON, YAML, XML (ugh!), etc.).

[dnephin]

http://colander.readthedocs.org/en/latest/ might be a decent option for validating the config. It validates a schema on the python side so it doesn't tie you to yaml. I've used it before without many complaints.

[montells]

How do we specify :ro to the fig.yml for volumes

[mpe]

Spot the bug:

 volumes:
      - var:postgres/:/var/lib/postgresql/data

Result is:

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.main", line 30, in main
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.docopt_command", line 21, in sys_dispatch
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.command", line 31, in dispatch
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.docopt_command", line 24, in dispatch
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.command", line 50, in perform_command
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.docopt_command", line 27, in perform_command
  File "/code/build/fig/out00-PYZ.pyz/fig.cli.main", line 297, in start
  File "/code/build/fig/out00-PYZ.pyz/fig.project", line 148, in start
  File "/code/build/fig/out00-PYZ.pyz/fig.service", line 80, in start
  File "/code/build/fig/out00-PYZ.pyz/fig.service", line 212, in start_container_if_stopped
  File "/code/build/fig/out00-PYZ.pyz/fig.service", line 233, in start_container
ValueError: too many values to unpack

[dnephin]

#442 would "fix" this (kind of). I guess it should add a check to make sure that the 3rd part is either ro or rw, and raise ConfigError if it's anything else.

[draghuram]

@aanand, Is there any consensus on the best way to implement the validation? I wanted to check first before I give it a shot.

[dnephin]

Not at all consensus, but I have been thinking that jsonschema would be a good option. It's language agnostic (so other tools interested in supporting the compose format can do so), and it should support validation for all the cases here.

[funkyfuture]

@draghuram what's your status on this? maybe i will have the time to take care of this within the next days.

[draghuram]

@funkyfuture As you can see, I created a PR with all the work that I have done. Basically, there is a JSON schema file and code to validate the configuration against that. At some point, I was able to run all the tests successfully but that was some time back. I think there are two more things that need to be done:

• Write tests covering invalid configurations
• Make sure that Schema covers everything that we need.

Please feel free to build on top of this PR as I may not be able to spend time on this issue.

[kumavis]

+1 to keeping yaml
+1 to validating the schema

[mnowster]

I majority of these concerns should now be covered/addressed by #1808

[KamilKopaczyk]

Could it be more strict? Perhaps allowed values for keys such as restart?

Lately i was investigating why one of my containers isn't restarting, when others do. I looked at my yaml file and everything was ok. Except for a typo i didn't see ;)

restart: unless-stopeed

You can type whatever you want as a value and nothing stops you from running containers this way.

Using docker-compose version 1.6.2

[dnephin]

Accepted values for restart should be validated by the server. If we were to validate them on the client side then we'd have to constantly be updating the list as the server changed. I think this bug should be reported at docker/docker. It should fail with an error if the restart value is not correct, I guess currently it's just ignoring the value.

[jamshid]

Doesn't sound like I'll be able to put SERVICE_IS_PRIVILEGED=true in an .env file and build a docker-compose.yml like:

  myservice:
    privileged: ${SERVICE_IS_PRIVILEGED}
    ...

Maybe this env variable limitation could be mentioned in:
https://docs.docker.com/compose/environment-variables/

[lmurali89]

I continue to get this error message on windows.

ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

docker-compose.yaml/yml file is in the directory mentioned. docker and docker compose versions are shown below. someone please help me at the earliest

docker --version
Docker version 19.03.5, build 633a0ea

docker-compose --version
docker-compose version 1.25.4, build 8d51620