Image inheritance

[matthuisman]

#625

I have a base image that I use for PHP (PHP-CLI & PHP-FPM)

Would be cool if you could tell fig to inherit a previous image as a base.
The child image would then have all the attributes of the parent, and then you would be able to override the parents attributes / add attributes.

It is simply generating a different Docker Run command based on the parents command.
No extra building / pulling etc would be needed.

Current Method

ServiceA:
  build: /docker/php
  volumes:
    - /docker/websites:/docker/websites
  links:
    - redis
  command: php service.php
((docker run -v /docker/websites:/docker/websites --links redis:redis ae3454 php service.php))

ServiceB:
  build: /docker/php
  volumes:
    - /docker/websites:/docker/websites
  links:
    - redis
  command: php5-fpm -F
((docker run -v /docker/websites:/docker/websites --links redis:redis ae3454 php5-fpm -F))

Proposed Method

ServiceA:
  build: /docker/php
  volumes:
    - /docker/websites:/docker/websites
  links:
    - redis
  command: php service.php
((docker run -v /docker/websites:/docker/websites --links redis:redis ae3454 php service.php))

ServiceB:
  inherits: ServiceA
  command: php5-fpm -F
((docker run -v /docker/websites:/docker/websites --links redis:redis ae3454 php5-fpm -F))

Behind the scenes

ServiceA = {'container-id' => 'ae3454',
          volumes'=>['/docker/websites:/docker/websites'], 
          links=>['redis:redis'], 'command'=>'php service.php'}

ServiceB = {'command'=>'php5-fpm -F'}

ServiceB = Merge ServiceB ServiceA  (B overrides A)

print Service B:
{'container-id' => 'ae3454',
 volumes'=>['/docker/websites:/docker/websites'], 
 links=>['redis:redis'], 'command'=>'php5-fpm -F'}

*Generate Docker Run Commands*

docker run -v /docker/websites:/docker/websites --links redis:redis ae3454 php service.php
docker run -v /docker/websites:/docker/websites --links redis:redis ae3454 php5-fpm -F

This will help reduce code duplication and make it easy to modify attributes just in the parent image.
Say my PHP images now required another service (MYSQL), I would simply need to add the mysql link to the first image.

At present, I would need to duplicate this change to all the other images which could lead to broken images if one is missed.

What would need deciding is what attributes get inherited, what attributes overwrite the parents and what attributed append to the parents.

Volumes - Append?
Environment variables - Append?
Ports - Don't inherit?

[thaJeztah]

Inspired by "inherits"; perhaps "extends" is another alternative.

[aanand]

I strongly suspect this should be a Docker-level concern, not Fig's realm. See the discussion here: moby/moby#735

[thaJeztah]

I strongly suspect this should be a Docker-level concern

hm, yes, if multiple images should be built using FROM <image-for-other-service>

However, if I understand correctly (I may be on the wrong foot here), the idea here is to re-use the image that was built for a service in another service, i.e.

serviceA:
  build: ./service

serviceB:
  image: <image that was built for service A>

serviceC:
  image: <image that was built for service A>

Basically, this should work already if the correct image-name is referred to (but haven't tested it), but may fail if the services are started in the incorrect order.

The inherit part is not really part of the original proposal/feature request, and (again, if I understand correctly), is meant to copy the settings of another service (as specified in fig.yml), and override only those settings that are different. This request should probably be move to another issue (imo)

[matthuisman]

Cleaned up my comments.
All summed up now in my first post.
Renamed issue.

[aanand]

OK. I can see the value in something like an image_from key, which inherits just the image.

However, I don't think inheriting other configuration is a good idea - shared portable configuration should instead be moved into the Dockerfile. As for non-portable configuration like volumes and ports:

• you can already share volumes with volumes_from
• you never want to share ports config, because they'll clash

[matthuisman]

Tested working! :)

I will attempt to create a patch / pull request.

BTW - Love the build system for fig - using a docker image to build - genius!
I've always wanted to package my own Python apps into an executable, will be copying this method.

Here is the output using my code changes

fig.yml extract

(phpfrpm inherits volumes_from and links from php)

php:
  build: /docker/BUILDS/php
  volumes_from:
    - volumes
  links:
    - redis
    - mysql

phpfpm:
  inherits: php
  command: php5-fpm -F

fig build

Building php...
.......
Step 9 : CMD php --help
 ---> Using cache
 ---> e8c443fb0a8d
Successfully built e8c443fb0a8d
Building phpfpm...
.......
Step 9 : CMD php --help
 ---> Using cache
 ---> e8c443fb0a8d
Successfully built e8c443fb0a8d

(technically we wouldn't need to run the 2nd build command on this occasion as it ends up with same container - but if there were build differences it would be different)

fig up -d

Creating docker_volumes_1...
Creating docker_mysql_1...
Creating docker_redis_1...
Creating docker_php_1...
Creating docker_phpfpm_1...
Creating docker_nginx_1...

fig ps

docker_php_1       php --help                       Exit 0
docker_phpfpm_1    php5-fpm -F                      Up

[matthuisman]

Test2:
ServiceB inherits volumes & environment from serviceA.
ServiceB adds a volume
ServiceC inherits volumes & environment from serviceB (therefore ServiceA)

Maybe 'EXTENDS' is a better name for the option?

fig.yml

serviceA:
  image: ubuntu:14.04
  volumes:
    - /serviceA:/serviceA
  environment:
    PROJECT: services
    SERVICE: serviceA
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo ServiceA $(date) > /serviceA/date; sleep 1; done"

serviceB:
  inherits: serviceA
  volumes:
    - /serviceB:/serviceB
  environment:
    SERVICE: serviceB
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo $(cat /serviceA/date); echo ServiceB $(date) > /serviceB/date; sleep 1;  done"

serviceC:
  inherits: serviceB
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo $(cat /serviceA/date); echo $(cat /serviceB/date); sleep 1;  done"

fig build

serviceA uses an image, skipping
serviceB uses an image, skipping
serviceC uses an image, skipping

fig up -d

Creating docker_serviceA_1...
Creating docker_serviceB_1...
Creating docker_serviceC_1...

fig ps

docker_serviceA_1   /bin/sh -c while true; do  ...   Up
docker_serviceB_1   /bin/sh -c while true; do  ...   Up
docker_serviceC_1   /bin/sh -c while true; do  ...   Up

docker ps

f3e568a0ba53        ubuntu:14.04          "/bin/sh -c 'while t   9 seconds ago       Up 8 seconds  docker_serviceC_1
05d3dcc8c0ab        ubuntu:14.04          "/bin/sh -c 'while t   20 seconds ago      Up 19 seconds  docker_serviceB_1
ca4b5b15b7c9        ubuntu:14.04          "/bin/sh -c 'while t   31 seconds ago      Up 30 seconds  docker_serviceA_1

fig logs serviceC

serviceC_1 | services
serviceC_1 | serviceB
serviceC_1 | ServiceA Thu Nov 6 03:12:26 UTC 2014
serviceC_1 | ServiceB Thu Nov 6 03:12:26 UTC 2014
serviceC_1 | services
serviceC_1 | serviceB
serviceC_1 | ServiceA Thu Nov 6 03:12:27 UTC 2014
serviceC_1 | ServiceB Thu Nov 6 03:12:27 UTC 2014
.....

fig stop

Stopping docker_serviceC_1...
Stopping docker_serviceB_1...
Stopping docker_serviceA_1...

fig rm

Going to remove docker_serviceC_1, docker_serviceB_1, docker_serviceA_1
Are you sure? [yN] y
Removing docker_serviceA_1...
Removing docker_serviceB_1...
Removing docker_serviceC_1..

[dnephin]

You can accomplish this with yaml anchors and aliases:

serviceA: &serviceA                                                                                                                                                                                                                 
  image: ubuntu:14.04                                                                                                                                                                                                               
  volumes:                                                                                                                                                                                                                          
    - /serviceA:/serviceA                                                                                                                                                                                                           
  environment:                                                                                                                                                                                                                      
    PROJECT: services                                                                                                                                                                                                               
    SERVICE: serviceA                                                                                                                                                                                                               
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo ServiceA $(date) > /serviceA/date; sleep 1; done"                                                                                                          

serviceB: &serviceB                                                                                                                                                                                                                 
  << : *serviceA                                                                                                                                                                                                                    
  volumes:                                                                                                                                                                                                                          
    - /serviceB:/serviceB                                                                                                                                                                                                           
  environment:                                                                                                                                                                                                                      
    SERVICE: serviceB                                                                                                                                                                                                               
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo $(cat /serviceA/date); echo ServiceB $(date) > /serviceB/date; sleep 1;  done"                                                                             

serviceC:                                                                                                                                                                                                                           
  << : *serviceB                                                                                                                                                                                                                    
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo $(cat /serviceA/date); echo $(cat /serviceB/date); sleep 1;  done" 

[matthuisman]

I thought the best way to show the code would be via a pull request

https://github.com/docker/fig/pull/625/files

As you can see, it's very simple

[dnephin]

It is relatively short, but I still think this can be accomplished directly from yaml.

[matthuisman]

serviceA: &serviceA                                                                                                                                                                                                                 
  image: ubuntu:14.04                                                                                                                                                                                                               
  volumes:                                                                                                                                                                                                                          
    - /serviceA:/serviceA    
  ports:
    - '80:80'                                                                                                                                                                                                   
  environment:                                                                                                                                                                                                                      
    PROJECT: services                                                                                                                                                                                                               
    SERVICE: serviceA                                                                                                                                                                                                               
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo ServiceA $(date) > /serviceA/date; sleep 1; done"                                                                                                          

serviceB: &serviceB                                                                                                                                                                                                                 
  << : *serviceA                                                                                                                                                                                                                    
  volumes:                                                                                                                                                                                                                          
    - /serviceB:/serviceB                                                                                                                                                                                                           
  environment:                                                                                                                                                                                                                      
    SERVICE: serviceB                                                                                                                                                                                                               
  command: /bin/sh -c "while true; do echo $PROJECT; echo $SERVICE; echo $(cat /serviceA/date); echo ServiceB $(date) > /serviceB/date; sleep 1;  done"  

Would that still work?

[dnephin]

I believe you'd have to:

serviceB: &serviceB                                                                                                                                                                                                                 
  << : *serviceA
  ports: []

While it may seem inconvenient to have to override things with an empty list, at least that way it's explicit. Otherwise the options which are copied or not copies are hidden in fig, and requires a lot more documentation for users to understand.

[matthuisman]

Wouldn't the documentation explaining how to use YAML be more though?

Rather than
'inherit' will inherit all the parents parameters except for foo bar

For users, wouldn't it better to have built-in support rather than needing to know YAML?

The main point of fig is to make setting up docker images easier isn't it?

[matthuisman]

Also, having it built-in to fig gives some useful feedback if you typed something wrong
"cant find parent"

What happens if you do the YAML wrong?
Breaks the whole config?

[dnephin]

I suspect errors in yaml would break the config, yes. I've found the PyYaml error messages to be quite helpful in general.

Let's see how @aanand and @bfirsh feel about it

[matthuisman]

I just think if there's a way to encourage more services more easily, it will stop people being tempted to do:

phpService:
command: service phpfpm start && php myscript.php

[k4nar]

I agree that if it's possible using pure YAML it should not be an option added in fig. But we should add an example in the doc explaining how to do that using YAML.

[zeevallin]

The case where I'd like to use this is when I have a worker and web app running from the same code base. Currently using yaml inheritance I will end up with two images being built which is a waste of space and time.

What if there was a statement to use the image from another service?

web: &app
  build: .
  command: bundle exec unicorn_rails
  environment:
    RAILS_ENV: development
    REDIS_URL: redis://redis/0
    DATABASE_URL: postgres://postgres/dev

worker:
  <<: *app
  use: web
  command: bundle exec sidekiq

[matthuisman]

You need 2x docker "instances" anyway.
You can't have 1x docker image doing 2x different commands unless you have some code in the image to start both commands.

The 2nd image will be the same, so will just use the cache of the first.
So, no extra space used.

All FIG does is read config files and then run the Docker commands for you in the background.

Your generated commands would be something like

docker run ae368852 bundle exec unicorn_rails
docker run ae368852 bundle exec sidekiq

ae368852 = is just a random container ID

So, using the same container, but just creating 2x "instances" of it.
Exactly what you want.

[zeevallin]

@matthuisman You can in fact start start two different containers from on the same image, so I'm not quite sure what you mean.

As well as basing a fig service image on a local Dockerfile or an image from a registry, what I'm suggesting is allowing the direct use of another local service image, cut the overhead.

Instead of this:

CONTAINER ID        IMAGE                       NAMES
8acd16f92923        project_sidekiq:latest      project_sidekiq_1
92c327b5d762        project_web:latest          project_app_1

I want this:

CONTAINER ID        IMAGE                       NAMES
8acd16f92923        project_web:latest          project_sidekiq_1
92c327b5d762        project_web:latest          project_app_1

[dnephin]

I believe this was address with extends ?

[zeevallin]

@dnephin Doesn't it still end up different images only difference being the last commit? I might be wrong.

[dnephin]

The image tags are different, but the layers should all be the same.

If this is just about re-using the same image, I think that's covered by #963