Reorganize setup files & update Docker configurations

[arikfr]

This is a WIP, but due to many people starting to look into the Docker support, I thought I should share it already.

Some changes here from master:

• First of all I've reorganized the setup/ folder with files for each kind of deployment (Amazon Linux, Ubuntu, Docker) in their own directory.
• Docker: update supervisord config to output logs to stdout, so we can easily access/collect them in a Docker environment.
• Docker: update supervisord not to use sudo (you can specify a user in the config) and not to use bin/run (bin/run is used to source the .env file, which isn't necessary in Docker environment).
• In default Docker configuration, use the default postgres database and postgres user instead of creating special user for re:dash.
• Updated the DB bootstrap script to utilize docker-compose and the different containers.
• Configure CircleCI to build Docker images and push them to Docker Hub.

TODO:

• Setup password for the postgres user.
• Fix nginx volume mounting.
• Finish bootstrap.sh script for the ubuntu-docker image.
• Volume for supervisord config.

Comments are welcome.

Closes #450.

[arthurpsmith]

On the docker database setup process - how are you managing the postgres user accounts & passwords? From the postgres docker image documentation you can set an environment variable (as follows in the docker-compose.yml file:)
environment:
POSTGRES_PASSWORD: xxxxxx
that postgres will use to set up the initial administrative account; then if you pass that variable to the redash container it has access. Not sure how else you can handle this? Since these are running in separate containers I don't believe you can just use the username for access.

[arthurpsmith]

Hmm, I see you have a create_database.sh script - I guess you are doing some things directly on the postgres instance (with psql) rather than installing psql on the redash client? I guess that should work. But you still need to connect from the redash client via the python tools - I am getting this to work by setting the DATABSE_URL to REDASH_DATABASE_URL=postgresql://postgres:xxxxxx@postgres/redash (where the xxxx is the POSTGRES_PASSWORD used in setting up the postgres container). However then you need to set up the 'redash' database in the postgres instance also before anything else.

[arthurpsmith]

OK, I've been trying to get this to work but it doesn't seem possible. If I try to run with a fresh postgres container (nothing in the database):

docker-compose run postgres su postgres -c 'createdb redash --owner=postgres'

then it can't connect to the database because the container it's bringing up isn't running the database, it's running the createdb command. I think all the database and user initialization needs to be done from the redash container, it can't be done (easily) on the postgres container. If we don't want to install the psql client then it needs to happen from python - I would guess via 'manage.py'. For example the following seems to work:

docker-compose run redash ./manage.py database create_tables

so I would propose adding commands for ./manage.py database create (to be run before create_tables) and ./manage.py users create --reader (to create the redash_reader account) - I can take a look at this if you like.

[arikfr]

@arthurpsmith

1. About postgres password: it does work without it. But I've realized that it's better to set a password anyway, so I'll add that.
2. The connection string I've used is: postgresql://postgres@postgres/postgres. Once I set a password it will become: postgresql://postgres:XXXX@postgres/postgres (note that I used the default postgres database).
3. Re. the issue you're having with creating db: I first boot the postgres container (with docker-compose up redash, but can be done with just docker-compose up postgres), and then run the create database script. The containers the script uses are temporary containers, that connect to the one that actually running the database.

[arthurpsmith]

Oh - you're using the default 'postgres' database instead of 'redash'?! That does work then, ok! For reference here is what works for me now with the current redash docker image:
(1) Use the below docker-compose.yml (with modified postgres password and cooke secret values)
(2) Run 'docker-compose up postgres'
(3) run your create_database.sh
(4) then 'docker-compose up'
and go to port 8080 on the docker host to see the app in action!

redis:
  image: redis:2.8

postgres:
  image: postgres:9.3
  environment:
    POSTGRES_PASSWORD: xxxxxxx
    PGPASSWORD: xxxxxxx
  volumes:
   - /opt/postgres-data:/var/lib/postgresql/data

redash:
  image: everythingme/redash
  links:
   - redis
   - postgres 
  environment:
    REDASH_REDIS_URL: "redis://redis:6379/1"
    REDASH_DATABASE_URL: "postgresql://postgres:xxxxxxx@postgres/postgres"
    REDASH_STATIC_ASSETS_PATH: "../rd_ui/dist/"
    REDASH_LOG_LEVEL: "INFO"
    REDASH_WORKERS_COUNT: "6"
    REDASH_COOKIE_SECRET: "yyyyzzzzz"
  ports:
   - "8080:5000"

[astewart-twist]

I still think the docker-compose file should not assume anything about the host environment/filesystem, particularly for the volumes under the postgres section. What is /opt/postgres-data on your host system? Is it empty or populated, and what permissions does the path have?

After attempting to create that path on my system (Mac, btw), I get the following:

docker-compose up postgres

...

postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... initdb: could not create directory "/var/lib/postgresql/data/pg_xlog/archive_status": Permission denied
postgres_1 | initdb: removing contents of data directory "/var/lib/postgresql/data"
redash_postgres_1 exited with code 1
Gracefully stopping... (press Ctrl+C again to force)

[arthurpsmith]

I start with no /opt/postgres-data directory on the docker host. Docker
should create it with the right permissions for postgres to use it
(owned by user 999). The purpose of it is to have a location for a
persistent database; if you'd prefer somewhere else feel free to set it
to a suitable location. I'm running this on a mac with docker-machine
0.4.1, docker-compose 1.4.2.

Arthur

On 10/21/15 4:31 PM, astewart-twist wrote:

I still think the docker-compose file should not assume anything about
the host environment/filesystem, particularly for the volumes under
the postgres section. What is /opt/postgres-data on your host system?
Is it empty or populated, and what permissions does the path have?

After attempting to create that path on my system (Mac, btw), I get
the following:

|docker-compose up postgres ... postgres_1 | fixing permissions on
existing directory /var/lib/postgresql/data ... ok postgres_1 |
creating subdirectories ... initdb: could not create directory
"/var/lib/postgresql/data/pg_xlog/archive_status": Permission denied
postgres_1 | initdb: removing contents of data directory
"/var/lib/postgresql/data" redash_postgres_1 exited with code 1
Gracefully stopping... (press Ctrl+C again to force) |

—
Reply to this email directly or view it on GitHub
#620 (comment).

[astewart-twist]

Oh, no, I'm sure that's a fine location. I just assumed it had to exist before it could be mounted by docker. I bet that's why I got that error.

I re-ran after removing that directory and it deploys up fine now. Very cool!

I wonder if the whole thing could be better integrated into the 'docker-compose up' workflow. The order of operations sorta requires the sequential flow which docker-compose doesn't seem to really handle in a straight-forward manner. Probably the easiest thing to do would be to prepend the redash entrypoint to check for existence of the redash db in postgress and run create_database.sh if not.

[arikfr]

@astewart-twist all the docker-compose files we will provide in the repo will be for reference only, so it seems ok to have some assumptions there, and each one can adapt them to his own environment.

As for flow -- I will check again when writing the documentation, but iirc, you can do docker-compose up and then run the create database script. re:dash can start without a database. I wouldn't want to add the create database script inside the container, mainly because it might be used in an environment where it's not relevant.

[arikfr]

And I really appreciate the discussion here, as it helps me understand the gaps I need to fill when preparing this and the documentation.

[arthurpsmith]

FYI I thought we could get away without the nginx front-end, but it turned out the python flask server didn't behave well with Amazon elastic load balancers. I was able to use the standard nginx:1 container image with only a slight modification of your nginx.conf file (change the access_log and error_log entries so they don't require subdirectories of /var/log) and the following setting in docker-compose.yml:

nginx:
  image: nginx:1
  links:
   - redash
  ports:
   - "80:80"
  volumes:
   - /opt/redash-nginx.conf:/etc/nginx/nginx.conf:ro

note that I also had to copy the nginx.conf file to /opt/redash-nginx.conf on the docker host to make this work. Depending on what your filesystem is on your docker host you might want to do something different (as with the persistent postgres db files). Or maybe there should be a custom redash-nginx docker image that includes the conf file so this isn't necessary.