docker-compose fails setup

[maartenbreddels]

Hi,

I follow the quickstart, by downloading the docker-compose.yml, running docker-compose up, then I go to http://127.0.0.1:8000/setup, enter a username 'maarten' and get:

lakefs_1    |      ██╗      █████╗ ██╗  ██╗███████╗███████╗███████╗
lakefs_1    |      ██║     ██╔══██╗██║ ██╔╝██╔════╝██╔════╝██╔════╝
lakefs_1    |      ██║     ███████║█████╔╝ █████╗  █████╗  ███████╗
lakefs_1    |      ██║     ██╔══██║██╔═██╗ ██╔══╝  ██╔══╝  ╚════██║
lakefs_1    |      ███████╗██║  ██║██║  ██╗███████╗██║     ███████║
lakefs_1    |      ╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝╚═╝     ╚══════╝
lakefs_1    |
lakefs_1    | │
lakefs_1    | │ If you're running lakeFS locally for the first time,
lakefs_1    | │     complete the setup process at http://127.0.0.1:8000/setup
lakefs_1    | │
lakefs_1    |
lakefs_1    | │
lakefs_1    | │ For more information on how to use lakeFS,
lakefs_1    | │     check out the docs at https://docs.lakefs.io/quickstart/repository
lakefs_1    | │
lakefs_1    |
lakefs_1    | time="2020-11-12T13:41:32Z" level=info msg="schema migrated" func="db.(*DatabaseMigrator).Migrate" file="build/db/migration.go:49" direction=up host="127.0.0.1:8000" method=POST path=/api/v1/setup_lakefs request_id=e4eacba2-ea5a-491b-a77b-28821f2e72df service_name=rest_api took=3.740318ms
lakefs_1    | time="2020-11-12T13:41:32Z" level=error msg="SQL query failed with error" func="db.(*dbTx).Get" file="build/db/tx.go:83" args="[Admins 2020-11-12 13:41:32.291128212 +0000 UTC m=+8.923394987]" error="scany: rows final error: ERROR: duplicate key value violates unique constraint \"auth_groups_unique_display_name\" (SQLSTATE 23505)" query="INSERT INTO auth_groups (display_name, created_at) VALUES ($1, $2) RETURNING id" took="4.118µs" type=get

Am I doing something wrong, or is the docker image broken?

cheers,

Maarten

[ozkatz]

Thanks @maartenbreddels -
I'm trying to reproduce this without much success. Given the error shown in the log you've attacned, it seems that the database has already been initialized previously - could it be that the postgres container is reused from a previous lakeFS installation?

Could attach the output of docker ps -a?

[maartenbreddels]

That's what I was thinking as well, also, changing the username for instance didn't matter.

$ docker ps -a                                                                                                                                                                                             dev
CONTAINER ID        IMAGE                                                                 COMMAND                   CREATED             STATUS                       PORTS                                                                                                NAMES
eed0d8bb6171        treeverse/lakefs:latest                                               "/app/wait-for postg…"    28 seconds ago      Exited (0) 11 seconds ago                                                                                                         lakefs-example_lakefs_1
884f248042f0        postgres:11                                                           "docker-entrypoint.s…"    28 seconds ago      Exited (0) 10 seconds ago                                                                                                         lakefs-example_postgres_1

[ozkatz]

That's weird indeed. For some reason I can't seem to reproduce.
Can you attach the output of docker image inspect treeverse/lakefs:latest and also the docker-compose.yml you're using?

[maartenbreddels]

[
    {
        "Id": "sha256:17c6872cd6c08693ff02f925c23dc5168d61381c967c035744cf6bc6c8f34988",
        "RepoTags": [
            "treeverse/lakefs:0.16.0",
            "treeverse/lakefs:latest"
        ],
        "RepoDigests": [
            "treeverse/lakefs@sha256:48c9975c4702a0534580eb4f48c1296cff8acc553009e506a2addc9adf6a2880"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2020-11-12T11:33:16.096172922Z",
        "Container": "48dd32fed5f430c6de6fa2abe9aec64bf4760766749657229767fbdb2c9b0568",
        "ContainerConfig": {
            "Hostname": "48dd32fed5f4",
            "Domainname": "",
            "User": "lakefs",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/app:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "CMD [\"run\"]"
            ],
            "Image": "sha256:95683398f8d146c28fea62d47ef5ad51bdefe57d4da2e2ac5de8eadaafee533b",
            "Volumes": {
                "/etc/lakefs.yaml": {}
            },
            "WorkingDir": "/home/lakefs",
            "Entrypoint": [
                "/app/lakefs"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "19.03.13+azure",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "lakefs",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/app:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "run"
            ],
            "Image": "sha256:95683398f8d146c28fea62d47ef5ad51bdefe57d4da2e2ac5de8eadaafee533b",
            "Volumes": {
                "/etc/lakefs.yaml": {}
            },
            "WorkingDir": "/home/lakefs",
            "Entrypoint": [
                "/app/lakefs"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 100187796,
        "VirtualSize": 100187796,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/mnt/vaex/docker/overlay2/9149a9e86cb75cfe1b05ad9095d1e3d0ad6ea0b16e512a46415485a564a27445/diff:/mnt/vaex/docker/overlay2/489a02420de6cd61f0951de767029b8a724977cbd9113d96317c5462aa9b9ccd/diff:/mnt/vaex/docker/overlay2/a85e18b228300e6748b39b872953d09e819b5e48b7df591d9d47577f38ae0ca3/diff:/mnt/vaex/docker/overlay2/dfc401d33d695ee972838fde4beeacbd2d7aff2f02ff56b14bb5c389e753d365/diff:/mnt/vaex/docker/overlay2/2247b30404b14ddfe374be18f24810082d60034c15f31bc3bbc9a5463e42fe9d/diff",
                "MergedDir": "/mnt/vaex/docker/overlay2/97f89c18a5b5dd30270056580162eda91e8cec93ca4debe996fb4e38922737eb/merged",
                "UpperDir": "/mnt/vaex/docker/overlay2/97f89c18a5b5dd30270056580162eda91e8cec93ca4debe996fb4e38922737eb/diff",
                "WorkDir": "/mnt/vaex/docker/overlay2/97f89c18a5b5dd30270056580162eda91e8cec93ca4debe996fb4e38922737eb/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:50644c29ef5a27c9a40c393a73ece2479de78325cae7d762ef3cdc19bf42dd0a",
                "sha256:1e7b0cc662fe2c249bc5374b15844fc68c4b60f3db40c75b074c5c46f5908694",
                "sha256:086870432ef5891570ad7d07b12373021f17910a7449342ce3b17a50d2450f07",
                "sha256:17eedd6cb70a1e4e1f9c94eb5d7f7786f44cc82b503e47370500277483e047a5",
                "sha256:dbcab38404a3b6e8f75e52c8495a88734c78d5d252b4ae06962c853d2a33f2bc",
                "sha256:379c691a197cb900ad9d7c6f52081076e44853e4234156f655f768b744cddb44"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

version: '3'
services:
  lakefs:
    image: "treeverse/lakefs:${VERSION:-latest}"
    ports:
      - "8000:8000"
    depends_on:
      - "postgres"
    environment:
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=${LAKEFS_AUTH_ENCRYPT_SECRET_KEY:-some random secret string}
      - LAKEFS_DATABASE_CONNECTION_STRING=${LAKEFS_DATABASE_CONNECTION_STRING:-postgres://lakefs:lakefs@postgres/postgres?sslmode=disable}
      - LAKEFS_BLOCKSTORE_TYPE=${LAKEFS_BLOCKSTORE_TYPE:-local}
      - LAKEFS_BLOCKSTORE_LOCAL_PATH=${LAKEFS_BLOCKSTORE_LOCAL_PATH:-/home/lakefs}
      - LAKEFS_GATEWAYS_S3_DOMAIN_NAME=${LAKEFS_GATEWAYS_S3_DOMAIN_NAME:-s3.local.lakefs.io:8000}
      - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-}
      - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_SECRET_KEY=${AWS_SECRET_ACCESS_KEY:-}
      - LAKEFS_LOGGING_LEVEL=${LAKEFS_LOGGING_LEVEL:-INFO}
      - LAKEFS_STATS_ENABLED
    entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
  postgres:
    image: "postgres:${PG_VERSION:-11}"
    command: "-c log_min_messages=FATAL"
    environment:
      POSTGRES_USER: lakefs
      POSTGRES_PASSWORD: lakefs
    logging:
      driver: none

[ozkatz]

I'm using an identical setup and cannot reproduce :(
The only thing I can think of is that maybe something is causing the setup form to submit twice, since this submission triggers the creation of the db tables and the subsequent creation of default policies and groups that are failing.

What browser are you using?
can you try reproducing in incognito mode or after disabling extensions?

[maartenbreddels]

Also tried it in firefox, same result.
Maybe also worth mentioning, I see this basic auth popup:

That doesn't seem right either right?

[ozkatz]

just to make sure - are you deleting the previous containers between attempts? (i.e. docker rm to the postgres container as visible in docker ps -a)
The basic auth dialog appearing on /setup is indeed a bug - opened a seperate issue for that - #916

[maartenbreddels]

Got it! the docker rm did the trick. Do you know what was happening? I've never experience something like this with docker. Also, docker-compose up --force-recreate didn't help here.

[ozkatz]

Happy it works for you now! Haven't seen such a behavior before.
Also weird that --force-recreate didn't solve it.

Haven't given up quite yet on trying to recreate it on my environment - any other information you think is relevant regarding the initial setup that failed or about your environment that might help?

[maartenbreddels]

I think, in the beginning, the setup page might have been opened twice (I clicked the link in the terminal, and I manually went to localhost:8000 or something). Thanks for you help!

[mariobuikhuizen]

--force-recreate doesn't recreate volumes. Maybe --renew-anon-volumes would help here.

[ozkatz]

Ah! thanks @mariobuikhuizen - that explains it