Skip to content

Improve file-service Docker Compose File #3816

@AnzhiZhang

Description

@AnzhiZhang

Proposed Changes

It might be good to improve the Docker Compose file for LakeFS in our file-service. To be specific:

  1. Remove version. In the current version of Docker Compose, the version is obsolete. We are not including this in the single node installation compose file. See https://docs.docker.com/reference/compose-file/version-and-name/
  2. Update container_name. Currently, we have minio, postgres, and lakefs as container names. However, this name is globally on a machine. We should give a namespace to avoid collisions and inconvenience. Similarly to what we did in the single node configuration, I propose texera-lakefs-minio and the others.
  3. Give default volumes. Currently, minio and postgres data are persisted using auto-created Docker volumes, which will be a generated hash name. We have a comment in the Docker Compose file to help developers assign a physical location to persist them. We can set default volumes like minio_data to better persist data and prevent loss.
Image Image

Changes for Developers to Apply

After this change, developers need to:

  • Clone those auto-generated volumes to the new name.
  • Recreate containers.

Here is a step-by-step guide:

  1. Go to core/file-service/src/main/resources.
  2. Stop and remove containers.
docker compose down
  1. Merge main to your local branch.
  2. Start containers.
docker compose up -d
  1. Stop containers.
docker compose stop
  1. Copy volumes. Replace two volume names only. New volume names are texera-lakefs_postgres_data and texera-lakefs_minio_data. The old volume name is generated volume name, it looks like 68bbd428f6a7331a964df929b1effa0837a14f451a5eeb18e0f96d6736471ed8.
docker run --rm \
  --volume ${OLD_VOLUME_NAME}:/source \
  --volume ${NEW_VOLUME_NAME}:/destination \
  ubuntu:latest \
  bash -c "echo 'copying volume ...'; cp -Rp /source/* /destination/"

Command from: https://stackoverflow.com/a/74953859/19420247

Note: If you are using Docker Desktop, you can use the "Clone" button in the volumes tab to simply complete this step.

Image
  1. Start containers.
docker compose up -d

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions