Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔨 Add ssh-client + update git using debian backports in Docker image #3338

Conversation

dmateusp
Copy link
Contributor

@dmateusp dmateusp commented May 11, 2021

resolves #3337

Description

Adds ssh-client and updates the git version (using backports) to enable using dbt deps with the Docker image, on private GitHub repositories, through SSH

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue -- TODO: please help me out, the Docker image changed quite a bit since I last played with it, and I'm not sure how to get it to build locally with the new --build-args
  • This PR includes tests, or tests are not required/relevant for this PR -- note: the tests run on a different Docker image
  • I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

@cla-bot cla-bot bot added the cla:yes label May 11, 2021
@jtcohen6
Copy link
Contributor

TODO: please help me out, the Docker image changed quite a bit since I last played with it, and I'm not sure how to get it to build locally with the new --build-args

This is a bit outside my wheelhouse... @kwigley any chance you could give a pointer?

@kwigley
Copy link
Contributor

kwigley commented May 13, 2021

@dmateusp this seems totally reasonable, I was able to test locally locally with success and the image size didn't seem to change too much. I'd love to have you test this out as well with our wacky build args before merging.

The build args are specifically meant for the build workflow that is set up, we'd have to move some things around in that process to make these build args a little more reasonable. This is something I'd like to do to make this more accessible.

In the meantime, here is a workaround. BASE_REQUIREMENTS_SRC_PATH can point to any file in docker/requirements, these are 3rd party dependencies of dbt. WHEEL_REQUIREMENTS_SRC_PATH is meant to point file listing paths to the dbt packages, but for this it can be the path to a file that contains the dbt requirement (literally a requirements file with just dbt==0.19.1 in it). DIST_PATH would point to where the python distribution files generated during the build process live, so that they can be copied into the image. This is pointless for testing this out, this can point to any directory. This might throw off the image size a little. Putting this together:

docker build -f docker/Dockerfile . --build-arg  BASE_REQUIREMENTS_SRC_PATH=./docker/requirements/requirements.0.19.1.txt --build-arg WHEEL_REQUIREMENTS_SRC_PATH=./docker/requirements/wheels.txt --build-arg DIST_PATH=docker -t dbt-test

where ./docker/requirements/wheels.txt is the new file I created for the dbt requirement (0.19.1 in this case). Let me know how this goes.. this should be easier to do 😅

@dmateusp
Copy link
Contributor Author

@kwigley thank you for sharing the steps! I published dmateusp/dbt:0.20.0b1 and was able to confirm that our internal build can pull dependencies now without needing to install sftp and git anymore

Copy link
Contributor

@kwigley kwigley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dmateusp!

@jtcohen6 jtcohen6 merged commit 4d5d0e2 into dbt-labs:develop May 24, 2021
@dmateusp
Copy link
Contributor Author

Thanks for merging :) @kwigley are you going to push a new version of the Docker image to dockerhub or should I wait for the next dbt version ?

@kwigley
Copy link
Contributor

kwigley commented May 26, 2021

@dmateusp publishing images is so coupled with the automated release process, it may have to wait for the next release. I'll update if this is not the case!

@dmateusp dmateusp deleted the dmateusp/add_ssh_client_update_git_version_docker_image branch May 26, 2021 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Need a more recent version of git, and ssh-client installed in the dbt Docker image
3 participants