Containerize

[gzt5142]

Figure out how to run the cli from within a docker container.

[gzt5142]

Success in running the crawler in a container.

The dockerfile is based on a CHS-provided base, which introduced some challenges (version of python, version of some python libs, etc). It is also about 900Mb, which feels high.

Aim to refactor using a python/alpine base image to see if it is easier to work with and perhaps has a smaller footprint.

[gzt5142]

Also -- need to find a better way to pass secrets to the container.
My test bench used docker-compose, with password set as environment variable. This is not a long-term solution, as the password lives in a text file within the repo. For the time being, I am hitting a toy database on my desktop, so the risk associated with this password is trivial. When we scale to the bigger, centralized system, risk will require a different mechanism.

[gzt5142]

Getting help from @ewojtylko

[gzt5142]

Possible images to try from the 'official' docker registry:
https://hub.docker.com/_/python

• python 3.10 slim
• python 3.10 alpine

[dblodgett-usgs]

Did you progress the secrets issue with @ewojtylko -- seems like something that is set up in our existing containers that you could copy?

[ewojtylko]

The secrets can be loaded from a .env that is .gitignore'd. Works with the docker-compose just fine and if that doesn't fit our needs, then secrets can be fed in from mirrored CI Pipeline I think.

[gzt5142]

proof-of-concept is merged in... this issue still open until we get a container working in the cloud.

[ewojtylko]

Okay, fantastic. Opening another branch for pipeline development. Won't push to remote until it has an issue to go with. Let me know if you are cool with me creating issues for the board.

[ewojtylko]

Mirrored repo is up. Pretty crude, I think, and took way longer than expected (figuring out the SSL issue was... interesting) but its available as a personal project on the CHS gitlab where gitlab ci scripts can be used. Next up is getting the image on ECR. Tagging @dblodgett-usgs so he knows that I am running a bit behind.