Including Poetry files for Dependency Management

[vedosis]

Added required files, imported existing requirements.txt and requirements.dev.txt dependencies.

https://python-poetry.org/docs/ is a PEP 582 compatible dependency backend. I chose it because pipenv (personal favorite) doesn't seem to be moving forward at the moment, pdm doesn't play well with some datascience libs (pycocotools in particular, scipy in general), and poetry generates pip compatible requirements.txt files for "production."

In theory, we shouldn't need anything extra for the docker environment if we used the poetry output to fix the dependency tree in a point in time. Additionally, it'll be easier to upgrade ALL THE THINGS by using a more diverse versioning syntax (https://python-poetry.org/docs/dependency-specification/)

poetry install to setup a dev environment
poetry install --no-dev to setup a prod environment

To install to root system (Docker)

poetry config settings.virtualenvs.create false
poetry install --no-dev

[derneuere]

Hey @vedosis, thanks for your contribution. This does sound like a better solution than just using a regular requirements file.
When learning about poetry I found a couple of questions regarding LibrePhotos:

At the moment we install pytorch and pyvision in the dockerfile and install script. Would it be possible to add this dependency to poetry?

https://github.com/LibrePhotos/librephotos-docker/blob/7713e6ffecddb5542c7b8cc767a32703887b0b85/backend/Dockerfile#L16

We also build dlib from source, and I am not sure how to exclude them when building the requirement file. Or do we then have to spin up a private repository to add it to the poetry file via the url tag?

https://github.com/LibrePhotos/librephotos-docker/blob/7713e6ffecddb5542c7b8cc767a32703887b0b85/backend/Dockerfile#L19-L26

[vedosis]

What a dizzying little rabbit hole you found for me. Like seriously. People seem to have strong opinions in this area.. Long story short, "let me see if I can." I can use the url option, but there's some issues with cross dependencies. i.e. torch:1.8.1+cpu does not provide the 1.8.1 requirement from torchvision:0.9.1 because reasons? 🤷🏻‍♂️

Let me play with this a little while and get back to you.

[vedosis]

Just bringing this back up here. I've filed a pull request upstream to poetry which fixes the most immediate problem. If they approve it, we can move torch and torchvision into the spec (per my updated pull request).

Still thinking about and working on dlib.

[vedosis]

Follow up question @derneuere, is there a reason we're building from src on dlib? There's a pypi package available https://pypi.org/project/dlib/#files.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[derneuere]

Awesome work!

Dlib has to be compiled AFAIK. The dlib wheel from pip will also compile it, but it does not have an option to compile it explicitly to CPUs that don't have AVX/SSE. The GitHub action build servers all have new CPUs that have these instruction sets, so the wheel will just compile it for AVX.

This leads to an error where, for example, LibrePhotos will just constantly reboot on old xeon processors.

Maybe I missed something in the documentation 😅