How to exclude a certain package from pip-sync?

[Midnighter]

This is a more broad question about how to approach a specific workflow.

My use case is the following: I want to layer multiple Docker images. In the easiest case I have one base image that uses pip-tools to install a number of packages. Subsequent images "inheriting" from that base image then need to specify all compiled requirements files in order to not remove packages with pip-sync. So far so good.

My problem is that the base image should also install a large package from a local source. I can simply specify this dependency with -e path/to/local/package and be done with it but that requires me to keep content of that directory around. Since I try to keep my Docker layers small I would ideally only install the package once and then remove the directory again. However, if I remove the directory pip-sync obviously fails. So is there an easy way for me to exclude that local package from being considered by pip-sync? That would make my life a lot easier.

[atugushev]

Good question! Maybe we need to implement something like --no-uninstall or --safe option?

[Midnighter]

--no-uninstall for this kind of workflow would be stellar indeed!

[AndydeCleyre]

Another thought, @Midnighter :

After you build your frozen .txts and start layering, is there any real advantage to using pip-sync at all, over plain old pip install -r? IMO one of the greatest strengths of this project is that it produces highly compatible, long-standard specification files, and doesn't generally need to be used during deployment operations.

If you have in your .in file:

file:PATH/TO/PKG#egg=PKGNAME==PKGVER

that line will be present in the .txt as well, and pip install -r should not mess with its installation if it detects the req is already satisfied.

[merwok]

A big feature of pip-sync is uninstalling now-unneeded packages.
The presence of a package can change behaviour, if if was installed as a soft dependency of another project that uses package detection to enable optional features, so people wanting truly reproducible environments will use pip-sync.

[AndydeCleyre]

@merwok pip-sync will not guarantee reproducible behavior if the packages are relying on package detection to determine dependencies anyway (#992).

In this context, though I can't see the build scripts, I'd imagine there's a lot of predictability ensured by the container build process.

[merwok]

I was talking about runtime, not install-time 🙂

[AndydeCleyre]

@merwok

Sorry, my mind might be elsewhere, but I don't know what you're getting at.

My understanding is that this issue is about deployment workflow.

[Midnighter]

My understanding is that this issue is about deployment workflow.

Indeed, the context is about layering 2-3 container images and making them fully reproducible/deterministic.

After you build your frozen .txts and start layering, is there any real advantage to using pip-sync at all, over plain old pip install -r? IMO one of the greatest strengths of this project is that it produces highly compatible, long-standard specification files, and doesn't generally need to be used during deployment operations.

This is a fair point and that is actually what I have been doing to get around the issue. It just seems natural to use pip-sync when I use pip-compile to prepare the requirements, hence this issue. If I'm not missing out on anything else that pip-sync is doing then I'm happy to continue to use the following command (I do generate and want to check hashes):

pip install --require-hashes -r requirements.txt