Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerfile: retain the pip download cache between builds #6035

Merged
merged 2 commits into from
Apr 18, 2023

Conversation

SpecLad
Copy link
Contributor

@SpecLad SpecLad commented Apr 17, 2023

Motivation and context

This speeds up the build when the entire step can't be cached (e.g. the requirements file changed), but the package list remains mostly the same.

The savings are... rather underwhelming, actually. I have observed about a minute in savings, although it obviously depends on the network connection speed. I think this is because pip is inefficient at loading from its own cache (I have observed it loading the entire cached file into memory, for example).

Still, savings are savings, and we're getting them basically for free, so why not.

Note that I only persist the HTTP cache, and not the wheel cache. That's because any wheels that pip builds could depend on the system packages, and I don't want old wheels to be reused if the system packages change.

Also, disable the pip autoupdate checks, which isn't much of an optimization, but it gets rid of some pointless warnings.

How has this been tested?

docker build

Checklist

  • I submit my changes into the develop branch
  • [ ] I have added a description of my changes into the CHANGELOG file
  • [ ] I have updated the documentation accordingly
  • [ ] I have added tests to cover my changes
  • [ ] I have linked related issues (see GitHub docs)
  • [ ] I have increased versions of npm packages if it is necessary
    (cvat-canvas,
    cvat-core,
    cvat-data and
    cvat-ui)

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.

SpecLad added 2 commits April 13, 2023 21:07
It's unnecessary during Docker builds and clutters the output.
This speeds up the build when the entire step can't be cached (e.g. the requirements
file changed), but the package list remains mostly the same.

The savings are... rather underwhelming, actually. I have observed about a
minute in savings, although it obviously depends on the network connection
speed. I think this is because pip is inefficient at loading from its own cache
(I have observed it loading the entire cached file into memory, for example).

Still, savings are savings, and we're getting them basically for free, so
why not.

Note that I only persist the HTTP cache, and not the wheel cache. That's because
any wheels that pip builds could depend on the system packages, and I don't want
old wheels to be reused if the system packages change.
@SpecLad SpecLad requested a review from azhavoro as a code owner April 17, 2023 18:32
@nmanovic nmanovic merged commit 65d43aa into cvat-ai:develop Apr 18, 2023
@SpecLad SpecLad deleted the docker-build-opt branch April 18, 2023 18:32
mikhail-treskin pushed a commit to retailnext/cvat that referenced this pull request Jul 1, 2023
This speeds up the build when the entire step can't be cached (e.g. the
requirements file changed), but the package list remains mostly the
same.

The savings are... rather underwhelming, actually. I have observed about
a minute in savings, although it obviously depends on the network
connection speed. I think this is because pip is inefficient at loading
from its own cache (I have observed it loading the entire cached file
into memory, for example).

Still, savings are savings, and we're getting them basically for free,
so why not.

Note that I only persist the HTTP cache, and not the wheel cache. That's
because any wheels that pip builds could depend on the system packages,
and I don't want old wheels to be reused if the system packages change.

Also, disable the pip autoupdate checks, which isn't much of an
optimization, but it gets rid of some pointless warnings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants