-
Notifications
You must be signed in to change notification settings - Fork 1.7k
ci: don’t store Pip HTTP cache #3167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Summary: The Pip HTTP cache stores all downloaded wheels and is never evicted. With new `tf-nightly` wheels every day, this adds up quickly. We last cleared our Travis caches about a month ago, and they’re up to 14.3 GB. Investigation shows that the Pip HTTP cache accounts for the majority of the cache (about 70% after about a month of cache accrual), and also that jobs with larger caches have significantly longer startup times, with delta on the order of 8 minutes (again, after about a month). Also, uploading large caches at the end of a job can take minutes, and Travis doesn’t report success until this finishes. Fetching `tf-nightly` should be comparatively cheap. Caches may need to be cleared for this to take effect. We’ll find out. Test Plan: See what Travis thinks. wchargin-branch: ci-drop-pip-http-cache
|
This PR reduces the “before install” time (i.e., time spent by Travis The “install” time is increased from 3m36s to 3m56s, which seems Travis is a bit deceptive about the time that it takes for it to restore |
|
|
||
| cache: | ||
| # Reuse the pip cache directory across build machines. | ||
| pip: true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pip: false then explain why opted out? Some stupid person like me may flip it back on thinking that it will improve our CI (and saved the world) :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea; thanks. Done.
|
@stephanwlee pointed out offline that while this is an improvement, it
The cron job deletion is kind of ugly (to have effectful state in the The cache structure requires a bit of extra logic and thinking, but If we run into issues where we decide that we want one-day caches back, |
wchargin-branch: ci-drop-pip-http-cache wchargin-source: 87d25ebd4ffd6db7bff21fa755b897892df1488d
|
Following up: Caches appear to be holding steady at around 3500 MB now. |
Summary:
The Pip HTTP cache stores all downloaded wheels and is never evicted.
With new
tf-nightlywheels every day, this adds up quickly. We lastcleared our Travis caches about a month ago, and they’re up to 14.3 GB.
Investigation shows that the Pip HTTP cache accounts for the majority of
the cache (about 70% after about a month of cache accrual), and also
that jobs with larger caches have significantly longer startup times,
with delta on the order of 8 minutes (again, after about a month). Also,
uploading large caches at the end of a job can take minutes, and Travis
doesn’t report success until this finishes. Fetching
tf-nightlyshouldbe comparatively cheap.
This reverts part of #2278.
Test Plan:
This PR reduces the “before install” time (i.e., time spent by Travis
internals before it gets to our script, including restoring cache) from
9m40s to 4m59s, a 48% improvement. The “install” time is increased from
3m36s to 3m56s, which seems acceptable.
wchargin-branch: ci-drop-pip-http-cache