feat(ci): enable pip cache

[SauravMaheshkar]

This PR sets the cache and cache-dependency-path argument of actions/setup-python to enable pip caching.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

Hi @SauravMaheshkar, thanks for opening a PR! If I understand it correctly this integration hashes the requirements/setup files to cache dependencies. In that case we would be quite blind to breaking changes from new versions of a library, right?

[SauravMaheshkar]

@lvwerra as far as I understand, the action reuses cache if the files listed under the cache-dependency-path haven't been changed for while. If the files have been recently changed then it updates the cache. So we should not be blind to breaking changes from new versions.

[lvwerra]

I think the issue is that we don't change the files in question often so it is possible that there are breaking changes that we don't notice because we never updated the cached files. Is there an option to set a cache lifetime?

[SauravMaheshkar]

AFAIK there isn't a option to set cache lifetime however we can use the cache-hit flag to check if a cache hit has occurred. Source

[lvwerra]

Hi @SauravMaheshkar, not sure I follow how the cache-hit would help. I would like to at least clear the cache once every 1-2 days since libraries might be updated that the we won't install due to caching.

[SauravMaheshkar]

Hey @lvwerra thanks for the feedback, you do make an interesting case. Upon searching around for a bit, we can create another workflow similar to the one in the Github Docs which can clear all the caches every n days.

[SauravMaheshkar]

Do you want me to work on that as well in this PR ?

[lvwerra]

Hi @SauravMaheshkar, yes, if we can delete the cache every 1-2 days adding caching would be a good feature.

[SauravMaheshkar]

@lvwerra I added a proposal for a workflow in e19ce20

[lvwerra]

Awesome work @SauravMaheshkar , that looks great to me! What do you think @younesbelkada ?

[younesbelkada]

Thanks a lot for this awesome feature!
I am totally ok for this PR, I just have few (basic) questions

• Where the duration of the cache refreshing mechanism (you stated each 1 or 2 days) can be changed on the file you have added?
• Why the changes on tests.yml are needed? It seems to be applied only on python 3.8, do we need to change it as well for other python versions?
  Thanks a lot!

[lvwerra]

For the first question - cron: "0 0 * * *" specifies refresh at midnight (see here). Also curious about the second one!

[younesbelkada]

For the first question - cron: "0 0 * * *" specifies refresh at midnight (see here).

Awesome thank you!

[SauravMaheshkar]

Thanks a lot for this awesome feature! I am totally ok for this PR, I just have few (basic) questions

• Where the duration of the cache refreshing mechanism (you stated each 1 or 2 days) can be changed on the file you have added?
• Why the changes on tests.yml are needed? It seems to be applied only on python 3.8, do we need to change it as well for other python versions?
  Thanks a lot!

We can run the tests on other python versions as well. The changes proposed by this PR is independent of the python version. We can test for more python versions as well.

[younesbelkada]

Thanks a lot for clarifying!

The changes proposed by this PR is independent of the python version.

I am a bit confused by this, can you elaborate a bit more on that?

Also I can see that the modified block on tests.yml only touches the check-code-quality workflow. Ultimately my question is if you can point me to the place that ensures the caching refreshing mechanism is triggered on the CI runners?

Again thank you!

[lvwerra]

Hi @SauravMaheshkar just to double check: it seems like the cache is only a pip download cache and the libraries still need to be installed (which takes a significant amount of time). Is this expected?

[SauravMaheshkar]

Hi @SauravMaheshkar just to double check: it seems like the cache is only a pip download cache and the libraries still need to be installed (which takes a significant amount of time). Is this expected?

Yes it's just a download cache AFAIK