MRG: Use caching in Github Actions and Azure Pipelines #8695
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
larsoner
reviewed
Jan 4, 2021
.github/workflows/compat_minimal.yml
Outdated
| id: cache-dataset | ||
| with: | ||
| path: /home/runner/mne_data | ||
| key: ${{ runner.os }}-dataset |
There was a problem hiding this comment.
I don't think we need to separate by OS, the data files should all be the same
|
Among the first batch of runners, no one detected the cache: and Failing skips the I'll push an empty commit to confirm that they at least restore the cache successfully. |
larsoner
reviewed
Jan 4, 2021
|
|
|
I see no problem at restoration 👌 :
|
|
Do you want to try dealing with testing dataset |
|
You'll probably be more efficient than me 😅 . Go ahead if you have the time. |
|
Will do now, could really use something in my wheelhouse to jump-start things over here... |
|
@GuillaumeFavelier if you want to look into it, we should do the same for Azure |
|
@GuillaumeFavelier you can use something like the following in a one-liner When Dan's #8679 is in progress (probably after my PR then yours) then this line can probably be adjusted fairly easily |
|
Checking the key based on the testing version is a success!
|
|
The Perl extension is not available on MacOS: |
|
Also, the cache is different depending on the OS: On MacOS: For the others (on Linux): |
|
Because there is a And no way to make their tip work since it's not a fixed string but it comes from a variable:
I'll |
|
I don't know what to say 🤦 |
|
The jobs on Windows detected the key: And restored the cache correctly But the data are downloaded anyway because the path is different on Windows. |
|
Hah! 44 sec to download the data on And 1m 11 sec to get it from cache: So we actually get a slowdown from this on Azure at least (!) but at least it should be more reliable. |
|
Hm... But for some reason it downloads more no? Restoring the cache: Compared to |
GuillaumeFavelier
changed the title
That's probably the uncompressed size, vs the download which is compressed. |
|
Unrelated failure on Maybe one last empty commit for testing but everything is ok for me. Is there something missing @larsoner ? |
I don't think we need the empty commit, I'll just restart the segfaulted GitHub actions build, and it looks like everything else should be okay. Will merge once CIs are happy |
GuillaumeFavelier
changed the title
|
Thanks @GuillaumeFavelier ! |
|
Something interesting about caching:
Reference: https://github.com/actions/cache#cache-scopes |
larsoner
added a commit
to larsoner/mne-python
that referenced
this pull request
Jan 7, 2021
* upstream/master: MAINT: Actually use all caches (mne-tools#8702) MRG: Prototype of notebook viz (ipyvtk) (mne-tools#8503) MRG: Use caching in Github Actions and Azure Pipelines (mne-tools#8695)
cbrnr
pushed a commit
to cbrnr/mne-python
that referenced
this pull request
Jan 15, 2021
* First test [skip azp][skip circle] * Download anyway [skip azp][skip circle] * Unify cache key [skip azp][skip circle] * Test new key * Test linux_pip [skip azp][skip circle] * Try again [skip azp][skip circle] * Deploy to all jobs * Try another syntax * Test new key format [skip azp][skip circle] * Use home env variable * Fix key [skip azp][skip circle] * Try naive version on Azure * Try again [skip github][skip circle] * Try again [skip github][skip circle] * Try another syntax [skip github][skip circle] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Test again? [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Debug variable [skip github] * Test new key format [skip github] * Test new key format [skip github] * Test new key format [skip github] * Try again [skip github] * Try again [skip github] * Try again [skip github] * Deploy to all jobs * Try another syntax [skip github][skip circle] * Try another syntax [skip github][skip circle] * Try another syntax [skip github][skip circle] * Test again
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR follows #8685 (comment) and attempts to solve the recent failures related to testing data download.
ToDo:
$HOMEforpathvariable