-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MAINT: Report download time and size #11635
Conversation
…ctions] [skip cirrus]
Looks like this might be one culprit, looking locally after watching CircleCI have a 10-minute timeout trying to download it:
we already have a mirror on |
Okay @drammock this one is ready for review. Mirroring data on OSF.io took time from ~45 min down to ~15 min as you can see here (along with all timings for downloads -- I killed the build once that step was done): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a nice improvement. CIs are not happy though. Cirrus fails with
mne/datasets/sleep_physionet/tests/test_physionet.py:132: in test_sleep_physionet_age
paths = age.fetch_data(subjects=[0], recording=[1], path=physionet_tmpdir)
<decorator-gen-588>:12: in fetch_data
???
mne/datasets/sleep_physionet/age.py:129: in fetch_data
sz += os.path.getsize(psg_fname)
/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/genericpath.py:50: in getsize
return os.stat(filename).st_size
E FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/pytest-of-admin/pytest-0/physionet_files0/physionet-sleep-data/SC4001E0-PSG.edf'
which looks related / legit; haven't looked at any other failures
there are also
|
I think this PR broke circle:
|
* upstream/main: (50 commits) BUG: Fix bug with paths (mne-tools#11639) MAINT: Report download time and size (mne-tools#11635) MRG: Allow retrieval of channel names via make_1020_channel_selections() (mne-tools#11632) Fix index name in to_data_frame()'s docstring (mne-tools#11457) MAINT: Use VTK prerelease wheels in pre jobs (mne-tools#11629) ENH: Allow gradient compensated data in maxwell_filter (mne-tools#10554) make test compatible with future pandas (mne-tools#11625) Display SVG figures correctly in Report (mne-tools#11623) API: Port ieeg gui over to mne-gui-addons and add tfr gui example (mne-tools#11616) MAINT: Add token [ci skip] (mne-tools#11622) API: One cycle of backward compat (mne-tools#11621) MAINT: Use git rather than zipball (mne-tools#11620) ENH: Speed up code a bit (mne-tools#11614) [BUG, MRG] Don't modify info in place for transform points (mne-tools#11612) [BUG, MRG] Fix topomap extra plot generated, add util to check a range (mne-tools#11607) ENH: Add mne-bids-pipeline to mne sys_info (mne-tools#11606) MAINT: `coding: utf-8` is implicit in Python 3 (mne-tools#11599) ENH: Read eyetracking data (Eyelink) (Fork of mne-tools#10855 ) (mne-tools#11152) MAINT: In Python 3, do not prefix literals with `u` (mne-tools#11604) MAINT: object is an implicit base for all classes (mne-tools#11601) ...
* upstream/main: BUG: Fix bug with paths (mne-tools#11639) MAINT: Report download time and size (mne-tools#11635) MRG: Allow retrieval of channel names via make_1020_channel_selections() (mne-tools#11632) Fix index name in to_data_frame()'s docstring (mne-tools#11457) MAINT: Use VTK prerelease wheels in pre jobs (mne-tools#11629) ENH: Allow gradient compensated data in maxwell_filter (mne-tools#10554) make test compatible with future pandas (mne-tools#11625)
This reverts commit 8fc3d07.
* upstream/main: (152 commits) FIX: missing channels/fiducials can be np.nan (mne-tools#11634) use py3.10 in precommit config (mne-tools#11648) MAINT: Unify GH Actions pytest (mne-tools#11644) MRG: Rename "Discourse" link in top navigation to "Forum" [ci skip] (mne-tools#11649) ENH: Add support for Harmonic Field correction (mne-tools#11536) Add pre-commit (mne-tools#11541) BUG: Fix bug with paths (mne-tools#11639) MAINT: Report download time and size (mne-tools#11635) MRG: Allow retrieval of channel names via make_1020_channel_selections() (mne-tools#11632) Fix index name in to_data_frame()'s docstring (mne-tools#11457) MAINT: Use VTK prerelease wheels in pre jobs (mne-tools#11629) ENH: Allow gradient compensated data in maxwell_filter (mne-tools#10554) make test compatible with future pandas (mne-tools#11625) Display SVG figures correctly in Report (mne-tools#11623) API: Port ieeg gui over to mne-gui-addons and add tfr gui example (mne-tools#11616) MAINT: Add token [ci skip] (mne-tools#11622) API: One cycle of backward compat (mne-tools#11621) MAINT: Use git rather than zipball (mne-tools#11620) ENH: Speed up code a bit (mne-tools#11614) [BUG, MRG] Don't modify info in place for transform points (mne-tools#11612) ...
* upstream/main: (117 commits) FIX: missing channels/fiducials can be np.nan (mne-tools#11634) use py3.10 in precommit config (mne-tools#11648) MAINT: Unify GH Actions pytest (mne-tools#11644) MRG: Rename "Discourse" link in top navigation to "Forum" [ci skip] (mne-tools#11649) ENH: Add support for Harmonic Field correction (mne-tools#11536) Add pre-commit (mne-tools#11541) BUG: Fix bug with paths (mne-tools#11639) MAINT: Report download time and size (mne-tools#11635) MRG: Allow retrieval of channel names via make_1020_channel_selections() (mne-tools#11632) Fix index name in to_data_frame()'s docstring (mne-tools#11457) MAINT: Use VTK prerelease wheels in pre jobs (mne-tools#11629) ENH: Allow gradient compensated data in maxwell_filter (mne-tools#10554) make test compatible with future pandas (mne-tools#11625) Display SVG figures correctly in Report (mne-tools#11623) API: Port ieeg gui over to mne-gui-addons and add tfr gui example (mne-tools#11616) MAINT: Add token [ci skip] (mne-tools#11622) API: One cycle of backward compat (mne-tools#11621) MAINT: Use git rather than zipball (mne-tools#11620) ENH: Speed up code a bit (mne-tools#11614) [BUG, MRG] Don't modify info in place for transform points (mne-tools#11612) ...
CircleCI has timed out a couple of times lately due to our 3h limit. It costs more $$$ to increase this limit and I think it's pretty reasonable anyway, so let's try to make it faster. 2h is spent in the doc build mostly due to examples/tutorials -- which we should certainly continue to try to optimize -- but recently 45min has been spent in dataset downloading. This PR adds some diagnostic download time/size
logger
to our fetching that should help us see it in logs, and might be nice for users.