Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] Optimize cache for datasets downloading #9

Closed
2 tasks
frgfm opened this issue Oct 2, 2019 · 1 comment
Closed
2 tasks

[ci] Optimize cache for datasets downloading #9

frgfm opened this issue Oct 2, 2019 · 1 comment
Assignees
@frgfm
Labels
ext: tests Related to tests help wanted Extra attention is needed
Milestone
0.2.0

Comments

@frgfm
Copy link
Member

frgfm commented Oct 2, 2019 •
edited
Loading

🚀 Feature

Save downloaded datasets in CI cache.

Motivation

With the addition of datasets.OpenFire, the test require the download of the entire dataset. Considering that the samples are scattered in multiple URLs, the download is taking at least several minutes.

Pitch

  • find a unique identifier to checksum all dataset URLs
  • save cache once the job is completed to avoid downloading again if the file is the same
@frgfm frgfm added help wanted Extra attention is needed ext: tests Related to tests module: ci labels Oct 2, 2019
@frgfm frgfm added this to the 0.1.0 milestone Oct 2, 2019
@frgfm frgfm removed this from the 0.1.0 milestone Oct 11, 2019
@frgfm frgfm self-assigned this Dec 27, 2020
@frgfm frgfm added this to the 0.1.1 milestone Dec 27, 2020
@frgfm frgfm removed the module: ci label Jan 31, 2021
@frgfm frgfm modified the milestones: 0.1.1, 0.1.2 Feb 28, 2021
@MateoLostanlen MateoLostanlen mentioned this issue Apr 15, 2021
Closed
@frgfm frgfm modified the milestones: 0.1.2, 0.2.0 Jul 4, 2022
@frgfm
Copy link
Member Author

frgfm commented Jul 17, 2022

Fixed by the num_samples mechanism from #33 & #156

@frgfm frgfm closed this as completed Jul 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frgfm frgfm

Labels
ext: tests Related to tests help wanted Extra attention is needed
Projects
None yet
Milestone
0.2.0
Development

Successfully merging a pull request may close this issue.

open_firev2 MateoLostanlen/pyro-vision
1 participant
@frgfm