File Download: Allow user to download all files from a dataverse at once.

[eaquigley]

---

Author Name: Kevin Condon (@kcondon)
Original Redmine Issue: 4086, https://redmine.hmdc.harvard.edu/issues/4086
Original Date: 2014-06-06
Original Assignee: Gustavo Durand

---

This was requested by a dv admin, Janina:

Allow a user to download all files from a dataverse at once, see RT#179817

do you know if there is a way to download all of the files in our Dataverse at once?

[scolapasta]

Moved to 4.1 to decide if we actually want something like this (what if a dataverse has GBs and GBs of files?)

[kcondon]

Under the Download button, the UI currently says, All files from this dataset, but it is grayed out.
A user was asking whether this was broken, see RT #196683, also Liz added a ticket to allow selecting all files at once for download: #1988

[raprasad]

Based on # of files + sizes, can there be a quick estimate of whether:

• This can be done fairly fast (in seconds) for that user OR
• Request into a queue
  • User sees message saying to expect an email + dv notification for when the download is ready.

[sbarbosadataverse]

@scolapasta
This was requested by Harvard GSD just today and had to be passed to Kevin.

[kcondon]

Will bring up at GIRT to see whether it is realistic to pursue this.

[donsizemore]

Feature request (which might better belong in a separate issue?):

Thu-Mai often needs to download all files from a dataset in their original format. I'm looking into scripting this through the Native and Data Access APIs, but a GUI option for archivists would be extremely helpful.

[pdurbin]

@donsizemore yeah, a separate issue might be nice. What you and Thu-Main want is actually a smaller "user story" since it's limited to a single dataset. This issue is about a whole dataverse, which might contain sub-dataverses (which might contain sub-dataverses).

[pdurbin]

@donsizemore this issue came up today. Weren't you saying in IRC that you cooked up a script? Want us to take a look? 😄

[pdurbin]

"all mine does is download all files in a given dataset in original format writing out the original filename, but it could be smartly rewritten and extended" -- @donsizemore at http://irclog.iq.harvard.edu/dataverse/2017-04-07#i_51359

[pdurbin]

When we tried to estimate #4529 about downloading all file based on a persistent ID (DOI or Handle) we decided against implementing that feature due to concerns over performance problems: #4529 (comment)

This issue represents even more load on the server so by the same logic we wouldn't implement this either.

[djbrooke]

I'm going to close this for now. For the performance concerns, we could possibly revisit after implementing Lambda functions (#6093) that would take zipping datasets off the application server, or if we decide to pre-zip and store content in support of #6085. We'd need to take non-S3 installations into consideration.

[mankoff]

Hello. I'm interested in this feature (and commented recently on a related issue). I have a question after reading this thread:

Why is zipping required?

Based on my (limited, ancient) webserver admin experience, if the dataset is exposed as a folder, the individual files could be downloaded with wget. Compression can happen on-the-fly (and only by certain filetype?) by the webserver (e.g. Apache), or no compression at all, and the Dataverse does not need to bulk-zip everything before the download begins.

[pdurbin]

@mankoff I appreciate your out of the box thinking! Thanks for commenting on #4529 and #6505 as well! Let's move the conversation to one of those issues since they're still open. Alternatively, you're welcome to open a dedicated issue about this idea.