Download Manager related help text

[posixeleni]

A user recently reported (here https://help.hmdc.harvard.edu/Ticket/Display.html?id=232704) that they have had several people report to them that they were unable to download the files from their dataset. The file would start to download ok and then hang up, resulting in them having to re-start the whole process which for a larger file can take a lot of time.

I spoke with @landreev about this yesterday and we should look into a way to support the ability to resume a download which got interrupted.

[pdurbin]

@pameyer and I talked about this a bit at http://irclog.iq.harvard.edu/dataverse/2016-01-08#i_29469 and it's definitely something that's of interest for @sbgrid . Their in-house system uses rsync for this. I'm not sure the best way to implement this in Dataverse. This feature also seems strongly related to the Data IO Component BRD @raprasad is working on so I left a comment on it to check if it's in scope or not.

[pdurbin]

@bmckinney I'm giving you this issue to think about and discuss with @pameyer with regard to the needs of @sbgrid. Definitely open to suggestions. 😄

[bmckinney]

Thanks, I definitely have some thoughts on this. I'll plan to discuss with @pameyer and share a google doc.

[bmckinney]

See: https://docs.google.com/document/d/1JDCKh8UzB2fQoRkmlCU8BrY0iwH0dEGnOAgJhRZVepM/edit?usp=sharing

[pdurbin]

@bmckinney nice! Please see also the diagram by @michbarsinai at #2728 (comment) and the BRD: Data IO Component doc @raprasad has been working on.

[pdurbin]

@bmckinney I just mentioned your desktop app idea to @landreev and a standup and he agrees that you need something on the client side to coordinate resumption of uploads and downloads. Also, I'm reminded of a demo @astrofrog gave to me, @mcrosas (and others?) in May 2015. I just looked at http://irclog.iq.harvard.edu/search.pl?channel=dataverse&q=trobitaille but I can't find a reference to any code anywhere but it was written in Python. I do see a mention of Dataverse at https://github.com/glue-viz/glue/blob/16bc539736cb580ac4d42ccf5f6741ada553fde1/glue/main.py#L257 and there was a little discussion at dvn/swordpoc#3 . Anyway, if we do decide to pursue some sort of desktop client for Dataverse, let's create a new GitHub issue to track it.

[bmckinney]

before i forget, this project/protocol is somewhat relevant: http://tus.io/ caveats: only deals with uploads, currently no java server implementation. it could be a nice option for "somewhat large" file uploads

[pdurbin]

"Resumable upload supports uploading videos that are up to 1.75GB and 45 minutes long." https://developers.facebook.com/docs/graph-api/video-uploads#resumable via http://socialwg.indiewebcamp.com/irc/social/2016-06-10/line/1465619059043

Too small for our requirements but potentially interesting.

[pdurbin]

This issue is highly related since rsync supports the ability to resume uploads and downloads: Data Capture Module (rsync support) #3145

Truth be told, this issue is a little more about the download side of things and the DCM is about uploads.

[pameyer]

DCM is about uploads - but integration of components for rsync downloads are in other issues.

wget-able links will help here as well (aka - users will be able to do resumable downloads without other changes to dataverse, within the restrictions of files/datasets that will display a download link.

[pdurbin]

@pameyer good point about wget. I chronicled my experience with wget --continue at #5092 and it sounds like we're going to promote the use of non-browser tools like wget in #4949.

[mheppler]

Stumbled across this download issue, and saw the wget conversation. Reminded me that we added a bit of help text to the Download Package File from S3 #4949 issue.

We currently display a Download URL on the file pg, which provides download access via the API.

I suggest that we could resolve this issue by displaying similar help text above this Download URL field that explains the wget and download manager methods, as well as a link to the User Guide.

[pdurbin]

@mheppler fine with me.