SWORD upload limit, change from int to long

[raprasad]

Currently the SWORD library returns a Java int for getMaxUploadSize

see: https://github.com/IQSS/dataverse/blob/master/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordConfigurationImpl.java#L126

Change from an int to a long``to accommodate 10gb files*. Or anything beyond Java's``Integer.MAX_SIZE of 2,147,483,647

• contact sword team
• potentially change upstream code, etc.

Related: #2149

[pdurbin]

From looking at the DVN 3.x code at https://github.com/IQSS/dvn/blob/97fc08b8d9364ddcacd24045483086d4ed7610d6/DVN-root/DVN-web/src/main/java/edu/harvard/iq/dvn/api/datadeposit/SwordConfigurationImpl.java#L96 it looks we put in equivalent behavior in DVN 3.x.

We would try Integer.parseInt(maxUploadInBytes) and catch NumberFormatException and return unlimited and log the following:

logger.fine("Could not convert " + maxUploadInBytes + " from JVM option " + jvmOption + " to int. Setting Data Deposit APU max upload size limit to unlimited.");

(Whoops. "API" that should have been!) 👎

The point is that DVN 3.6 and Dataverse 4.0 are going to work the same way in this regard. Good, I guess but I agree that this should be an long rather than an int. An int is too small. Big data, here we come. 👍

Let me reach out to http://www.mail-archive.com/sword-app-tech@lists.sourceforge.net/ and get a pulse on this. My last pull request at swordapp/JavaServer2.0#6 still hasn't been commented on and there was no reply to http://www.mail-archive.com/sword-app-tech@lists.sourceforge.net/msg00421.html but I'd like to try.

The maintainer (@richard-jones) is looking for help in general: http://www.mail-archive.com/sword-app-tech@lists.sourceforge.net/msg00400.html

[kcondon]

The intent of our platform is to not have many built in limits but in practice we are not allowing 10GB uploads currently. So, since there is a dependency on SWORD we will mark this change as a feature.

[pdurbin]

For the record, I have not yet reached out the to the SWORD mailing list yet as I hinted I might (almost a year ago): #2169 (comment)

[tomck]

Could I get clarification on this? The docs, referencing this issue, say "If the MaxFileUploadSizeInBytes is NOT set, uploads, including SWORD may be of unlimited size." I removed the limit by using curl -x DELETE, but it seems the text above the file upload box isn't variable and still displays "File upload limit is 2GB per file."

Attempting to upload via Dropbox and my temp uploads seem to hit 7.4GB and halt. An upload of a tar.xz file got stuck after uploading (I can't find it in the actual data storage folder on the system and it's not showing up in the file listing, is the system trying to extract it?).

edit: issue was due to multiple temp files filling disk, switching to using a larger storage disk and relocating files and using symlinks for temp files seems to have resolved it mostly.

[pdurbin]

@tomck it sounds like you've hit bug #3071 (File Upload: Make upload limit text dynamic based on setting) which is fixed in the "develop" branch and will be part of Dataverse 4.4.

If the docs on MaxFileUploadSizeInBytes aren't clear, please let us know.

Since you seem to be operating on somwhat large files, you may be interested in the work going on at #3145 (still early days).

[pdurbin]

They're working on SWORD v3: https://groups.google.com/d/msg/dataverse-community/D7lRTA3f8Hc/nX0v50MKAgAJ

Hopefully this issue will be addressed there. This issue doesn't seem to affect very many people. Closing.

[pdurbin]

These days we're running @poikilotherm's version of the SWORD library (🎉 see PR #8000) so I just opened gdcc/sword2-server#43 about the upload limit.