File Upload: completed files remain in the "select files to add" box after finishing upload

[pameyer]

Uploaded ~150 ~11 MB files individually (not in zip/tgz)

• create new dataset
• files -> "+ Select Files to Add"
• select all (in browser window)
• small number of files (not fully reproducible - ~2-5) remain in the "Select Files to Add" box after upload completes (judging by drop in server load, presence of these files in the unlabeled box of files uploaded below, and presence on the filesystem (w\ matching checksums). Number of files shows up correctly after dataset save; suggesting that it's a UI problem. Observed in both firefox and safari.

Possibly related : #2491 , #2482

[pdurbin]

I'll probably try to reproduce this with files from https://data.sbgrid.org/dataset/133/ based on what @pameyer mentioned at http://irclog.iq.harvard.edu/dataverse/2016-01-20

[pameyer]

For reproducibility, I was seeing this on 4.2.2

[pdurbin]

I was seeing this on 4.2.2

Thanks. I wonder if the upgrade from Primefaces 4 to 5 in #2730 as part of 4.2.3 will make any difference in reproducing this bug.

[pdurbin]

On my laptop I'm on commit a90be2a of the 4.2.3 branch (which already has the Primefaces 5 upgrade added in #2730) and I just tried uploading all 148 files from https://data.sbgrid.org/dataset/133/

The main thing I want to do for now is upload a screenshot of what I'm seeing, which I'll put below, to make sure this is what @pameyer is talking about. Rather than 2-5 files showing up under "Select Files to Add" there are 8 in the screenshot.

Other observations (while using Firefox) are that my Mac was "beachballing" during the upload and I couldn't switch tabs. I'm pretty sure my fans were blowing pretty hard. That is to say this feel like an expensive operation for a browser.

---

Update: I showed the screenshot to @scolapasta and we agree that it's a bug that dos_mad_2_070.img, for example, shows at the top under "Select Files to Add" since dos_mad_2_070.img appears in the file list lower down.

[pameyer]

@pdurbin - that screenshot matches what I was seeing. The number of files varied (I deleted and re-uploaded a few times).

I didn't notice any beachballing - but I wasn't watching for it.

[pameyer]

@pdurbin - the fans spinning up on your laptop might have been the dataverse app (ingestion?) rather than the browser?

[pdurbin]

@pameyer I had that thought as well but I don't have any evidence one way or the other. :)

[pdurbin]

@bmckinney for now can you please see if you can reproduce this in the "develop" branch?

[bmckinney]

@pdurbin and @pameyer - confirmed with develop branch

Also, not only do random successful uploads linger in the "Select files" section, but one of the 120 files I attempted to upload failed to upload and it wasn't listed in either section. If I count the files in dataverse's temp folder (prior to saving the dataset), it is also 1 shy of the 120 total. This bug is more concerning to me!

[pdurbin]

If I count the files in dataverse's temp folder (prior to saving the dataset), it is also 1 shy of the 120 total.

I assume you're talking about the JSF temp folder? (#2848 is about documenting all of them.) If so, you could repeat your test via SWORD to avoid JSF.

[bmckinney]

also seeing lots of java.util.ConcurrentModificationException when checking for duplicates

[bmckinney]

yeah, i think edu.harvard.iq.dataverse.EditDatafilesPage#isDuplicate (which generates java.util.ConcurrentModificationException) is the culprit.

when i comment out the md5 stuff in that method and return false in all cases, i'm unable to reproduce the bug (all files reported uploaded and no lingering files in the top section)

[kcondon]

There is a pull request made to fix this. Will test and close.

[kcondon]

OK, the fix did not correct the one missing file issue, that is being tracked in: #3116
We are seeing "hung up files" when uploading zip files, before unpacking, also in #3116.
I have not yet reproduced the initial problem reported in this ticket.

[kcondon]

Tested the above scenario with the mentioned files. Was able to upload twice without errors or missing/ hung files. Closing this particular issue but as mentioned, some weirdness remains and will be addressed in #3116. Closing.