Enable asset upload

[waxlamp]

We need a way to upload assets from the web app, since most intended users of the archive will expect to be able to do that.

See dandi/dandiarchive-legacy#380 (comment).

[yarikoptic]

How realistic and "important" this to be done in any foreseeable future? ATM "upload" involves validation, metadata extraction, possibly fancy multi-threaded uploading for zarrs etc. May be with some https://pyodide.org compiled dandi-cli implementation though...

[satra]

i would say we need to focus on users that do not know the terminal, but let's consider all the things that would need to be done for such users.

[waxlamp]

At the moment, we tell users to go to the CLI for any upload needs at all. I'm just suggesting that if there is a feasible upload path for, say, single NWB files in the frontend, that's something we might be able to support without much trouble, while still leaving difficult cases like Zarr upload to other mechanisms.

I consider this a major usability issue for the archive as it currently operates. But this would be a great thing to discuss in depth, because perhaps I am misunderstanding the need for it. My argument comes down to: the DANDI Archive is a web application that lets people log in, download data, and create dandisets, but it doesn't allow upload when the average user would expect it to. This would address the "data upload" part of the archive's mission.

I'm also not suggesting we work on this right away, but it would be good to get some clarity on how important this is to us.

[satra]

@waxlamp - i agree with this being available, but this upload interface would still have to implement some of the sanity checks and validation that the CLI currently does.

[yarikoptic]

An idea: may be it would be some basic/naive JS upload script, uploading to some staging area on some EC, then (optionally if requested) running dandi organize, checking with user if ok, and then dandi validate and only then triggering actual upload from that EC2 to the archive if all good. if validation fail -- ask to either ignore or to reupload or fixup...

for reupload - if naive JS would not be that naive but implement some rsync like block-wise checksumming index (may be some node rsync implementation) for that upload and for reupload reuploads only changed parts (e.g. only some metadata changed) -- that would be quite nice.

for fixup - give some interface to change that nwb, e.g. metadata, so it could be changed straight on EC2, without causing traffic again. Might be actually even just creating the .overwrite.json file to accompany (hdmf-dev/hdmf#677 (comment))

May be all of that could be actually interfaced through some /uploads/interactive/{initialize,{upload_id}/reply} interfaces going through those stages described above and then JS just being a quite dummy interpreter for the questions which API would provide in response and user would provide responses in /reply.