api, storage-service, frontend: File upload efficiency

[SamuelPull]

Ideally, documents are streamed directly to intended storage, without temporarily keeping them in redux store, in memory on server, or such.

File encoding and file transfer are two separate concepts, these need not to be done in one issue, but backwards compatibility is a must. System must know which file needs base64 conversion and which doesn't.

[jzakotnik]

@SamuelPull I think this is fixed, because it was tested up to 100mb? Or something missing?

[SamuelPull]

@SamuelPull I think this is fixed, because it was tested up to 100mb? Or something missing?

@jzakotnik Partially. Large documents can be uploaded and they travel from browser to api not encoded in base64, but downstream from there nothing changed. They are encoded to base 64 and stored as such in external storage, taking up more space (130% filesize) and computation, and making it inconvenient to inspect their content directly in blob storage.
It's not critical, but it will be much more efficient and inline with best practices to stream files directly to the storage if possible.
Slightly worse option is to create a separate file upload endoint in the api.
Currently every file is buffered in redux store before upload, sent together with workflow item data, and buffered in the api before being sent to storage. It was a good process when files were written directly in the chain, and were limited to smaller max size.
So there are at least two venues for improvement - file format and transfer. Outside of upload possibly even more, for example some api calls access the same file multiple times in different function calls.
I would like this issue to be left open as a known area of possible improvement, not that it needs a fix asap.