rclone is SLOW on WebDAV

[cyberpower678]

Despite setting the transfers option to 32 parallel transfers, and the logs are confirming that rclone has set the --transfers option to 32, and also setting the NextCloud chunk size to 1 Gi, also confirmed by the logs, uploading to NextCloud is so painfully slow, the buffer backs up almost immediately, despite NextCloud living in a neighboring k3s pod, ie on the same machine. Latency between them is 3 microseconds, and obviously network stack IO is limited by how fast the CPU can move the data through the k3s cluster.

Setting the option appears to do nothing as it seems to me the uploads are still happening one at a time. Can something be done about this. Transfer speeds are only 25 Mbps to NextCloud which is quite awful for this setup. Camera events are often many little files rather than few big files making the opening and closing of the connection/sockets more limiting than the bandwidth. As such parallel uploads would be immensely useful.

[ep1cman]

The way UPB currently works, is that it invokes the rclone rcat command once per downloaded file (which is stored in memory where possible to avoid writing to disk). As such, any kind of parallelism rclone provides is pointless, because it is only ever handling a single file at a time. The uploader in UPB is also sequential and does not fire multiple uploads at once (same for the uploader).

I could see it being possible to have an optional upload parallelism within UPB but I don't think there is any way to make use of rclones parallelism without writing files to disk.

[cyberpower678]

Maybe I can fashion up something for you.

[cyberpower678]

#153

I threw something together for you, but I haven't tested it since I lack time to do that right now. Should just work, in theory.

[alash2k]

Hi, I was trying to decrease the replication lag from NVR to cloud and I came across this. Is this working and can it be used?

Can this be merged to main?