Support user upload directly to S3

[kontrollanten]

Describe the problem to be solved

Currently users are uploading their files to the PeerTube server. This may affect the upload speed but it also makes PeerTube hard to scale. In case running a single PeerTube server behind a load balancer and that server is replaced by another, it will make any ongoing uploads to fail.

Describe the solution you would like

Uploads directly to S3. Based on this guide https://www.altostra.com/blog/multipart-uploads-with-s3-presigned-url it could work as follows:

1. Client creates a POST video request to the PT server
2. Server creates a multi part upload and pre-signed URL:s for each part of the upload. The upload is stored in the database, paired with the video id. Pre-signed URL:s are returned to the client.
3. Client saves the response in local storage, to be able to resume in case of closing the tab.
4. Client uploads to S3.
5. Client sends ETag and part number to PT server after each finished upload part.
6. Server stores ETag and part number in database.
7. Client sends finish request to PT server.
8. Server ends the multipart upload.

This would be a great step forward to be able to scale PeerTube.

[emansom]

The node-uploadx component used to handle uploads within PeerTube supports S3. 🎉🚀

Hook that up to object storage config and it should work. 😊

[kukhariev]

uploadx can upload data directly to Google Cloud Storage(clientDirectUpload option) , but not to S3 storages.

Here is a proposed client uploader that supports both local storage and S3.
server draft

[emansom]

uploadx can upload data directly to Google Cloud Storage(clientDirectUpload option) , but not to S3 storages.

Here is a proposed client uploader that supports both local storage and S3.
server draft

Misinterpreted the issue on first read, thought it was talking about S3 upload on server side.

Thanks for implementing the client bits! 🚀

[kontrollanten]

@kukhariev I saw that you released new versions with support this. Huge thanks for your work! 🚀

Should we treat it as experimental or should it be stable?

[kukhariev]

@kontrollanten i hope - stable. It's difficult to testing.

[kontrollanten]

I've been looking deeper into this now. The current video upload functionality parses the video file to get the resolution and video duration when upload is done. I propose that instead of parsing the video file during the HTTP request, it should be handled by a job. By putting workload from the web server we minimize the risks of crashing ongoing processes when doing restarts/upgrades of PeerTube. In the long run PeerTube should support remote workers for handling jobs, whom can be terminated in a controlled way (i.e. terminate when all ongoing jobs are done).

@Chocobozzz any comments, or should I start working on a PR where video parsing is done in a job?

[kontrollanten]

I created a sequence diagram showing a possible workflow

https://sequencediagram.org/index.html#initialData=C4S2BsFMAIGUGZoBMQCdIGNjQK4AdwB7AQyQCgzitDVoBBcEDSMgM0IDthiB3SAZ0IBbGKwBM8RAAVIkVABUcAIxZDiwOSGLgkAkAHMOTTv2gAxAAwB2AMIA2aFML9g+9KYDEku1YCcARjIkdWIlYn4YACVIFE90cjUNVC0dPUNjDlNLCzoADjhEDyVwHBYyBiZIAFoAPhk5RRUALikAeVh5aAB6fCJSKvccNWKWCuYAHiqq+oVlSBb3Aw4Y6ABVSIAZaHZaBFwCEnIx6pqEJtWD0mgANxBdQmg8YlRgfnLGCamzgFF5Yn1oMQOEhHs9sBwhipUO9KrUZo15m0Ot1bvcyPC5nDnK53E1oDZ0OoYAA1O6QB6QLhgACe6NksxUtWisSadCQIIwhI0VVR5KqrBAUCqACtCEoYZ9pvSEU1SWi3IR8NAPGJxJArEpoJzIESeWTCPzBdVRZqPDwABZgGB0hqYmrMkD8JoAcUg2F5DxE3GC3BtDJOZ2gAEkjKAiTd9chCDwOH1yBxCBpoIRrnJHNK5i1nhFtkbtjRkDhUOoQJxAcDoMBzZCOMRBW8MYy6ti3AImgSdUm5eSzHnKaBgLSE0mU2nG-MO+GPVVgMXMhhCCgOACTW9KUggA

[aeharding]

This issue feels like the natural next step now that remote transcoding is finished (#947).

PeerTube would never have to touch a single video file!

[kontrollanten]

Updated draft including runners and chapters:

https://sequencediagram.org/index.html#initialData=C4S2BsFMAIGUGZoBMQCdIGNjQK4AdwB7AQyQCgzitDVoBBcEDSMgM0IDthiB3SAZ0IBbGKwBM8RAAVIkVABUcAIxbsuvAcNETEAIhlzFK6KhwcOc3dGL9oU+QCUyQ4sDkhi4JAJABzDkyctgBiAAwA7ADCAGx2hPzAvui2AMSS0eEAnACMZEiuxEo2MA6QKKno5C5uqB5ePv6BHCGhoXQAHHCIKUrgOCxkDEyQALQAfAYKypAAXFIA8rDy0AD0+ESkI8k4Lr0sQ8wAPCMjk0azUsl+FkjQAKoOADLQ7LQIuAQk5AejYwgzd0+pGgADcQN5CNA8MRUMB+INGEcTv8AKLyYi+awcW7Q2HQDg7FSoBHDcZnaZzRbLFZgiFkckqMnxRLJGbQSLoVwwABq4MgkMgXDAAE96bIpoyJsykgIZnQkLcHGYLKgAFKEJQkpGncXnGa8un2BwnCa6ikAcUg2CV5jk6s1RuO-wAIoQeBwNrdWCAoGQOIQ3NBCCC5HZHHMYfxRD7RDRkDhUK4QJwsbdgAALQkcYg++H+wPB0NGiOoKMvGMvOMYdPEPA1eFGslmlQzABUrYWS1WtZANOyK1MttLKwAVhr+CsAN5jpR3O4ASWdAF8ViJuPluO2ADpcTNCJTZ3NsgDaAF1J+WoGylCBs6hhQAaaDAYV4WbPvcHnPgAA+eHQYKQDw0BLju1a1vWJ7ns+IAiBghDeGyBL7nIT6gBA74JLUHCYkuYqGNMTIJDK-AzBykBctABr8sEFaCuhopAA