Support S3 storage to store videos

[Chocobozzz]

We already have a documentation to mount a s3 storage, but there are some limitations and issues with fuse-mounted s3 storage. A proper support of s3 within PeerTube to store videos (webtorrent versions and HLS versions) would fix them.

[pierreozoux]

S3 is a product that belongs to AWS, and the common technology is called ObjectStore, so if you want to be agnostic about it, I kindly recommend to name it ObjectStore :) (But the awefull truth is that there is no specification, and S3 is the specification..)

[ghost]

I did try to use Backblaze B2 and PeerTube constantly fails to upload relatively big files (~300 MB) and start uploading again with a different name (I checked backend).
Also while uploading it consumes massive amount of "Download traffic" Bandwidth.

Definitely we need a better support for S3.

@Chocobozzz Thanks for your work

[arthurzenika]

We are using with OVH S3 https://www.ovhcloud.com/en/public-cloud/object-storage/ (which uses openstack's swift implementation) on kubernetes with s3fs sidecar (see https://docs.joinpeertube.org/install-unofficial?id=kubernetes) . Happy to help out with testing if this development starts.

[ghost]

Same problem as @grubeli with big files on my instance using S3FS with Wasabi : I get an error (browser) after the upload and before the Publish button becomes available (more than 5 minutes...).
Then (several hours later) it result with two videos.

[kontrollanten]

@manuelviens If you open up Chrome devtools, can you report the HTTP response code you see when the browser error appears (it should be 500, 504 or something similar)? A screenshot at the Console and Network tabs works as well.

[kukhariev]

@kontrollanten, @manuelviens ,
can you test with the s3fs -o nomultipart option?

[kontrollanten]

@kukhariev For me that doesn't help. I still get a 502 from my load balancer after 4 minutes.

The default timeout value in a browser is 300 seconds, but most network infrastructures include proxies and servers whose timeout is not that long.

https://tools.ietf.org/id/draft-loreto-http-bidirectional-07.html#timeouts

I can find two possible solutions for this:

A) After the upload is done, let the client do long polling until the file has moved. That would mean that the request is closed each 15 second and a new request is done until the move is done.
B) Add deeper S3 support in PeerTube. An example would be that the file is uploaded to a temporary directory on the server. And then the file is moved to S3 by a job. But this may lead to issues about how to handle full disk, etc.

[pingiun]

I want to try and make an implementation that uses the s3 api. I already have a basic version working. I will create a draft PR to get feedback on this

[Hoerli1337]

I currently have a problem with the S3 connection.

• I have everything installed locally on my home network
• PeerTube is installed in a LXC container and available via HTTPS
• I use MinIO as S3 storage
• MinIO is running on a local system on my network
• I currently do NOT use SSL for MinIO, because it is in the private network and I cannot roll out a valid SSL certificate
• MinIO is available on port 9000

Uploading videos works, but forwarding to MinIO does not.

Error message:

Dec 19 19:02:20 ctPeerTube peertube[16331]: [ctpeertube.fritz.box:443] 2021-12-19 19:02:20.883 error: Cannot execute job 1 in queue move-to-object-storage. {
Dec 19 19:02:20 ctPeerTube peertube[16331]:   "payload": {
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "videoUUID": "2c243f73-0864-40de-93fc-9281e7e49f23",
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "isNewVideo": true
Dec 19 19:02:20 ctPeerTube peertube[16331]:   },
Dec 19 19:02:20 ctPeerTube peertube[16331]:   "err": {
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "stack": "Error: getaddrinfo ENOTFOUND peertube.192.168.178.200\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:72:26)",
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "message": "getaddrinfo ENOTFOUND peertube.192.168.178.200",
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "errno": -3008,
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "code": "ENOTFOUND",
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "syscall": "getaddrinfo",
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "hostname": "peertube.192.168.178.200",
Dec 19 19:02:20 ctPeerTube peertube[16331]:     "$metadata": {
Dec 19 19:02:20 ctPeerTube peertube[16331]:       "attempts": 1,
Dec 19 19:02:20 ctPeerTube peertube[16331]:       "totalRetryDelay": 0
Dec 19 19:02:20 ctPeerTube peertube[16331]:     }
Dec 19 19:02:20 ctPeerTube peertube[16331]:   }
Dec 19 19:02:20 ctPeerTube peertube[16331]: 

My configuration:

object_storage:
  enabled: true

  # Without protocol, will default to HTTPS
  endpoint: 'http://192.168.178.200:9000'
  # 's3.amazonaws.com' or 's3.fr-par.scw.cloud' for example

  region: 'de-home-rock64'

  credentials:
    # You can also use AWS_ACCESS_KEY_ID env variable
    access_key_id: 'User-Name-Of-MinIO'
    # You can also use AWS_SECRET_ACCESS_KEY env variable
    secret_access_key: 'SuperSecretKey'

  # Maximum amount to upload in one request to object storage
  max_upload_part: 2GB

  streaming_playlists:
    bucket_name: 'peertube'

    # Allows setting all buckets to the same value but with a different prefix
    prefix: 'livestream/'

    # Base url for object URL generation, scheme and host will be replaced by this URL
    # Useful when you want to use a CDN/external proxy
    base_url: ''

  # Same settings but for webtorrent videos
  videos:
    bucket_name: 'peertube'
    prefix: 'videos/'
    base_url: ''

Maybe someone has an idea.
I'm new to the S3 storage world and just tinkering with PeerTube.

NextCloud works great with MinIO as external storage.

[pingiun]

@Hoerli1337 a closed issue is not really a good place to ask new questions but I can answer your question. In your case the issue is that you're using an IP address as the s3 host. I think you may be able to solve this by adding entries in your hostfile (/etc/hosts):

192.168.178.200 s3host
192.168.178.200 peertube.s3host

Then use s3host in the object_storage.endpoint instead of your ip address. You can see how s3 gets reached here: https://github.com/Chocobozzz/PeerTube/blob/develop/server/lib/object-storage/urls.ts#L31-L35

[Hoerli1337]

Thanks for the quick reply.
I currently have not been sure if it is a bug on my part or if it is a PeerTube bug.
That's why I wrote here instead of opening a new issue.
But the answer was good :)
At least this has brought me one step further now.
Now I get the following error message:

Dec 19 20:16:05 ctPeerTube peertube[712]: [ctpeertube.fritz.box:443] 2021-12-19 20:16:05.489 info: Moving video 4551d8ee-1a01-44c2-9e58-5178908503e3 in job 7.
Dec 19 20:16:05 ctPeerTube peertube[712]: [ctpeertube.fritz.box:443] 2021-12-19 20:16:05.525 error: Cannot execute job 7 in queue move-to-object-storage. {
Dec 19 20:16:05 ctPeerTube peertube[712]:   "payload": {
Dec 19 20:16:05 ctPeerTube peertube[712]:     "videoUUID": "4551d8ee-1a01-44c2-9e58-5178908503e3",
Dec 19 20:16:05 ctPeerTube peertube[712]:     "isNewVideo": true
Dec 19 20:16:05 ctPeerTube peertube[712]:   },
Dec 19 20:16:05 ctPeerTube peertube[712]:   "err": {
Dec 19 20:16:05 ctPeerTube peertube[712]:     "stack": "Error: write EPROTO C0F75595A57F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n\n    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)",
Dec 19 20:16:05 ctPeerTube peertube[712]:     "message": "write EPROTO C0F75595A57F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n",
Dec 19 20:16:05 ctPeerTube peertube[712]:     "errno": -71,
Dec 19 20:16:05 ctPeerTube peertube[712]:     "code": "EPROTO",
Dec 19 20:16:05 ctPeerTube peertube[712]:     "syscall": "write",
Dec 19 20:16:05 ctPeerTube peertube[712]:     "$metadata": {
Dec 19 20:16:05 ctPeerTube peertube[712]:       "attempts": 1,
Dec 19 20:16:05 ctPeerTube peertube[712]:       "totalRetryDelay": 0
Dec 19 20:16:05 ctPeerTube peertube[712]:     }
Dec 19 20:16:05 ctPeerTube peertube[712]:   }
Dec 19 20:16:05 ctPeerTube peertube[712]: }

[pingiun]

That seems like it's trying to do https to a http url, but i'm not sure why that might be the case. Have you used an http:// url like in your previous comment?