Alternative solution for using external S3-Storage

[LukasSchulz]

Describe the problem to be solved

I have set up a testing-instance of peertube a while ago and I installed it following the excellent documentation given.
I didn't want to use the peering-aspect of peertube and therefore - to make it scalable I followed the installation instructions for mounting an external s3-storage with s3fs and using that as the storage for peertube.

The problem is that this setup was extremely slow and unstable for my instance. First of all, ffmpeg was slowed down by the s3fs always needing to check whether the file exists and other things, additionally files somehow went missing and videos became unplayable. - Maybe it's because i used Harddisks for my s3-storage with minio, but this shouldn't be a problem.

Describe the solution you would like:
But I found a solution:
In the second installation of peertube I am now using rclone and an external storage to sync the files on the s3-storage and the local storage of peertube.
The rclone was fairly easy to set up and it is run by a systemd-service which is triggered repeatedly after every sync using a systemd-timer unit.

It is really reliable and has almost all the advantages of the s3fs - but the most important thing is that it hasn't caused any problems as of this writing.

The only problem is that it takes some (configurable) amount of time for the sync to happen and therefore it can take sometimes 5 minutes for the video to be synced with the s3-bucket.

If this is a thing that should be supported I could write some piece of documentation for my setup.

Describe alternatives you have considered
I thought about alternatives to the s3fs approach but this is the best thing I could come up with.

[kontrollanten]

This sounds interesting! I've experience the same problem as you with s3fs. I just saw that s3fs has a caching feature, have you tried that? Either way I'm interested in seeing your solution. Thanks!

[PhieF]

Can you tell me if you have the following issue:

with rclone, when importing a video from youtube, I got conflicts on 1080p definition, which leads to 1080p to not work at all. That's why I switched to s3ql which doesn't allow for public bucket for s3ql is really good and with the caching is immediate

with s3fs / rclone mount / goofys you can avoid using your server as a proxy and directly stream the video from your s3 bucket, but with s3ql you need to use your server as a proxy

[LukasSchulz]

I just saw that s3fs has a caching feature, have you tried that?
No, I haven't tried that (at least I don't think so)

with rclone, when importing a video from youtube, I got conflicts on 1080p definition, which leads to 1080p to not work at all.
I don't know how you are getting conflicts with the 1080p definition. I imported videos using https and youtube-dl. I don't have any problems with youtube at all. - could you send me a video which didn't work so I can try it on my instance?

I also don't exactly understand what you mean by

but with s3ql you need to use your server as a proxy
The traffic gets redirected in the nginx-config of peertube to pass all requests to the .mp4-files to the s3-bucket directly.
Sure, I do need the storage on the peertube-computer, but that also means that if somehow the network fails, peertube doesn't error out or gets confused with the missing mount.

My command for syncing is rclone sync /mnt/peertube-storage/ cdn-node-1:peertube -v
I configured the s3-bucket using rclone config

As I said before, I set up a timer unit, which runs the rclone sync repeatedly.

Links to my setup:
rclone sync service: https://gitlab.infra4future.de/infra/doku/-/blob/master/hacc-media/cdn-sync.service
rclone sync timer: https://gitlab.infra4future.de/infra/doku/-/blob/master/hacc-media/cdn-sync.timer
the nginx-config was edited exactly the same way as in the s3fs-setup.
I also changed the peertube-paths to /mnt/peertube-storage for videos, streaming-playlists, previews (and also thumbnails, because I decided to serve them by the s3-bucket too), because I have a storage mounted at /mnt/peertube-storage.

[PhieF]

OK, so you only use rclone as a backup, not as a primary storage solution ?

Because rclone also has a "mount" option

[Chocobozzz]

Merging this issue in #3661

[raphgilles]

I just want to react here.

For my instance, I'm using SSHFS with a Hetzner Storage Box (because it's not so expansive). It's not exactly the same approach but it's working like a charm !

[lucacalcaterra]

I just want to react here.

For my instance, I'm using SSHFS with a Hetzner Storage Box (because it's not so expansive). It's not exactly the same approach but it's working like a charm !

@raphgilles i work on hetzner on server with quite space (few Tb of space). But i would like to migrate to a storage box. Any way our peertube visitors consume at least 25 Tb at month of traffic on my server. If i'll buy a storage box and configure it as video files repo (NFS or CIFS from proxmox) visitors traffic is applicable as external traffic or as internal ? In first case i should have the biggest package with 30Tb/month. Which suggestions can you give me ? Thanks !

[rajasekharponakala]

@raphgilles, hey any documentation on how to attach hetzner storage box to a bare instance..?