Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement remote transcoding #5769

Merged
merged 20 commits into from
May 9, 2023
Merged

Implement remote transcoding #5769

merged 20 commits into from
May 9, 2023

Conversation

Chocobozzz
Copy link
Owner

@Chocobozzz Chocobozzz commented Apr 21, 2023

After a few weeks of work, I'm happy to share the work on remote transcoding.

This PR will be merged on develop quickly to avoid conflicts and so we can test the feature in real world scenarios. It's doesn't mean that PR reviews/comments won't be taken into account :)

Here's the main features of remote transcoding:

  • Handles VOD and Live (miss studio transcoding, see below)
  • Extensible so we can process other heavy jobs by remote runners in the future (transcription for example)
  • PeerTube runner CLI script easy to use:
    • Inspired by Gitlab runners: a PeerTube runner can process jobs of a any PeerTube instance (after having used a registration token)
    • No native dependency (nginx, postgresql, sqlite etc) or public IP needed: a runner can be quickly spawn from the CLI on any server/computer
    • The runner script is bundled with its dependencies in one file so it's easy to install or use with npx
    • Multiple runners can run on the same machine with different configurations

There's still work to do (in another PR or directly on develop):

  • Handle studio transcoding jobs
  • Write openapi doc
  • Write admin documentation on remote transcoding
  • Use a working directory that is like tmp but not removed each time peertube is restarted (so we can store files needed by some jobs between 2 peertube restarts)
  • Check server perf because chunks updates for lives may be heavy

Architecture details:

  • A remote runner can register on the PeerTube instance using a registration token generated (shared secret) by the admin
  • If the remote runners feature is enabled by the admin, transcoding jobs are created in the database and not in Redis
  • PeerTube waits for any registered runner to process the job
  • The runner can abort (on runner restart), error, update (to upload live chunks or notify about VOD progress) or success (VOD or live transcoding ended) the job
  • The runner and PeerTube communicate using classic REST API. The runner is only in client mode so it doesn't need to expose any API to PeerTube. A websocket (using socket.io) can be established between peertube and the runner to quickly notify about available jobs (useful to transcode lives)
  • PeerTube runner is in a dedicated package directory (I would like to use Lerna in the future, but there's still work to do to organize PeerTube components)
    • The admin can register, unregister or list registered PeerTube instances using the CLI
    • In server mode, the runner checks for available jobs on registered PeerTube instances
    • To process VOD jobs, the runner downloads the input file, and upload the result
    • To process live jobs, the runner ingests the RTMP URL and regularly uploads .m3u8 playlists and .ts chunks

@ROBERT-MCDOWELL
Copy link
Contributor

it would be nice to think about cluster architecture, like several nodes sharing the same NFS storage and have the task able to check between nodes which one is working on the transcoding....

@Chocobozzz Chocobozzz force-pushed the feature/remote-transcoding branch 3 times, most recently from 9364d90 to 1287c14 Compare May 3, 2023 11:50
@Chocobozzz Chocobozzz force-pushed the feature/remote-transcoding branch 2 times, most recently from beef706 to cdeff5f Compare May 5, 2023 14:32
@Chocobozzz Chocobozzz merged commit 2160925 into develop May 9, 2023
@Chocobozzz Chocobozzz deleted the feature/remote-transcoding branch May 10, 2023 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants