Hardware accelerated encoding using Intel QuickSync via VAAPI

[lowne]

Hardware accelerated encoding using Intel QuickSync via VAAPI. Tested VP8 and H264 on a Coffee Lake CPU.

Adding support for AMD/VAAPI should AFAICT just be a matter of further adding the necessary packages. The PR leaves open the possibility of adding NVENC down the line - the hw encoder to use is requested via env var.

The end user is required to (via docker-compose or whatever else):

1. pass /dev/dri to the container
2. pass an env var NEKO_HWENC=VAAPI
3. optionally pass an env var RENDER_GID=<gid> with the group id of the render user (or the user that owns /dev/dri/renderD*) - it'll be fetched automatically otherwise (as long as /dev/dri is passed through) but it's not well tested yet

Notes

Sometimes on container startup X will throw a fit with (possibly non-exhaustive list):

• Xorg: ../../../../../../include/privates.h:121: dixGetPrivateAddr: Assertion 'key->initialized' failed.
• Xorg: ../../../../dix/privates.c:384: dixRegisterPrivateKey: Assertion '!global_keys[type].created' failed.

as soon as neko starts (and then neko itself dies). When that happens supervisord will restart both and soon enough they'll both manage to say alive and settle down.
Looking for clues reveals a bunch of long-standing X bugs with a bunch of supposed patches and the general advice of "start X as root kthx" (which is not the case in neko, as it stands).

All it seems to do in practice is delay startup by 2-3 seconds, but it's worth keeping in mind AND testing. From the looks of it, it might be just a case of reordering/delaying stuff at startup, but I am not going to touch supervisord.

[yesBad]

👀, looks pretty interesting. Looking forward to NVENC at some point :)

[m1k1o]

Looks good!