@discordjs/opus version mismatch on linux

[me-vlad]

Vingester 2.2.5 for linux
OS: Fedora 34 x86_64, kernel 5.12.9-300, GPU Nvidia GeForce GT 755M, Gnome 40.1.0, Wayland 1.19.0, nouveau 1.0.17

While running frameless output with preview enabled - preview shows nothing on a linux Vingester version.
Same source with same settings on Windows version - works fine.
Screenshot:

[rse]

That's strange: I tested it once under Ubuntu with exactly the same broken preview as in your screenshot. I then enabled "Headless" mode and disabled "frameless" mode. For "headless" mode I've seen that my "Sample Rate" was not set, so I set it and I got a preview afterwards, too. Now the strange thing: I now switched back to "frameless" mode and it works now. Can it be that also for you in the fields of "headless" mode there are some problems? Can you show me the entire config or just export it and append it here?

[me-vlad]

Sure, exported config

%YAML 1.2
##
##  Vingester Configuration
##  Version: Vingester 2.2.5
##  Date:    2021-06-10 00:56
##

---

-   BrowserTitle:                  "The-Expert-4"
    BrowserInfo:                   "The Expert: IT Support (Square Project Episode 2)"
    BrowserWidth:                  854
    BrowserHeight:                 480
    BrowserColor:                  "transparent"
    BrowserZoom:                   1
    BrowserTrust:                  false
    BrowserNodeAPI:                false
    BrowserOBSDOM:                 false
    BrowserPersist:                false
    InputURL:                      "https://www.youtube.com/embed/ZOzzRlc_qho?autoplay=1&controls=0&rel=0"
    PatchDelay:                    0
    PatchFrame:                    ""
    PatchStyleType:                "inline"
    PatchStyleCode:                ""
    PatchScriptType:               "inline"
    PatchScriptCode:               ""
    Output1Enabled:                true
    Output1VideoPositionX:         0
    Output1VideoPositionY:         0
    Output1VideoDisplay:           1
    Output1VideoPinTop:            false
    Output1AudioDevice:            ""
    Output2Enabled:                false
    Output2VideoFrameRate:         30
    Output2VideoAdaptive:          false
    Output2VideoDelay:             0
    Output2AudioSampleRate:        48000
    Output2AudioChannels:          2
    Output2AudioDelay:             0
    Output2SinkNDIEnabled:         false
    Output2SinkFFmpegEnabled:      true
    Output2SinkFFmpegMode:         "cbr"
    Output2SinkFFmpegFormat:       "mpegts"
    Output2SinkFFmpegOptions:      "srt://127.0.0.1:10000?&pkt_size=1316&mode=caller"
    PreviewEnabled:                true
    ConsoleEnabled:                true
    DevToolsEnabled:               false
    Collapsed:                     false

[rse]

Very strange. If I wouldn't have seen the bug on my initial test, I wouldn't believe it. I've removed my entries, imported your file, enabled PREVIEW and it worked just fine (at least under this Ubuntu 20/04 under WSL under Windows 10, as I've no Linux desktop around, just servers):

Unfortunately, unless I can repeat the problem, I cannot figure out a solution, of course. We have to find a way to repeat the problem. Hmm...

[me-vlad]

I'll try to reproduce environment where this issue appears

[me-vlad]

Just installed Vingester from git (version 2.2.6)

1. cloned b554c31
2. npm install
3. npm start

During install got @discordjs/opus related messages (and @discordjs/opus starts compiling):

node-pre-gyp ERR! install response status 404 Not Found on https://github.com/discordjs/opus/releases/download/v0.5.0/opus-v0.5.0-node-v83-napi-v3-linux-x64-glibc-2.33.tar.gz node-pre-gyp WARN Pre-built binaries not installable for @discordjs/opus@0.5.0 and node@14.17.0 (node-v83 ABI, glibc) (falling back to source compile with node-gyp) node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/discordjs/opus/releases/download/v0.5.0/opus-v0.5.0-node-v83-napi-v3-linux-x64-glibc-2.33.tar.gz

After running npm start - preview works fine, main.log - without errors.

After starting downloaded from github (release 2.2.6) Vingester app - preview is missing (with same configs), and I see en error in the main.log file:

[2021-06-10 21:49:13.549] [error] (main) browser/worker-E24446D2: console: Uncaught Error: Cannot find module '/tmp/.mount_VingeshQ8eN4/resources/app.asar/node_modules/@discordjs/opus/prebuild/electron-v13.1-napi-v3-linux-x64-glibc-2.33/opus.node' Require stack: - /tmp/.mount_VingeshQ8eN4/resources/app.asar/node_modules/@discordjs/opus/lib/index.js - /tmp/.mount_VingeshQ8eN4/resources/app.asar/vingester-browser-worker.html

[rse]

Ah, ok. I see. The build-time warning indicated that you have a Linux system which does NOT match x64/glibc-2.33, because no pre-built binary was found. The run-time error in the second case indicates that the binary I have built under my Linux system does also not match your Linux system and hence it fails. The missing preview then is just a subsequence side-effect. So, we have to find out why the pre-build binaries do not match and break. According to your initial information, that's a latest Fedora 34/x64 system, right?

I've no such system currently available, so first have to setup it in a VM in order to better understand the particular problem. But in general, it is clear that a pre-built Linux binary can easily crash on certain Linux systems, as there are just too many small differences between the Linux systems. I built under Ubunti 20/04, you are under Fedora 34. This seems to be already far away. But good that we already figured out that the preview problem is actually just a side-effect or a completely different problem. Let's see what we can do...

[me-vlad]

One more side effect I met on linux with opus version mismatch - not working headless ffmpeg output.
I'll rename this ticket to reflect more common problem.

[rse]

Let me guess: the FFmpeg does not work for the same reason: the FFmpeg binary (built under Debian) does not run under this latest Fedora or at least crashes there.

[me-vlad]

I'm pretty sure that prebult opus for glibc-2.27 will work on most linux distros from debian stable to latest Ubuntu or Fedora because older glibc is compatible with newer versions (not vice versa).
I make a quick and dirty test: downloaded pre built opus for my node version and glibc-2.27 (opus-v0.5.3-node-v83-napi-v3-linux-x64-glibc-2.27.tar.gz) and replaced opus.node files in my custom build

node_modules/@discordjs/opus/prebuild/electron-v13.1-napi-v3-linux-x64-glibc-2.33/opus.node
node_modules/@discordjs/opus/prebuild/node-v83-napi-v3-linux-x64-glibc-2.33/opus.node

After npm start - preview and ffmpeg output works.

[me-vlad]

Let me guess: the FFmpeg does not work for the same reason: the FFmpeg binary (built under Debian) does not run under this latest Fedora or at least crashes there.

As I see Vingester uses static ffmpeg from https://johnvansickle.com/ffmpeg/
This version works on my system without craches - just checked with test srt output

./ffmpeg \
    -hide_banner \
    -loglevel info \
    -threads 0 \ 
    -f lavfi -i "smptebars=size=1920x1080:rate=25" \
    -f lavfi -i "sine=1000" \
    -filter_complex "fps=25,format=yuv420p,setsar=sar=1/1" \
    -strict -2 \
    -vsync -1 \
    -async 44100 \
    -r:v 25/1 \
    -c:v libx264 -preset:v veryfast -profile:v main -level:v 4.1 \
    -coder cabac \
    -refs 1 -bf 2 -keyint_min 25 -g:v 50 -flags +cgop -thread_type frame -trellis 0 \ 
    -b:v 3000k -minrate:v 3000k -bufsize:v 3000k -maxrate:v 3000k \
    -force_fps \
    -c:a aac \
    -b:a 128k -minrate:a 128k -maxrate:a 128k \
    -ar 44100 \
    -ac 2 \ 
    -f mpegts \
    "srt://127.0.0.1?pkt_size=1316"

[rse]

Then the FFmpeg problem is also a side-effect of the @discord/opus trouble

[rse]

But we cannot control what pre-built binary electron-builder fetches during the building of the Vingester binary on Ubuntu 20/04...

[me-vlad]

But we cannot control what pre-built binary electron-builder fetches during the building of the Vingester binary on Ubuntu 20/04...

Sure. Its a dirty hack, but for public release build process you can somehow "hardcode" prebuilt opus for your node version and glibc-2.27
opus-v${OPUS_VERSION}-node-v93-napi-v3-linux-x64-glibc-2.27.tar.gz

Any dinamically linked lib on different linux distros/versions can be showstopper.

For most glibc based distros it will be working sulution.

Not sure about musl libc based distros (https://wiki.musl-libc.org/projects-using-musl.html) but IMHO musl is more "exotic" case.

[rse]

Interesting: just a few hours ago Discord released version 0.5.3 of their NPM module, including new pre-build binaries.

But in order to understand the situation: you have a Fedora Linux 34 and it actually HAS a glibc 2.33. Nevertheless the binary bundled by Vingester crashes. But the glibc 2.27 version worked? Strange...

So, your recommendation is that Vingester replaces the glibc 2.33 binary with the glibc 2.27 one as this way Vingester has more chance to work, even under newer Linux versions (based on glibc 2.33), right?

[me-vlad]

Let me guess: the FFmpeg does not work for the same reason: the FFmpeg binary (built under Debian) does not run under this latest Fedora or at least crashes there.

A propos I see a comment about static ffmpeg on Ubuntu

            /*  NASTY WORKAROUND: on some Linux platforms (e.g. Ubuntu 20.10) the statically built
                ffmpeg(1) executable (built under Debian AFAIK) unfortunately segfaults, so at
                least once try to use an externally installed "native" ffmpeg(1) of the system  */

It's pretty strange. These crashes observed on a "real" Ubuntu OS or WSL/WSL2 ?

[rse]

The crashes were under non-WSL systems and with my other Electron application LiVE Receiver (but it uses FFmpeg the same way than Vingester). Mint Linux and Ubuntu Linux were those problematic platforms, as far as I can remember.

[rse]

Before I hack in the glibc-2.27 workaround, can you recheck the situation with the glibc-2.33 binaries for the just released @discord/opus 0.5.3? Not that we apply a downgrade workaround, although the newer pre-build binaries would work now.

[me-vlad]

But in order to understand the situation: you have a Fedora Linux 34 and it actually HAS a glibc 2.33. Nevertheless the binary bundled by Vingester crashes. But the glibc 2.27 version worked? Strange...

Yes, Fedora 34 ships with glibc 2.33. I'll try to explain the situation

1. Vingester app built on Ubuntu includes opus for glibc-2.31

ls -l /tmp/.mount_VingesHRAXAi/resources/app.asar.unpacked/node_modules/@discordjs/opus/prebuild/
total 0
drwxr-xr-x 2 root root 0 Jun 10 00:34 electron-v13.1-napi-v3-linux-x64-glibc-2.31
drwxr-xr-x 2 root root 0 Jun 10 00:34 node-v93-napi-v3-linux-x64-glibc-2.31

2. node_modules/@discordjs/opus/lib/index.js in Vingester app tries to find opus version for my actual glibc 2.33 ("module_path": "./prebuild/{node_abi}-napi-v{napi_build_version}-{platform}-{arch}-{libc}-{libc_version}/") and fails, as we see in log messages:

[2021-06-10 21:49:13.549] [error] (main) browser/worker-E24446D2: console: Uncaught Error: Cannot find module '/tmp/.mount_VingeshQ8eN4/resources/app.asar/node_modules/@discordjs/opus/prebuild/electron-v13.1-napi-v3-linux-x64-glibc-2.33/opus.node' Require stack: - /tmp/.mount_VingeshQ8eN4/resources/app.asar/node_modules/@discordjs/opus/lib/index.js - /tmp/.mount_VingeshQ8eN4/resources/app.asar/vingester-browser-worker.html

So, your recommendation is that Vingester replaces the glibc 2.33 binary with the glibc 2.27 one as this way Vingester has more chance to work, even under newer Linux versions (based on glibc 2.33), right?

I see as a possible solution:

1. fix @discordjs/opus to use packaged with application library version and not try to dynamically determine version on a user system because build OS and user OS can use different glibc versions.
2. as a helper step for wider compatibility - build Vingester with opus for glibc 2.27 (Ubuntu 18.04 LTS)

[me-vlad]

Before I hack in the glibc-2.27 workaround, can you recheck the situation with the glibc-2.33 binaries for the just released @discord/opus 0.5.3? Not that we apply a downgrade workaround, although the newer pre-build binaries would work now.

cloned 77b9642 built on my machine works (preview, frameless out, headless ffmpeg output to mkv/m4v/srt).

Can you share for tests 2.2.7 beta compiled on your build host?

[me-vlad]

The crashes were under non-WSL systems and with my other Electron application LiVE Receiver (but it uses FFmpeg the same way than Vingester). Mint Linux and Ubuntu Linux were those problematic platforms, as far as I can remember.

Just installed and updated Ubuntu 20.10 x86_64, ffmpeg 4.4 static amd64 from https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
Ffmpeg from CLI works without crashes.
May be some missing codecs/protocols or wrong binary architecture (arm64 instead amd64) was a problem?
If you get me more info about used ffmpeg options I'll try to figure out ffmpeg issue on Ubuntu.

[me-vlad]

JFYI - as a workaround on my linux machines I building Vingester from sources on target system and it works fine for me.

[rse]

Yes, that's the usual trouble under Linux all the time: the differences in the ABIs between different Linux distros are large enough that pre-built binaries can fail. It's the trouble with FFmpeg and here again with the Node module @discordjs/opus. I've no solution at hand beside the possibility to provide N different Vingester binaries for lots of Linux distros.