GStreamer Inter*-Elements introduce a/v-synchronity by retimestamping Buffers

[MaZderMind]

The inter* Elements

Voctomix internally uses gstreamers Inter* Elements (intervideosink, intervideosrc, interaudiosink and interaudiosrc) to separate different GStreamer-Pipelines, each concerned with only a partial Task. This makes the Architecture more modular and easier to extend.

But the Inter* Elements are very simple elements. *sink and *src Elements allocate a shared surface which both reference. Incoming frames (I'll say frames for now but the same applies to audio-buffers) are stored in a pointer in the surface by the *sink elements.

When a Frame is requested from the *src Element (based on the pipelines' fps), it checks if there is a buffer and if not creates a black frame. In both cases the buffer's timestamp is discarded and a new one is calculated based on the number of frames seen and the framerate.

Source and Sink pipelines run in different threads. When the source or the sink is not perfectly in sync, for example because there's a congestion around cpu resources and one thread can not be resumed in time, a Black frame is injected into the sink pipeline.

This results in two unwanted problems:

1. there's a flicker in the output-stream
2. then the late frame is now processed, its timestamp is 1/25s of a second off

When this happens more then once, video drifts with regard to audio. The root cause of this is

1. there is no queue between sink & src. In a multi-threaded data-processing system a FIFO with blocking would be required to safely pass the thread boundary without loosing data
2. the timestamps of the source are not preserved. Depending on the jitteryness of the audio-source this alone might result in bad a/v sync and drift (even without CPU congestion)

The inter* elements additionally have some restrictions that are problematic to us ut we've worked around them. For example they only support a very limited set of color modes and only a single stereo audio-track.

The InterPipe* Elements

Some years ago (but quite a while after we first used voctomix), GstInterpipe appeared. At a first glance it correctly attacks all of the above mentioned problems. Furthermore it is color-mode agnostic and can handle multiple audio-tracks.

It seems that switching from the existing inter* elements would at least be worth a try.

There are two areas where I think special care is needed:

TCP-Sources

In most places we not actively use the "black frame generation"-feature of the inter* elements. Most places just start up and then generate a continous stream of video-frames. This includes the image- and the decklink-sources.

The only place we actively depend on the inter* element generating black frames is in the tcp-sources. The tcp-source-pipelines start up and terminate with the incoming tcp-connection.

The new interpipe-elements do not generate black frames on themselfs when the upstream, they instead block the downstream-pipeline until an upstream pipeline is producing frames. Therefore the TCP-Sources need to startup with a videotestsrc based pipeline until a connection is made.

When the connection arrives the testsrc-pipeline can safely be stopped and the tcp-based pipeline can be started. The interpipe-element should block the downstream pipeline in the meantime.

Multi-Audio Routing

The complete multi audio routing is currently constructed around the limitation of the inter* elements to only support one stereo stream. Every input is routed to one stereo-stream and the complete mixing is defined in stereo-streams with one interaudiosink/src pair per stereo-stream.

Although this could be replicated with the interpipe-elements and would probably also fix the a/v sync issues, a much more optimized structure could be realized with the interpipe-elements, because they intrinsicly support an unlimited number of audio-streams. In my opinion it would be okay to change the input/output channel layout but some of the features of the multi-audio routing should be preserved. This requires further discussion and planning.

Subtasks

Subtasks (might be worthwile to create actual issues for them)

• replace inter* elements with interpipe for image- and decklink-video inputs (which always provide a constant video-stream)
• replace inter* elements with interpipe fpr tcp-video inputs (which needs a video-/audiotestsrc to produce a video-stream when there is nothing connected)
• refactor multiaudio-routing to pass all specified audio-streams along a single interpipe (this might very well change the format of stream input/output)
• provide a debian-package of interpipe via the voc's debian repo
• talk to @h01ger about if we even can package interpipe for debian (license is LGPL 2.1)

[MaZderMind]

@fightling I documented my current state of knowledge around this problem complex. Let's talk about it whenever you find the time.

[fightling]

replace inter* elements with interpipe for image- and decklink-video inputs (which always provide a constant video-stream)

I did the first part at branch feature/interpipe.

I also merged my experimental/generate-dot-of-pipelines branch to watch what's happening with the pipelines. And so I currently use the script doc/generate_pipeline_png.sh from the experimental branch for testing the pipelines.

The result are currently three errors:

> doc/generate_pipeline_png.sh

[..]

    INFO Voctocore: Running GObject-MainLoop
   DEBUG Voctocore: 

====================== UP AN RUNNING ======================

   ERROR AVRawOutput[mix_out]: Received Error-Signal on Output-Pipeline
   DEBUG AVRawOutput[mix_out]: Error-Details: #4: gstbasesrc.c(3452): gst_base_src_start (): /GstPipeline:pipeline6/GstInterPipeSrc:interpipesrc17:
Failed to start
   ERROR AVRawOutput[stream-blanker_out]: Received Error-Signal on Output-Pipeline
   DEBUG AVRawOutput[stream-blanker_out]: Error-Details: #4: gstbasesrc.c(3452): gst_base_src_start (): /GstPipeline:pipeline8/GstInterPipeSrc:interpipesrc24:
Failed to start
   ERROR StreamBlanker: Received Error-Signal on Mixing-Pipeline
   DEBUG StreamBlanker: Error-Details: #4: gstbasesrc.c(3452): gst_base_src_start (): /GstPipeline:pipeline7/GstInterPipeSrc:interpipesrc19:
Failed to start

[fightling]

@MaZderMind

replace inter* elements with interpipe fpr tcp-video inputs (which needs a video-/audiotestsrc to produce a video-stream when there is nothing connected)

I don't get that point. I am assuming you mean the tcp-video inputs in voctocore/lib/sources/tcpavsource.py where I can't find any use of inter* elements within the pipe.

[Florob]

As an aside: While decklink sources are probably more reliable than TCP ones, they can still be restarted (we only expose this in the network protocol). It would be interesting to see how interpipe handles this. We could just remove the restart support though. Its main use-case is getting AV sync back, which should not happen with interpipe.

@fightling inter* elements are only instantiated in AVSource from sources/avsource.py. All sources derive from it. I.e. your branch already uses interpipe for all source types which it probably shouldn't.

[fightling]

@fightling inter* elements are only instantiated in AVSource from sources/avsource.py. All sources derive from it. I.e. your branch already uses interpipe for all source types which it probably shouldn't.

@Florob Please can you explain to me, what source types shouldn't use interpipe? As I understand I have replaced all inter* with interpipe and so replaced it in all internal inter pipe communication. Why shouldn't I do that? I still don't get it :/

[Florob]

@fightling The ticket explains the reason in the "TCP-Sources" section.
Short version: inter{video,audio}* generate black frames/silence when there is no incoming data. This happens when nothing is connected to a TCP source and we (ab)use it to avoid pipeline stalls. interpipe doesn't have this feature.

[MaZderMind]

In the sources we probably want to have a videotestsrc running and upon disconnection of the TCP Source we switch the source end of the interpipe to the videotestsrc. Similar, upon connection, we switch it back to the new tcp-source.

Beause everything runs under the same clock, this should not introduce any delay or desync.

[MaZderMind]

@fightling actually we dont need the tcp sources for this congress. They are used solely for Pause- and NoStream video (which can instead be replaced with an image source) and -audio (which is not really critical).

Decklink- and Image-Sources should be simpler to migrate.

[fightling]

thx @MaZderMind for the hints.

btw: I have added an option -g to voctocore on the feature/interpipe branch to activate gstreamer message logging to get behind the problems that occur. And it's working and offers some clue about what happens:

(I'm currently using a videotestsrc to get arround the TCP source problem)

   DEBUG AVRawOutput[cam1_mirror]: Launching Output-Pipeline
   DEBUG AVRawOutput[cam2_mirror]: Launching Output-Pipeline
   DEBUG AVRawOutput[cam3_mirror]: Launching Output-Pipeline
   DEBUG AVRawOutput[grabber_mirror]: Launching Output-Pipeline
   DEBUG AudioMix: Launching Mixing-Pipeline
   DEBUG VideoMix: Launching Mixing-Pipeline
   DEBUG AVRawOutput[mix_out]: Launching Output-Pipeline
   ERROR Gst: Can not add listener interpipesrc17 because our caps are not defined yet
 WARNING Gst: Could not add listener interpipesrc17 to node video_mix_out
 WARNING Gst: Listener is not in the connected listeners list
   ERROR Gst: Can not add listener interpipesrc17 because our caps are not defined yet
 WARNING Gst: Could not add listener interpipesrc17 to node video_mix_out
   ERROR Gst: Could not listen to node video_mix_out
 WARNING Gst: error: Failed to start
 WARNING Gst: Failed to start in push mode
 WARNING Gst: Failed to activate pad
   DEBUG StreamBlanker: Launching Mixing-Pipeline
   ERROR Gst: Can not add listener interpipesrc19 because our caps are not defined yet
 WARNING Gst: Could not add listener interpipesrc19 to node video_mix_stream-blanker
 WARNING Gst: Listener is not in the connected listeners list
   ERROR Gst: Can not add listener interpipesrc19 because our caps are not defined yet
 WARNING Gst: Could not add listener interpipesrc19 to node video_mix_stream-blanker
   ERROR Gst: Could not listen to node video_mix_stream-blanker
 WARNING Gst: error: Failed to start
 WARNING Gst: Failed to start in push mode
 WARNING Gst: Failed to activate pad
   DEBUG AVRawOutput[stream-blanker_out]: Launching Output-Pipeline
   ERROR Gst: Can not add listener interpipesrc24 because our caps are not defined yet
 WARNING Gst: Could not add listener interpipesrc24 to node video_stream-blanker_out
 WARNING Gst: Listener is not in the connected listeners list
   ERROR Gst: Can not add listener interpipesrc24 because our caps are not defined yet
 WARNING Gst: Could not add listener interpipesrc24 to node video_stream-blanker_out
   ERROR Gst: Could not listen to node video_stream-blanker_out
 WARNING Gst: error: Failed to start
 WARNING Gst: Failed to start in push mode
 WARNING Gst: Failed to activate pad
   DEBUG Voctocore: Creating ControlServer
   DEBUG ControlServer: Binding to Source-Socket on [::]:9999
   DEBUG ControlServer: Setting GObject io-watch on Socket
   DEBUG root: running Voctocore
    INFO Voctocore: Running GObject-MainLoop
   DEBUG Voctocore: 

====================== UP AN RUNNING ======================

   ERROR AVRawOutput[mix_out]: Received Error-Signal on Output-Pipeline
   DEBUG AVRawOutput[mix_out]: Error-Details: #4: gstbasesrc.c(3452): gst_base_src_start (): /GstPipeline:pipeline6/GstInterPipeSrc:interpipesrc17:
Failed to start
   ERROR AVRawOutput[stream-blanker_out]: Received Error-Signal on Output-Pipeline
   DEBUG AVRawOutput[stream-blanker_out]: Error-Details: #4: gstbasesrc.c(3452): gst_base_src_start (): /GstPipeline:pipeline8/GstInterPipeSrc:interpipesrc24:
Failed to start
   ERROR StreamBlanker: Received Error-Signal on Mixing-Pipeline
   DEBUG StreamBlanker: Error-Details: #4: gstbasesrc.c(3452): gst_base_src_start ():

Here are the affected pipelines to which the errors relate to:

interpipesrc17 in avrawoutput-mix_out pipeline:

interpipesrc19 in streamblanker pipeline

interpipesrc24 in avrawoutput-stream-blanker_out pipeline:

Any ideas?

[MaZderMind]

@fightling The interpipe-Elements try to do automatic caps negotiation between source and sink (see Caps Negotiation and Dynamic Switching but they can also be fixed to a well defined set of caps (see caps-Property in gst-inspect Output.

As we know which caps we have both on the source- and the sink-side it might help to set the caps-property on all interpipe-elements.

[fightling]

@MaZderMind I already moved all the caps we used within Gst.Caps to the interpipes. I thought this would be the best practice because negotiating the caps twice should not be necessary. You can see the caps within the interpipe elements in the pipeline drawings.

[danimo]

Note: There are now gst-interpipe packages for Debian Stretch: http://pkg.c3voc.de/pool/main/g/gst-interpipe/

[fightling]

I found out today that the compositor takes buffers from interpipesrc if it has format=time set.

videotestsrc is-live=true ! interpipesink name=video_background
videotestsrc is-live=true ! interpipesink name=video_cam1_mixer
videotestsrc is-live=true ! interpipesink name=video_cam2_mixer
videotestsrc is-live=true ! interpipesink name=video_cam3_mixer
videotestsrc is-live=true ! interpipesink name=video_grabber_mixer

compositor name=mix
! identity name=sig
! fpsdisplaysink

interpipesrc
    listen-to=video_background
    format=time
! mix.

interpipesrc
    listen-to=video_cam1_mixer
    format=time
! mix.

interpipesrc
    listen-to=video_cam2_mixer
    format=time
! mix.

interpipesrc
    listen-to=video_cam3_mixer
    format=time
! mix.

interpipesrc
    listen-to=video_grabber_mixer
    format=time
! mix.

The pipeline above is working and quite similar to our compositor.

Another thought of me is about the queues and caps. I think we don't need any queues near interpipesink or interpipesrc because these filters are managing their own buffers for synchronization.
I already had to remove most capability filters and parameters to get the pipe of voctocore to start.
But I still get no images in the voctogui.

I also think we don't need the streamblanker construct anymore if we use the dynamic link switching capabilites of interpipe. Then - as I understand - we need to run gstd in the background.

My current voctocore pipelines look like this:

audiomix:

avrawoutput-cam1_mirror:

avrawoutput-cam2_mirror:

avrawoutput-cam3_mirror:

avrawoutput-grabber_mirror:

avrawoutput-mix_out:

avrawoutput-stream-blanker_out:

avsource:

streamblanker:

videodisplay:

videomix:

Any further ideas?

[fightling]

I did a try to use just one single pipeline in voctocore to get rid of all the AV sync problems and the inter* elements.

For the code look at branch feature/onepipe.

I can see each channel and the mix in vocotogui but the videos are not playing.

any ideas?

Here is what the whole pipeline looks like now (I put queues everywhere to grease everything but it doesn't help):

The stream blanker is currently disabled.

[fightling]

I have successfully attached a fpsdisplaysink behind the compositor. So it seems, that the pipeline is working but maybe the connection to voctogui is the remaining problem.

[fightling]

When I remove audio caps from the client side the videos are playing.

[fightling]

@danimo is this solved in voctomix2? @MaZderMind shall we close this?

[MaZderMind]

@fightling As far as i read the communication, there are no inter-elements in voc2mix, so this is probably not applicable anymore.
As far as closing goes I would suggest a Process for phasing over from v1 to v2 as outlined in this Wiki-Page: https://github.com/voc/voctomix/wiki/Proposal-for-v1-to-v2-Phase-Over-process
As long as v1 is not yet practically replaced by v2, the Issue is still present and should stay open (although for this specific issue it is quite implausible that there'll be any work done to it)

[fightling]

there are no inter*-elements anymore in voc2mix