Problem with newer ffmpegs (interlace-mode)

[MaZderMind]

Newer ffmpegs set the fields in the mkv stream different, when the interlace-mode is unknown.
Older ffmpegs just the the mode to progressive which is the value voctomix expects. Newer versions set it to undefined (which means that each frame has its own interlaced-flag). Voctomix can't handle this format.

Setting -field_order 0 on ffmpeg forces ffmpeg to write the interlace-mode as progressive. We should check if older ffmpegs can live with this additional flag and extend all ffmpeg-based example-script with it.

Edit: Additionally voctocore should print out a helpful message when such a situation is encountered with a hint on how to change the sourcing ffmpeg script, to make it compatible again.

[Florob]

Actually, having done some more testing right now it doesn't appear -field_order 0 (or -field_order progressive) helps. Looking at the mkv headers it seems FlagInterlaced is just always unset, making it default to undetermined.

[MaZderMind]

Command to test different ffmpec commandlines:

~ ffmpeg -t 1 -y -nostdin \
   -f decklink \
   -i 'DeckLink Mini Recorder@10' \
   -c:v rawvideo -c:a pcm_s16le \
   -pix_fmt yuv420p \
   -f matroska \
   /tmp/foo.mkv;  mkvinfo /tmp/foo.mkv | grep -iC3 Interlace

According to the EMBL-Spec:

A flag to declare is the video is known to be progressive or interlaced and if applicable to declare details about the interlacement. (0: undetermined, 1: interlaced, 2: progressive)

[MaZderMind]

@Florob setting the -field_order progressive directly before -f matroska on my ffmpeg (its a custom build i don't know the exact corresbonding version) seems to help:

~ ffmpeg -t 1 -y -nostdin \
   -f decklink \
   -i 'DeckLink Mini Recorder@10' \
   -c:v rawvideo -c:a pcm_s16le \
   -pix_fmt yuv420p \
   -field_order progressive -f matroska \
   /tmp/foo.mkv;  mkvinfo /tmp/foo.mkv | grep -iC3 Interlace
ffmpeg version N-81660-g93e0410 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.2) 20160609
  configuration: --prefix=/usr/local --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-nonfree --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-decklink --extra-cflags=-I/usr/include/decklink-sdk/ --enable-libfdk_aac --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-openal --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libxvid --enable-libzvbi --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      55. 29.100 / 55. 29.100
  libavcodec     57. 55.101 / 57. 55.101
  libavformat    57. 49.100 / 57. 49.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 62.100 /  6. 62.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
[decklink @ 0xbff560] Found Decklink mode 1920 x 1080 with rate 25.00(i)
[decklink @ 0xbff560] Frame received (#1) - No input signal detected - Frames dropped 1
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, decklink, from 'DeckLink Mini Recorder@10':
  Duration: N/A, start: 0.000000, bitrate: 830976 kb/s
    Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
    Stream #0:1: Video: rawvideo (UYVY / 0x59565955), uyvy422, 1920x1080, 829440 kb/s, 25 tbr, 1000k tbn, 1000k tbc
[matroska @ 0xc05740] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
    Last message repeated 1 times
Output #0, matroska, to '/tmp/foo.mkv':
  Metadata:
    encoder         : Lavf57.49.100
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080, q=2-31, 200 kb/s, 25 fps, 1k tbn, 25 tbc
    Metadata:
      encoder         : Lavc57.55.101 rawvideo
    Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
      encoder         : Lavc57.55.101 pcm_s16le
Stream mapping:
  Stream #0:1 -> #0:0 (rawvideo (native) -> rawvideo (native))
  Stream #0:0 -> #0:1 (pcm_s16le (native) -> pcm_s16le (native))
frame=   25 fps=0.0 q=-0.0 Lsize=   76127kB time=00:00:01.00 bitrate=623633.5kbits/s speed=1.08x    
video:75938kB audio:188kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.002800%
|  + Video track
|   + Pixel width: 1920
|   + Pixel height: 1080
|   + Interlaced: 2
|   + Colour space: length 4, data: 0x49 0x34 0x32 0x30
| + A track
|  + Track number: 2 (track ID for mkvmerge & mkvextract: 1)
~ 

[Florob]

From what I've heard the behavior seems to depend on the input too. Adding the flag in exactly that place to our source-testvideo-as-*.sh scripts does not appear to work for me.
I think @derpeter had success sourcing from a DeckLink card even completely without that flag.

[a-tze]

Unfortunately, this depends on FFmpeg version. Some big changes have been going on. Just today was another commit that affected this, and passing interlaced flag to playout still seems impossible (but that's not related to this problem).
Similar to -field_order is -vf setfields=1 , maybe you try that out?

[MaZderMind]

It seems we need a more controlled approach to this. Maybe setting up multiple ffmpeg-versions, testing each with both options given would make a more informed decision possible

[MaZderMind]

I conducted a matrix-test on this issue. For ffmpeg 2.5, 2.6, 2.7, 2.8, 3.0, 3.1, 3.2, 3.3 and master I compiled a static binary based on the latest patch-level for the respective version, linked with the latest decklink sdk. You can download the binaries here.

For all of these binaries, I conducted the following tests:

• Test: Read 1080p25 from Decklink, no additional options (test_1080p25_default)
• Test: Read 1080i25 from Decklink, no additional options (test_1080i25_default)
• Test: Read 1080p25 from Decklink, field_order=progressive on output (test_1080p25_field_order_progressive_out)
• Test: Read 1080i25 from Decklink, field_order=progressive on output (test_1080i25_field_order_progressive_out)
• Test: Read 1080p25 from Decklink, field_order=progressive on input (test_1080p25_field_order_progressive_in)
• Test: Read 1080i25 from Decklink, field_order=progressive on input (test_1080i25_field_order_progressive_in)
• Test: Read 1080p25 from Decklink, field_order=progressive on input and output (test_1080p25_field_order_progressive_in_out)
• Test: Read 1080i25 from Decklink, field_order=progressive on input and output (test_1080i25_field_order_progressive_in_out)

You can find the build instructions and the actual command run for each of these tests here:
https://c3voc.mazdermind.de/permanent/voctomix-issue-137-ffmpeg-interlaced-mkv/test-procedure

The raw test-results can be found here:
https://c3voc.mazdermind.de/permanent/voctomix-issue-137-ffmpeg-interlaced-mkv/raw-results.zip

I structured and color-coded the results into an spreadsheet, which can be found here:
https://c3voc.mazdermind.de/permanent/voctomix-issue-137-ffmpeg-interlaced-mkv/results.ods

The results are quite interesting:

My observations are as followes:

• It seems, that up to ffmpeg 3.0, no interlacing-marker was set at all.
• 3.1 starting to set it to undetermined in the default config
• 3.2 and 3.3 changed by now defaulting to progressive
• master started to properly detect the decklink-format and setting the output flag accordingly

regarding -field_order progressive on the output

• using -field_order progressive on the output-side does not change anything up to ffmpeg 3.0 and it does not produce an error either
• 3.1, 3.2 and 3.3 set the output-flag to progressive when the field_order flag is present on the output-side
• master still takes the input mode and does ignore the field_order flag on the output-side

regarding -field_order progressive on the input

• master does however respect the field_order flag on the input-side
• all other versions ignore the field_order flag on the input side

My conclusion is therefore as follows:

• In order to support all currently relevant ffmpeg versions, the example scripts should add -field_order progressive on ~~~the input and~~~ the output side
• When voctomix encounters interlaced input, it should report a useful message which points to
  • this issue
  • adding -field_order progressive to ~~~the input and~~~ the output
  • a link to an example script

[Florob]

What does respecting -field_order progressive on the input side actually mean? I'd expect that to treat the input as progressive, even when it is actually interlaced. I'm not sure why that would be desirable, particularly because you say master also correctly detects the decklink format.

[MaZderMind]

It seems there is another issue playing a role in this. Testing on gstreamer 1.8.3 (the version packaged in xenial-updates), matroskademux seems to mis-interpret the EBML-Header value. For empiric observation I created mkv-files with each of the observed interlace-modes (absent, 0, 1 and 2): https://c3voc.mazdermind.de/permanent/voctomix-issue-137-ffmpeg-interlaced-mkv/all-mode-mkvs.zip

running all of these files against a simple gstreamer pipeline (see test-procudure) results in this output:

no-interlaced-field.mkv
video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1

interlaced-0.mkv
video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1

interlaced-1.mkv
video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1

interlaced-2.mkv
video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)25/1

So it seems the interlaced-field is treated as a boolean field, with absence or 0 being interpreted as no interlacing whereas 1 and 2 being interpreted as interlaced content. This is on par with the code found in matroska-demux.c: https://github.com/GStreamer/gst-plugins-good/blob/286df32/gst/matroska/matroska-demux.c#L676 (commit-id is most recent at time of writing).

This contradicts the Matroska Specification stating

FlagInterlaced A flag to declare is the video is known to be progressive or interlaced and if applicable to declare details about the interlacement. (0: undetermined, 1: interlaced, 2: progressive)

FieldOrder Declare the field ordering of the video. If FlagInterlaced is not set to 1, this Element MUST be ignored. (0: Progressive, 1: Interlaced with top field display first and top field stored first, 2: Undetermined field order, 6: Interlaced with bottom field displayed first and bottom field stored first, 9: Interlaced with bottom field displayed first and top field stored first, 14: Interlaced with top field displayed first and bottom field stored first)

From the code it looks like the FieldOrder Flag is totally ignored by the matroskademux

Edit: reported bug at gstreamer bugzilla: https://bugzilla.gnome.org/show_bug.cgi?id=787206

[MaZderMind]

@Florob at c3voc we run a very crude mode, where we feed interlaced video into voctomix, pretending it is progressive and deinterlacing whatever comes out of the other end. This mode of operation is not desirable but I have not been able to work enough with one of the Panasonic cameras to see if it can actually send 1080p25 via SDI, so for the time being we still run this crude hack. With ffmpeg master it will require us to fake the input being progressive, despite it being correctly identified as interlaced.

But you're right in the point that such a hack should not be in the voctomix example script. I modified my comment above to reflect that.

[MaZderMind]

Accepting that this is a bug in gstreamer which is present in all recent relases and given that we want voctomix to be compatible with the ffmpeg- and gstreamer-versions out there, I'd like to propose to add a work-around to voctomix.

Requiring a master-build of gstreamer or an old ffmpeg-Version is not a desirable option to me.

To be specific I would port the fix I made for the multiview-monitor to voctomix.
It would intercept the source negotiation and warn when an interlaced-marker is seen.

The warning should point out that voctomix was informed that the source-video is interlaced but this might actually be untrue, because there is a bug in the matroska-demuxer of gstreamer.
Therefore voctomix would fake the caps to look like they're progressive and warn the end-user that she or he is responsible to only feed progressive content to voctomix.

@Florob, @a-tze, could you live with that kind-a solution?

[a-tze]

I would disagree with "that she or he is responsible to only feed progressive content to voctomix" because it is our reality to tunnel interlaced content - which is perfectly fine. The warning text you propose could mislead users to thinking that they have to deinterlace content before feeding it into voctomix, which is not true. What voctomix actually needs is getting fed with content that is flagged as progressive.
Thinking about that and what voctomix actually does I think the warning should be more like "It seems you're feeding interlaced content, I will handle it as being progressive, please note that voctomix output will also be flagged as progressive while it may actually be interlaced". In those cases where the user explicitly flags the feed as progressive (e.g. with -field_oder) we can assume that (s)he knows what (s)he's doing.
The fix itself is fine for me.

[Florob]

I think a lot of this boils down to discussion we could also have on #106.
While it is the VOC's reality I'm personally rather unhappy with feeding interlaced content through voctomix. We have seen what is possibly the worst case result in the voctogui previews, with huge visible tearing. I'm honestly a bit baffled it doesn't show up more. Maybe the scaling factors we choose everywhere else are just more fortunate.
The builtin decklink sources actually have yadif in the chain (automatically bypassed on progressive inputs), and I've been using that setup for a while now.

That said, what we do want is loads of flexibility in handling/reinterpreting content. Simply because incorrectly flagged content is so common. I.e. even in the absence of the gstreamer bug I think we'd still want this.
I don't think blindly forcing caps to be progressive is a good idea though, I'd prefer this to be a per source option, because there might be better things to do.

Example:
The Panasonics will happily output 1080psF25. This is progressive, but the Decklinks will see it as interlaced, because that is the transport. Reinterpreting this as progressive is fine and we need this capability. However, my understanding is that, depending on the way the interlaced content is actually represented, a simple reinterpretation might not suffice. Ideally we'd have a way to use a simple weave deinterlacer which would gracefully handle cases like TFF vs BFF.
The joke here is, feeding from ffmpeg we might actually have to force the caps to interlaced, rather than progressive, so we can weave it (for the case where we got undetermined).

[MaZderMind]

@Florob In my mind the point of this Issue is to allow sourcing content with recent ffmpeg versions (whichever kind of content that is). The whole voctomix core is not capable of handling content marked as interlaced so at some point in the input chain something has to mark the content as progressive.

Ideally this would be ffmpeg as the root source, but the gstreamer bug does prevent this. Next in line is our TCPAVSource implementation. Following your preference I go forward and disallow sources which are marked as mixed or interlaces, printing a warning which is along the lines of

The video-stream received was marked as being of [interlaced|mixed (interlaced/progressive)] content. Voctomix can only handle content which is marked as being progressive (although it can handle interlaced content if you deinterlace the output correctly – but this is up to you).

Note that there is currently a bug in gstreamer which prevents correct detection of progressive content generated from ffmpeg 3.1+ – see #137

If you are sure that your source is correct, you can set the config-option [source.cam1] interpret-as-progressive=true to force voctomix to accept this source, despite it being marked [interlaced|mixed (interlaced/progressive)]

Setting this option would inject the capssetter, but this has to be a deliberate decision.

Regarding the decklinksources aready running yadif I opened #153 to update the example-scripts

[a-tze]

This will win the Pulitzer award for the most verbose warning message ;) Thumbs up for that one.

edit by @MaZderMind: moved discussion about deinterlacer in the decklink-sources to #154

[MaZderMind]

@a-tze I hope you don't mind that I moved your comment about the deinterlacer in the decklink-sources to #154 – i would like to keep this already rather long issue focused on the compatibility between ffmpeg and voctomix – the decklink sources are a totally different story.

[dnet]

What's missing for this to be implemented?

[MaZderMind]

Only Time, I guess.

[hayden-t]

I think this topic is affecting the demo's too, they wont run with the default config provided complaining about different cap settings

[MaZderMind]

@hayden-t most probably. It might be a good Idea to add the deinterlace = assume-progressive-Flag for the demo-config, too.

[hayden-t]

will try this, are we talking core config file ? i had some luck with mixed mode but this may be better.