Research about video backend alternative.

[ltbam]

fmv

fmv, full-motion video

Installation | Motivation | Findings | Getting Up And Running | Examples | API | See Also

fmv is a proof of concept implementation for handling full-motion video data. See Section "Goals"
for information. Currently, the main features are:

• One application, that displays the video feed and KLV metadata to stdout using gstreamer and libklv for doing so. The application is named fmv.exe and can be found here. Needs to be run via the command line and will give you help if not params are specified.

• Another application, that shows the integration of gstreamer with a UI framework. In this case
  gtk+, but the same applies to QT and others. The application is named fmvgui.exe and can be found
  here. Needs to be run via the command line and will give you help if not params are specified.

Motivation

Currently, we are using QGISFMV in Kadas. The current
solution uses FFMPEG command line to stdout, Python, and multiple Threads for demuxing video, audio
, and KLV metadata from transport stream containers. It currently only works with TS files, since the
RTP mode has been disabled (see MGDIRE_SB-836). Furthermore, the following caveats exist with QGISFMV:

• Some video files will not play.

• Large files cause issues.

• RTP works, but the video is not fluid (the changes needed to get it working are documented
  in MGDIRE_SB-836).

• The solution assumes that certain video codecs are installed. Hence, making installations of the
  plugins are environment-dependent and not standalone.

• In RTP mode there is also certainly a large amount of latency between sensor and rendering given the technology used. Hence, we are not seeing stuff in real-time.

Given these caveats, this proof of concept was implemented to find a possible approach to
improvement and to become more sure about its feasibility. In summary, the approach of
improvement should:

1. All kinds of videos SHOULD play.

2. Large files SHOULD not be an issue.

3. RTP SHOULD be fully supported with no issues.

4. SHOULD not assume codecs to be installed.

5. In RTP mode should have minimal latency between sensor and rendering.

Findings

To fix the above caveats (see Section "Motivation"), at first the proposed enhancements
of MGDIRE_SB-836 were pursued, yet

• QMediaPlayer has no way to demux TS files and access the metadata. See comments in MGDIRE_SB-836
  for more information.

• VLC has no own TS demuxing support and is less useful as a library since there focus is on developing
  a desktop application.

• Using FFMEG as a library, in a more integral way than currently would be doable. Yet, since the API
  etc. are less stable and spread over multiple sub-projects it was not directly considered.

Finally, it was decided to give gstreamer a try. Since its supports demuxing of transport streams if the KLV metadata is asynchronously encoded and can be integrated into GUI frameworks such as QT. gstreamer is a pipeline-based multimedia framework that links together a wide variety of media processing systems to complete complex workflows. For instance, gstreamer can be used to build a system that reads files in one format, processes them,
and exports them in another. The formats and processes can be changed in a plug-and-play fashion
(see also here).

After studying the documentation of gstreamer it was decided to implement a solution to see how
one can access video, audio, and KLV meta and also how one could integrate gstreamer into a
GUI framework such as QT (see Section "Installation" for the two applications). The findings of
using gstreamer and studying its docs are:

1. All kinds of videos should generally be playable.

2. Large files seem not to be an issue.

3. RTP should be fully supported (see https://gstreamer.freedesktop.org/documentation/additional/rtp.html?gi-language=c).

4. Codecs do not have to be installed.

5. In RTP mode latency between sensor and rendering are surely minimal since all is in C.

• gstreamer is a great library for multimedia processing.

• gstreamer as multiple bindings and especially Python. This would give us the possibility to stay in Python for the Plugin. Note, the POC was only implemented in C to stay near to the docs and tutorials to gain
  time be not porting everything to Python.

While the issues found by using gstreamer and studying the docs are:

• Sync KLV metadata is not supported in the tsdemux plugin and would need to be implemented for our use case (see here and here).

• Integration of gstreamer into a GUI framework is doable. Yet, the documentation is somewhat old and unclear about how. At the start of the POC implementation, it was envisioned to have KLV
  metadata displayed in the GUI (e.g. fmv.exe), but it was given up on since it caused problems.
  Please note that this is detail and surely doable, -- it was just not considered a top priority
  in answering the question on how gstreamer plays with others.

Currently, it can be said that gstreamer would be a good candidate to re-implemented the QGISFMV on top of it. The solution would look like

FIXME

and contain the following work:

FIXME

Installation

See dist folder for standalone applications.

Getting Up and Running

1. Install Visual Studio 2019 with C++ development tools
2. python build.py: Task 2
3. python build.py: Task 7
4. Code and whatever using Visual Studio 2019
5. Use python build as needed.

Examples

fmv

Medien1.mp4

fmvgui

fmvgui.mp4

See Also

• libklv: A C library for parsing KLV meta data and decoding MISB ST0601.9 used
  in this project and in large based on the works of libklv (KLV handling) and
  libklv (decoding).

• QGISFMV: The current QGIS plugin for full-motion video. Under the hood, it uses the FFMPEG command line
  in multiple Python threads and currently have the caveats mentioned in Section "Goals".

[All4Gis]

Hi @ltbam , can you fix the links? for example this https://github.com/All4Gis/QGISFMV/discussions/dist/fmv.zip is broken