Video4Linux and Genicam

[bhack]

Is there any planned to support Video4Linux and Genicam cameras?

[bhack]

Another alternative for Genicam is https://github.com/genicam/harvesters

[yongtang]

@bhack It looks like Video4Linux is GPL licensed. Not sure we could like it easily. For Genicam it looks like it is LGPL licensed. Might be possible for adding the support. Will spend some time to see if this is feasible or not.

/cc @ivelin in case you are interested as well.

[bhack]

Some python wrappers seems to me LGPL https://github.com/duanhongyi/pyv4l2 cause I think the header Is /* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */ /*
Check
https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/videodev.html#videodev2-h

[bhack]

Another cross platform alternative for USB camera was the libuvc python wrappers but I don't know if its Is currently maintained upstream https://github.com/pupil-labs/pyuvc

[bhack]

Another alternative for USB cams was to use ffmpeg directly https://trac.ffmpeg.org/wiki/Capture/Webcam

But then you will not control too much camera parameters.

[yongtang]

Thanks @bhack, that looks promising. I will take a look.

[ivelin]

@yongtang , thank you for the tag. At a first glance it is at a little bit lower level than my current plane of interest. I'm mostly working with cameras and other audio/video sources available over IP: RTP, RTSP , HTTP, RTMP, WebRTC. I will still keep an eye on this thread for intersections.

Just as a reference, here is a technical blog on my recent activities if anyone on this thread is interested.
https://webrtchacks.com/private-home-surveillance-with-the-webrtc-datachannel/

My experiments continue to show that there is quite a bit of friction in the transformation of media from existing IP wire formats to TF graphs. I remain interested in more direct ways to ingest media into TFIO, but it's not obvious how that can be done without architecting novel models that understand the time dimension and are not only based on individual image frames.

[yongtang]

@ivelin With respect to WebRTC, I think it might be much easier to build a custom ops in C++ to read WebRTC into tf graph. This could be more useful in certain situations (than low level RTP/etc).

[yongtang]

@bhack In order to use Video4Linux it looks like a USB camera might be needed. I don't have a USB camera or a Linux on hands, so it may take some more time.

I labeled the issue with help wanted in case anyone is interested in this topic and want to work together to bring Video4Linux/Video Capture into tensorflow-io.

In the mean time, I played with Video Camera on macOS, and is managed to build a dataset input in PR #821 . The PR #821 uses AVFoundations' Swift API, and then exported to C (cdicl) so that video capture's raw image is available as a Dataset, potentially usable by tf.keras.

At the moment, the raw image is in native NV12 (kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange in macOS) so additional conversion is needed to convert the raw image into RGB format. This will complete the whole pipeline for tf.data+tf.keras.

The conversion from NV12 => RGB is not in place yet, but I am looking into it.

In the mean time, once a pipeline is in place, it will be possible for us to port the pipeline on Linux with Video4Linux I think (when usb camera is available).

[bhack]

You don't need a camera. You can just run a Debian/Ubuntu image on with Docker and use a loopback device installing https://packages.debian.org/search?keywords=v4l2loopback

Edit:
On Docker for Mac you need to compile v4l2loopback model in linuxkit. See https://github.com/linuxkit/linuxkit/blob/master/docs/kernels.md#compiling-external-kernel-modules

[bhack]

Generally from "live" Video source you need to rectify/undistort frames.
OSX avfoundation has cameracalibration classes:
https://developer.apple.com/documentation/avfoundation/avcameracalibrationdata

What we could do for other cameras? Are there some helper ops that we could recover from https://github.com/tensorflow/graphics? What do you think?

[yongtang]

@bhack The PR #821 has been updated and now both macOS (through AVFoundation) and Linux (through Video4Linux) are supported.

Note the raw format is NV12 on macOS and YUYV on Linux (at least from the loopback device I tested).

[yongtang]

@bhack Once PR #821 is merged, I think decode yuyv and nv12 will be the next step before we do any calibration.

[bhack]

On TF/Graphics they have started a stub color conversion namespace but who know if they talked with TF.image team?
tensorflow/graphics#107

They had also request to expand over undistort:
tensorflow/graphics#173

[yongtang]

@bhack Making a change to other different repos might be a challenge, as it involved multiple teams, and tensorflow-io's maintainers are mostly outside of google. tensorflow-io also has image related APIs like decode_tiff/decode_webp/encode_bmp which are not available in tf.image. I think one solution is to have tensorflow-io have a consistent public facing namespace, and reuse APIs either tf.image or tf.graphics internally to allow a consistent ux for user.

[bhack]

/cc @julienvalentin

[yongtang]

@bhack PR #821 has been merged with video capture supported on Linux and macOS (Windows is not touched). Opened another issue #825 to track the progress on convert from NV12 and YUYV to RGB as the next step.

[bhack]

I think that at some point we could re-evaluate https://libcamera.org/