Multi Planar device support #233

argakon · 2023-10-05T19:15:57Z

Some devices doesn't have V4L2_BUF_TYPE_VIDEO_CAPTURE support, but they can be a multi planar devices and have a bit different api (V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE).
I have one of this devices NanoPC-T6 based on RK3588 SoC. One of the cool features of this device is HDMI IN support.
Also Orange Pi 5 Plus, Radxa Rock 5B and other SBC on RK3588 should work with it too.

v4l2-ctl --all output:

Driver Info:
	Driver name      : rk_hdmirx
	Card type        : rk_hdmirx
	Bus info         : fdee0000.hdmirx-controller
	Driver version   : 5.10.160
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Priority: 2
DV timings:
	Active width: 1920
	Active height: 1080
	Total width: 2640
	Total height: 1125
	Frame format: progressive
	Polarities: -vsync -hsync
	Pixelclock: 148492000 Hz (50.00 frames per second)
	Horizontal frontporch: 528
	Horizontal sync: 44
	Horizontal backporch: 148
	Vertical frontporch: 4
	Vertical sync: 5
	Vertical backporch: 36
	Standards: 
	Flags: 
DV timings capabilities:
	Minimum Width: 640
	Maximum Width: 4096
	Minimum Height: 480
	Maximum Height: 2160
	Minimum PClock: 20000000
	Maximum PClock: 600000000
	Standards: CTA-861
	Capabilities: Interlaced, Progressive
Format Video Capture Multiplanar:
	Width/Height      : 1920/1080
	Pixel Format      : 'BGR3' (24-bit BGR 8-8-8)
	Field             : None
	Number of planes  : 1
	Flags             : premultiplied-alpha, 0x000000fe
	Colorspace        : sRGB
	Transfer Function : Unknown (0x000000b8)
	YCbCr/HSV Encoding: Unknown (0x000000ff)
	Quantization      : Limited Range
	Plane 0           :
	   Bytes per Line : 5760
	   Size Image     : 6220800

Digital Video Controls

                  power_present 0x00a00964 (bitmask): max=0x00000001 default=0x00000000 value=1 flags=read-only

v4l2-ctl --list-formats-ext -d /dev/video0 output:

ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'BGR3' (24-bit BGR 8-8-8)
	[1]: 'NV24' (Y/CbCr 4:4:4)
	[2]: 'NV16' (Y/CbCr 4:2:2)
	[3]: 'NV12' (Y/CbCr 4:2:0)

Device has 4 pixel formats: NV24, NV16, NV12, BGR3 (BGR24). By default it works in NV24 mode. Device has no any controls, you can't set pixel format or resolution or fps.
To set device in BGR3 pixel format you have to use EDID.
I tested it with this one EDID:

wget https://raw.githubusercontent.com/pikvm/kvmd/master/configs/kvmd/edid/v3-hdmi.hex -O /edid.hex
v4l2-ctl --device=/dev/video0 --set-edid=file=/edid.hex --fix-edid-checksums --info-edid

Lines above turn on BGR3 pixel format, which is much easier to work and convert.

Command:

./ustreamer -m BGR24 -f 30

- INFO  [339862.490      main] -- Starting PiKVM uStreamer 5.43 ...
-- INFO  [339862.490      main] -- Using internal blank placeholder
-- INFO  [339862.491      main] -- Listening HTTP on [127.0.0.1]:8080
-- INFO  [339862.491    stream] -- Using V4L2 device: /dev/video0
-- INFO  [339862.491    stream] -- Using desired FPS: 30
-- INFO  [339862.491      http] -- Starting HTTP eventloop ...
================================================================================
-- INFO  [339862.491    stream] -- Device fd=8 opened
-- ERROR [339862.491    stream] -- Requested resolution=640x480 is unavailable
-- INFO  [339862.491    stream] -- Using resolution: 1920x1080, size:6220800
-- INFO  [339862.491    stream] -- Using format: BGR24
-- INFO  [339862.491    stream] -- Querying HW FPS changing is not supported
-- INFO  [339862.491    stream] -- Using IO method: MMAP
-- INFO  [339862.500    stream] -- Requested 5 device buffers, got 5
-- INFO  [339862.500    stream] -- Capturing started
-- INFO  [339862.500    stream] -- Using JPEG quality: 80%
-- INFO  [339862.500    stream] -- Creating pool JPEG with 4 workers ...
-- INFO  [339862.500    stream] -- Capturing ...

Environment:

rk3588-XYZ-ubuntu-jammy-desktop-5.10-arm64-YYYYMMDD.img.gz | jammy | Ubuntu 22.04 with GNOME and Wayland with recommended software | 5.10.y

pi@NanoPC-T6:~$ uname -a
Linux NanoPC-T6 5.10.160 #194 SMP Thu Sep 7 15:19:51 CST 2023 aarch64 aarch64 aarch64 GNU/Linux
pi@NanoPC-T6:~$ cat /etc/debian_version 
bookworm/sid

Probably some devices can have multiple planes, with different pixel format(for example), i can't test it, so i hardcode plane count to 1 (e.g. index=0).

I am not a C/C++ developer, so let me know about mistakes.

src/ustreamer/device.c

chraac · 2023-10-07T14:11:49Z

I've a RK3399 board which require multi-planar support too
v4l2-ctl --all output:

Driver Info:
        Driver name      : rkisp1_v0
        Card type        : rkisp1_mainpath
        Bus info         : platform:ff910000.rkisp1
        Driver version   : 4.19.193
        Capabilities     : 0x84201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : rkisp1
        Model            : rkisp1
        Serial           :
        Bus info         :
        Media version    : 4.19.193
        Hardware revision: 0x00000000 (0)
        Driver version   : 4.19.193
Interface Info:
        ID               : 0x03000007
        Type             : V4L Video
Entity Info:
        ID               : 0x00000006 (6)
        Name             : rkisp1_mainpath
        Function         : V4L2 I/O
        Pad 0x01000009   : 0: Sink
          Link 0x02000029: from remote pad 0x1000004 of entity 'rkisp1-isp-subdev': Data, Enabled
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 1920/1080
        Pixel Format      : 'YUYV' (YUYV 4:2:2)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Default
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Full Range
        Plane 0           :
           Bytes per Line : 3840
           Size Image     : 4147200
Selection Video Capture: crop, Left 0, Top 0, Width 1920, Height 1080, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags:
Selection Video Output: crop, Left 0, Top 0, Width 1920, Height 1080, Flags:
Selection Video Output: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags:

User Controls

            audio_sampling_rate 0x00981980 (int)    : min=0 max=768000 step=1 default=0 value=0 flags=read-only, volatile
                  audio_present 0x00981981 (bool)   : default=0 value=0 flags=read-only

Image Source Controls

              vertical_blanking 0x009e0901 (int)    : min=1 max=8 step=1 default=4 value=4

Image Processing Controls

                 link_frequency 0x009f0901 (intmenu): min=0 max=0 default=0 value=0
                                0: 310000000 (0x127a3980)
                     pixel_rate 0x009f0902 (int64)  : min=0 max=310000000 step=1 default=310000000 value=310000000 flags=read-only

Digital Video Controls

v4l2-ctl --list-formats-ext -d /dev/video0 output:

ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [1]: '422P' (Planar YUV 4:2:2)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [2]: 'NV16' (Y/CbCr 4:2:2)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [3]: 'NV61' (Y/CrCb 4:2:2)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [4]: 'YM16' (Planar YUV 4:2:2 (N-C))
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [5]: 'NV21' (Y/CrCb 4:2:0)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [6]: 'NV12' (Y/CbCr 4:2:0)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [7]: 'NM21' (Y/CrCb 4:2:0 (N-C))
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [8]: 'NM12' (Y/CbCr 4:2:0 (N-C))
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [9]: 'YU12' (Planar YUV 4:2:0)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [10]: 'YM24' (Planar YUV 4:4:4 (N-C))
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [11]: 'RGGB' (8-bit Bayer RGRG/GBGB)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [12]: 'GRBG' (8-bit Bayer GRGR/BGBG)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [13]: 'GBRG' (8-bit Bayer GBGB/RGRG)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [14]: 'BA81' (8-bit Bayer BGBG/GRGR)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [15]: 'RG10' (10-bit Bayer RGRG/GBGB)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [16]: 'BA10' (10-bit Bayer GRGR/BGBG)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [17]: 'GB10' (10-bit Bayer GBGB/RGRG)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [18]: 'BG10' (10-bit Bayer BGBG/GRGR)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [19]: 'RG12' (12-bit Bayer RGRG/GBGB)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [20]: 'BA12' (12-bit Bayer GRGR/BGBG)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [21]: 'GB12' (12-bit Bayer GBGB/RGRG)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8
        [22]: 'BG12' (12-bit Bayer BGBG/GRGR)
                Size: Stepwise 32x16 - 1920x1080 with step 8/8

Test this branch locally, it runs well!

src/ustreamer/device.c

argakon · 2023-10-07T15:29:53Z

I've a RK3399 board which require multi-planar support too

Could you confirm this PR works for your SBC?

chraac · 2023-10-07T15:41:15Z

I've a RK3399 board which require multi-planar support too

Could you confirm this PR works for your SBC?

yeah, it works, but latency little bit higher in 1280x720@30fps, and laggy in higher resolution, thought it's cause by the software MJPG compression

argakon · 2023-10-07T15:45:18Z

I've a RK3399 board which require multi-planar support too

Could you confirm this PR works for your SBC?

yeah, it works, but latency little bit higher in 1280x720@30fps, and laggy in higher resolution, thought it's cause by the software MJPG compression

good! on my hardware the latency is ok and yes it can be laggy for high resolutions, but i guess it depends on the hardware too, i have 3 cpu cores busy for 1920x1080x60.
I'am investigating mpp right now for H264/H265/MJPEG hw encoding, for future updates.

chraac · 2023-10-07T16:06:46Z

I've a RK3399 board which require multi-planar support too

Could you confirm this PR works for your SBC?

yeah, it works, but latency little bit higher in 1280x720@30fps, and laggy in higher resolution, thought it's cause by the software MJPG compression

good! on my hardware the latency is ok and yes it can be laggy for high resolutions, but i guess it depends on the hardware too, i have 3 cpu cores busy for 1920x1080x60. I'am investigating mpp right now for H264/H265/MJPEG hw encoding, for future updates.

nice work! let me know if need any help, have some experience in the video processing pipeline (capturing/encoding/decoding/rendering)

mdevaev · 2023-10-07T16:16:35Z

Ping me when it will be ready.

argakon · 2023-10-08T11:13:04Z

@mdevaev i think we are good

src/ustreamer/device.c

Signed-off-by: Artem Mamonov <artyom.mamonov@gmail.com>

mdevaev · 2023-10-08T16:20:48Z

I'll merge it to a separate branch and review again.

mdevaev · 2023-10-10T17:18:43Z

Okay, I made some fixes and merged it to master. Please test it.

argakon · 2023-10-12T21:15:46Z

Okay, I made some fixes and merged it to master. Please test it.

works as expected.

Eric-168 · 2023-11-17T14:03:48Z

Okay, I made some fixes and merged it to master. Please test it.

works as expected.

Looking forward to support for H264

argakon added 2 commits October 5, 2023 18:25

added multi planar device support (RK3588 HDMI IN)

d1d4e0d

sync with upstream version

462b707

mdevaev reviewed Oct 5, 2023

View reviewed changes