[BUG] rpicam-vid hangs when h264 codec is used #664
Comments
the0
changed the title
|
I can reporduce this as well with very low bitrate (< 100kbps) encoding. There does seem to be an error issued in the firmware logs: Interestingly, once this error message is repeated 6 times, the @6by9 any thoughts on this? |
|
One possible workaround is to use the libav backend for HW encoding. If I run something like rpicam-vid -v -t 0 --framerate 24 --width 720 --height 576 --bitrate 50kbps -n --denoise off -o /dev/null --codec libav --libav-format h264I still see the venc error log messages, but this does not lockup |
|
The encoder does have a known issue on shutdown if the last frame presented resulted in an abort due to rate control. It's down in the guts of the codec though. I don't recall any issue of the input buffers not being returned as there is the front end conversion from the input buffers to internal image pool and SAND format that decouples it. Aborted frames should result in a 0-length encoded frame with the appropriate timestamp being returned. Does rpicam-apps handle that combination correctly, or is it coupling return of frames to libcamera on getting the encoded frame out? |
|
It does seem to be related to how rpicam-apps drives the encoder, as the very same driver used via the libav backend seems to behave correctly. In my very brief debugging, it looks like calling |
|
@the0, did you try my suggestion of using |
Yes, I'm using "--codec libav --libav-video-codec h264_v4l2m2m --libav-format h264" without any problems now, thank you. However, I didn't manage to make "--libav-video-codec-opts 'preset=ultrafast;tune=zerolatency'" work. Neither "v4l2-ctl --list-ctrls-menu -d 11" nor "ffmpeg -h encoder=h264_v4l2m2m" mention any of these opts are available, is that correct? |
These options are only for the SW (libx264) codec. The hardware codec will simply ignore these settings. |
|
Is framerate option also being ignored by libav/h264_v4l2m2m? |
|
Framerate should be correct as far as the sensor will generate frames at the requested rate. However, elementary H.264 streams don't store timestamps, so have no real notion of farmerate in the file itself. |
|
Actually this looks like an FFmpeg issue, or possibly the interface to it/libav. There is the In the native rpicam-vid case, that is being set to the requested 24fps. In the FFmpeg case it is being left at the default of 30fps. |
|
I've not tested this, however: v4l2_m2m_enc is taking the frame rate from AVCodecContext * framerate.[num|den] libav_encoder.cpp sets a load of other parameters in AVCodecContext, but not that one |
|
That does get setup here: rpicam-apps/encoder/libav_encoder.cpp Line 26 in 70894c0 |
|
Missed that one. It shouldn't matter, but FFmpeg calls S_PARM after REQBUFS(vid-out-mplane) I think we're missing a commit in the V4L2 driver that pushes that setting down to the firmware. |
|
If I use the |
|
@naushir are there any workarounds to have hardware-accelerated 720x576@20 h264 capture at the moment? |
|
My earlier comment (#664 (comment)) should provide a workaround. Have you given it a try? |
|
I'm unable to capture at rates lower than 30 fps with libav/h264_v4l2m2m as discussed above. |
|
The framerate parameter will ensure the sensor frames at the correct rate. You can check this with the |
|
With libav/h264_v4l2m2m I'm getting empty pts file, but when I set framerate to 1 fps I saw it actually encodes at 1 fps. Thanks! |
6by9
added a commit
to 6by9/linux
that referenced
this issue
Apr 17, 2024
For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
|
raspberrypi/linux#6108 should fix libav not resulting with the correct frame rate on the stream. libav/h264_v4l2m2m doesn't support the --save-pts flag. |
|
Closing this now, fixed by using the libav workaround. |
6by9
added a commit
to 6by9/linux
that referenced
this issue
May 14, 2024
For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
pelwell
pushed a commit
to raspberrypi/linux
that referenced
this issue
May 14, 2024
For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
pelwell
pushed a commit
to raspberrypi/linux
that referenced
this issue
May 21, 2024
For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
pelwell
pushed a commit
to raspberrypi/linux
that referenced
this issue
May 24, 2024
For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
May 28, 2024
For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Nov 25, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Dec 7, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Dec 9, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Dec 10, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Dec 16, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Dec 16, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Dec 20, 2024
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 2, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 2, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 6, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 10, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 13, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 17, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 20, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Jan 24, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 3, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 3, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
6by9
added a commit
to 6by9/linux
that referenced
this issue
Feb 6, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. raspberrypi#4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (raspberrypi#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following raspberrypi#4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching raspberrypi#4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: vchiq-mmal: Add defines for mmal_es_format flags There is a flags field in struct mmal_es_format, but the defines for what the bits meant weren't included in the headers. For V4L2_PIX_FMT_NV12_COL128 support we need them, so add them in. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 10, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: vchiq-mmal: Add defines for mmal_es_format flags There is a flags field in struct mmal_es_format, but the defines for what the bits meant weren't included in the headers. For V4L2_PIX_FMT_NV12_COL128 support we need them, so add them in. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 10, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 10, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 17, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: vchiq-mmal: Add defines for mmal_es_format flags There is a flags field in struct mmal_es_format, but the defines for what the bits meant weren't included in the headers. For V4L2_PIX_FMT_NV12_COL128 support we need them, so add them in. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 17, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 17, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 19, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 24, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 24, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 24, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: vchiq-mmal: Add defines for mmal_es_format flags There is a flags field in struct mmal_es_format, but the defines for what the bits meant weren't included in the headers. For V4L2_PIX_FMT_NV12_COL128 support we need them, so add them in. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Feb 28, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
popcornmix
pushed a commit
to raspberrypi/linux
that referenced
this issue
Mar 3, 2025
This adds a V4L2 memory to memory device that wraps the MMAL video decode and video_encode components for H264 and MJPEG encode and decode, MPEG4, H263, and VP8 decode (and MPEG2 decode if the appropriate licence has been purchased). This patch squashes all the work done in developing the driver on the Raspberry Pi rpi-5.4.y kernel branch. Thanks to Kieran Bingham, Aman Gupta, Chen-Yu Tsai, and Marek Behún for their contributions. Please refer to the rpi-5.4.y branch for the full history. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Ensure OUTPUT timestamps are always forwarded The firmware by default tries to ensure that decoded frame timestamps always increment. This is counter to the V4L2 API which wants exactly the OUTPUT queue timestamps passed to the CAPTURE queue buffers. Disable the firmware option. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Add support for CID MPEG_HEADER_MODE Control V4L2_CID_MPEG_VIDEO_HEADER_MODE controls whether the encoder is meant to emit the header bytes as a separate packet or with the first encoded frame. Add support for it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04_services/codec: Clear last buf dequeued flag on START It appears that the V4L2 M2M framework requires the driver to manually call vb2_clear_last_buffer_dequeued on the CAPTURE queue during a V4L2_DEC_CMD_START. Add such a call. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/vc04-services/codec: Fix logical precedence issue Two issues identified with operator precedence in logical expressions. Fix them. #4040 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Switch to s32fract staging/bcm2835-codec: Add the unpacked (16bpp) raw formats Now that the firmware supports the unpacked (16bpp) variants of the MIPI raw formats, add the mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Log the number of excess supported formats When logging that the firmware has provided more supported formats than we had allocated storage for, log the number allocated and returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for pixel aspect ratio If the format is detected by the driver and a V4L2_EVENT_SOURCE_CHANGE event is generated, then pass on the pixel aspect ratio as well. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Implement additional g_selection calls for decode v4l_cropcap calls our vidioc_g_pixelaspect function to get the pixel aspect ratio, but also calls g_selection for V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT. Whilst it allows for vidioc_g_pixelaspect not to be implemented, it doesn't allow for either of the other two. Add in support for the additional selection targets. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add VC-1 support. Providing the relevant licence has been purchased, then Pi0-3 can decode VC-1. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Fix support for levels 4.1 and 4.2 The driver said it supported H264 levels 4.1 and 4.2, but was missing the V4L2 to MMAL mappings. Add in those mappings. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Set the colourspace appropriately for RGB formats Video decode supports YUV and RGB formats. YUV needs to report SMPTE170M or REC709 appropriately, whilst RGB should report SRGB. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Pass corrupt frame flag. MMAL has the flag MMAL_BUFFER_HEADER_FLAG_CORRUPTED but that wasn't being passed through, so add it. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not update crop from S_FMT after res change During decode, setting the CAPTURE queue format was setting the crop rectangle to the requested height before aligning up the format to cater for simple clients that weren't expecting to deal with cropping and the SELECTION API. This caused problems on some resolution change events if the client didn't also then use the selection API. Disable the crop update after a resolution change. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835: Allow compressed frames to set sizeimage (#4386) Allow the user to set sizeimage in TRY_FMT and S_FMT if the format flags have V4L2_FMT_FLAG_COMPRESSED set Signed-off-by: John Cox <jc@kynesim.co.uk> staging/bcm2835-codec: Change the default codec res to 32x32 In order to effectively guarantee that a V4L2_EVENT_SOURCE_CHANGE event occurs, adopt a default resolution of 32x32 so that it is incredibly unlikely to be decoding a stream of that resolution and therefore failing to note a "change" requiring the event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Add support for decoding interlaced streams The video decoder can support decoding interlaced streams, so add the required plumbing to signal this correctly. The encoder and ISP do NOT support interlaced data, so trying to configure an interlaced format on those nodes will be rejected. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Correct ENUM_FRAMESIZES stepsize to 2 Being YUV420 formats, the step size is always 2 to avoid part chroma subsampling. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Return buffers to QUEUED not ERROR state Should start_streaming fail, or buffers be queued during stop_streaming, they should be returned to the core as QUEUED and not (as currently) as ERROR. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Log MMAL flags in hex The flags is a bitmask, so it's far easier to interpret as hex data instead of decimal. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Allow custom specified strides/bytesperline. If the client provides a bytesperline value in try_fmt/s_fmt then validate it and correct if necessary. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835_codec: Add support for image_fx to deinterlace Adds another /dev/video node wrapping image_fx doing deinterlace. Co-developed-by: Dave Stevenson <dave.stevenson@raspberrypi.com> Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging/bcm2835-v4l2_codec: Fix for encode selection API Matches correct behaviour from DECODE and DEINTERLACE Signed-off-by: Dom Cobley <popcornmix@gmail.com> staging: bcm2835-codec: Allow decode res changed before STREAMON(CAPTURE) The V4L2 stateful video decoder API requires that you can STREAMON on only the OUTPUT queue, feed in buffers, and wait for the SOURCE_CHANGE event. This requires that we enable the MMAL output port at the same time as the input port, because the output port is the one that creates the SOURCE_CHANGED event. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Do not send buffers to the VPU unless streaming With video decode we now enable both input and output ports on the component. This means that buffers will get passed to the VPU earlier than desired if they are queued befoer STREAMON. Check that the queue is streaming before sending buffers to the VPU. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Format changed should trigger drain When a format changed event occurs, the spec says that it triggers an implicit drain, and that needs to be signalled via -EPIPE. For BCM2835, the format changed event happens at the point the format change occurs, so no further buffers exist from before the resolution changed point. We therefore signal the last buffer immediately. We don't have a V4L2 available to us at this point, so set the videobuf2 queue last_buffer_dequeued flag directly. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Signal the firmware to stop on all changes The firmware defaults to not stopping video decode if only the pixel aspect ratio or colourspace change. V4L2 requires us to stop decoding on any change, therefore tell the firmware of the desire for this alternate behaviour. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Queue flushed buffers instead of completing When a buffer is returned on a port that is disabled, return it to the videobuf2 QUEUED state instead of DONE which returns it to the client. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Correct flushing code for refcounting Completions don't reference count, so setting the completion on the first buffer returned and then not reinitialising it means that the flush function doesn't behave as intended. Signal the completion when the last buffer is returned. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Ensure all ctrls are set on streamon Currently the code was only setting some controls from bcm2835_codec_set_ctrls, but it's simpler to use v4l2_ctrl_handler_setup to avoid forgetting to adding new controls to the list. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H&V Flips to ISP The ISP can do H & V flips whilst resizing or converting the image, so expose that via V4L2_CID_[H|V]FLIP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-v4l2-codec: Remove advertised support of VP8 The support for this format by firmware is very limited and won't be faster than the arm. Signed-off-by: Dom Cobley <popcornmix@gmail.com> Pass V4L2_CID_MPEG_VIDEO_H264_MIN_QP/MAX_QP to bcm2835-v4l2-codec Following #4704. This is necessary to set up quantization for variable bitrate to avoid video flickering. staging/bcm2835-codec: bytesperline for YUV420/YVU420 needs to be 64 Matching #4419, the ISP block (which is also used on the input of the encoder, and output of the decoder) needs the base address of all planes to be aligned to multiples of 32. This includes the chroma planes of YUV420 and YVU420. If the height is only a multiple of 2 (not 4), then you get an odd number of lines in the second plane, which means the 3rd plane starts at a multiple of bytesperline/2. Set the minimum bytesperline alignment to 64 for those formats so that the plane alignment is always right. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging/bcm2835-codec: Allow a different stride alignment per role Deinterlace and decode aren't affected in the same way as encode and ISP by the alignment requirement on 3 plane YUV420. Decode would be affected, but it always aligns the height up to a macroblock, and uses the selection API to reflect that. Add in the facility to set the bytesperline alignment per role. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: codec: Add support for V4L2_PIX_FMT_RGBA32 format We already support V4L2_PIX_FMT_BGR32 which is the same thing with red and blue swapped, so it makes sense to include this variant as well. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> bcm2835-codec: Return empty buffers to the VPU instead of queueing to vbuf2 The encoder can skip frames totally should rate control overshoot the target bitrate too far. In this situation it generates an output buffer of length 0. V4L2 treats a buffer of length 0 as an end of stream flag, which is not appropriate in this case, therefore we can not return that buffer to the client. The driver was returning the buffer to videobuf2 in the QUEUED state, however that buffer was then not dequeued again, so the number of buffers was reduced each time this happened. In the pathological case of using GStreamer's videotestsrc in mode 1 for noise, this happens sufficiently frequently to totally stall the pipeline. If the port is still enabled then return the buffer straight back to the VPU rather than to videobuf2. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Add support for V4L2_PIX_FMT_NV12_COL128 V4L2_PIX_FMT_NV12_COL128 is supported by the ISP and the input of video_encode, output of video_decode, and both input and output of the ISP. Add in the plumbing to support the format on those ports. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set crop_height for compressed formats In particular for the encoder where the CAPTURE format dictates the parameters given to the codec we need to be able to set the value passed as the crop_height for the compressed format. There's no crop available for cropped modes, so always set crop_height to the requested height. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Set port format from s_selection s_selection allows the crop region of an uncompressed pixel format to be specified, but it wasn't passing the setting on to the firmware. Depending on call order this would potentially mean that the crop wasn't actioned. Set the port format on s_selection if we have a component created. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> bcm2835-codec: /dev/video31 as interface to image_encode JPEG encoder Signed-off-by: Maxim Devaev <mdevaev@gmail.com> bcm2835-v4l2-codec: support H.264 5.0 and 5.1 levels vc04_services: bcm2835-codec: Remove redundant role check vidioc_try_encoder_cmd checks the role, but the ioctl is disabled for any roles in which it is invalid. Remove the redundant check. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace ISP and deinterlace also need a mechanism for passing effectively an EOS through the pipeline to signal when all buffers have been processed. VIDIOC_ENCODER_CMD does exactly this for encoders, so reuse the same function for ISP and deinterlace. (VIDIOC_DECODER_CMD is slightly different in that it also passes details of when and how to stop, so is not as relevant). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Allow larger images through the ISP Whilst the codecs are restricted to 1920x1080 / 1080x1920, the ISP isn't, but the limits advertised via V4L2 was 1920x1920 for all roles. Increase the limit to 16k x 16k for the ISP. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> media: bcm2835-v4l2-codec: Enable selection ioctl for ISP The ISP cases do nothing. Remove the break that separates them from the deinterlace case so they now do the same as deinterlace. This enables simple width & height setting, but does not enable setting left and top coordinates. Signed-off-by: John Cox <jc@kynesim.co.uk> media: bcm2835-v4l2-codec: Add profile & level ctrls to decode In order to support discovery of what profile & levels are supported by stateful decoders implement the profile and level controls where they are defined by V4L2. Signed-off-by: John Cox <jc@kynesim.co.uk> vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl In adding the MPEG2/MPEG4/H264 level and profile controls to the decoder, they weren't declared as read-only, nor handlers added to bcm2835_codec_s_ctrl. That resulted in an error message "Invalid control" being logged every time v4l2_ctrl_handler_setup was called from bcm2835_codec_create_component. Define those controls as read only, and exit early from s_ctrl on read only controls. Fixes: "media: bcm2835-v4l2-codec: Add profile & level ctrls to decode" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835_codec: Set MPEG2_LEVEL control to READ_ONLY V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL was missed from "vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl" Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add V4L2_CID_MPEG_VIDEO_B_FRAMES control FFmpeg insists on trying to set V4L2_CID_MPEG_VIDEO_B_FRAMES to 0, and generates an error should it fail. As our encoder doesn't support B frames, add a stub handler for it to silence FFmpeg. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE For H264, V4L2_CID_MPEG_VIDEO_H264_I_PERIOD is meant to be the intra I-frame period, whilst V4L2_CID_MPEG_VIDEO_GOP_SIZE is the intra IDR frame period. The firmware encoder doesn't produce I-frames that aren't IDR as well, therefore V4L2_CID_MPEG_VIDEO_GOP_SIZE is technically the correct control, however users may have adopted V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. Add support for V4L2_CID_MPEG_VIDEO_GOP_SIZE controlling the encoder, and have VIDIOC_S_CTRL for V4L2_CID_MPEG_VIDEO_H264_I_PERIOD update the value for V4L2_CID_MPEG_VIDEO_GOP_SIZE (the reverse is not implemented). Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add missing alignment for V4L2_PIX_FMT_RGBA32 The patch adding image encode (JPEG) to the driver missed adding the alignment constraint for V4L2_PIX_FMT_RGBA32, which meant it ended up giving a stride and size of 0. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Downgrade the level for a debug message The debug message from bcm2835_codec_buf_prepare when the buffer size is incorrect can be a little spammy if the application isn't careful on how it drives it, therefore drop the priority of the message. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: bcm2835-codec: Correct alignment requirements for YUYV The firmware wants the YUYV format stride alignment to be to a multiple of 32pixels / 64 bytes. The kernel driver was configuring it to a multiple of 16 pixels / 32 bytes, which then failed when it tried starting to stream. Correct the alignment requirements. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Fix up for 6.8 - use ignore_cap_streaming Drops downstream patch to v4l2_mem2mem, and uses the new mainline flag to achieve the same functionality Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: 32bpp RGB formats need a 64byte alignment The firmware needs 16 pixel alignment on RGBx 32bpp formats, which would be 64 byte. The driver was only setting 32byte alignment. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835_codec: Pass framerate to the component if set late For video encoding, if the framerate was set after the component was created, then it wasn't set correctly on the port, and an old value was encoded in the bitstream. Update the port status when the framerate is set. raspberrypi/rpicam-apps#664 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: vc04_services: bcm2835-codec: Drop include Makefile directive Drop the include directive. They can break the build, when one only wants to build a subdirectory. Replace with "../" for the includes in the bcm2835-v4l2-codec instead. The fix is equivalent to the four patches between 29d49a7 ("staging: vc04_services: bcm2835-audio: Drop include Makefile directive")...2529ca2 ("staging: vc04_services: interface: Drop include Makefile directive") Fixes: afaec52 ("staging: vc04_services: Add a V4L2 M2M codec driver") Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-v4l2-codec: Register with vchiq_bus_type Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: vc04_services: bcm2835-codec: Explicitly set DMA mask The platform model originally handled the DMA mask. Now that we are on the vchiq_bus we need to explicitly set this. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com> staging: bcm2835-codec: Disable HEADER_ON_OPEN for video encode Video encode can defer generating the header until the first frame is presented, which allows it to take the colourspace information from the frame rather than just the format. Enable that for the V4L2 driver now that the firmware populates all the parameters. raspberrypi/firmware#1885 Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> staging: bcm2835-codec: Add support for H264 level 5.0 and 5.1 We do NOT claim to support decoding in real-time for these levels, but can transcode some content, and handle 1920x1200. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: codec: Allocate the max number of buffers on the VPU The VPU's API can't match the use of VIDIOC_CREATE_BUFS to add buffers to the internal pool whilst a port is enabled, therefore allocate the maximum number of buffers possible in V4L2 to avoid the issue. As these are only buffer headers, the overhead is relatively small. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com> vc04_services: vchiq-mmal: Add defines for mmal_es_format flags There is a flags field in struct mmal_es_format, but the defines for what the bits meant weren't included in the headers. For V4L2_PIX_FMT_NV12_COL128 support we need them, so add them in. Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
After some time of normal functioning rpicam-vid just hangs without producing any video data.
Bug report
bug-report.txt
debug.log
strace.log
Additional context
Same happens with two other RPi0 and two RPi4, all fully updated. Tried both 32 and 64 bit OS/package versions, different ov5647 modules, libcamera/rpicam-apps from repos and sources. I noticed that problem occurs more frequent when low bitrates are used, tried 10k-512k range.
Probably related:
UPD: with "LIBCAMERA_LOG_LEVELS=*:DEBUG rpicam-vid -v -t 0 --framerate 24 --width 720 --height 576 --bitrate 20480 -n --denoise off -o /dev/null" hangs within first 10 frames, log attached.
debug2.log
UDP 2: does not hang with "--codec mjpeg" istead of h264.
The text was updated successfully, but these errors were encountered: