Check DRM plane type before attaching it as main plane

[shengwen-tw]

During the development of our RISC-V system emulator, semu, we encountered an issue where DirectFB2 failed to work with DRM mode, producing the following error:

(!) Core/LayerRegion: dfb_layer_region_realize() in dfb_layer_region_flip_update() failed!
    --> An invalid argument has been specified

Upon investigation, we found that the virtio-gpu driver (for GPU virtualization) in the Linux kernel allocates two planes: a PRIMARY plane using the XRGB8888 format and a CURSOR plane using the ARGB8888 format.

static const uint32_t virtio_gpu_formats[] = {
	DRM_FORMAT_HOST_XRGB8888,
};

static const uint32_t virtio_gpu_cursor_formats[] = {
	DRM_FORMAT_HOST_ARGB8888,
};

...
struct drm_plane *virtio_gpu_plane_init(struct virtio_gpu_device *vgdev,
					enum drm_plane_type type,
					int index)
{
...
	if (type == DRM_PLANE_TYPE_CURSOR) {
		formats = virtio_gpu_cursor_formats;
		nformats = ARRAY_SIZE(virtio_gpu_cursor_formats);
		funcs = &virtio_gpu_cursor_helper_funcs;
	} else {
		formats = virtio_gpu_formats;
		nformats = ARRAY_SIZE(virtio_gpu_formats);
		funcs = &virtio_gpu_primary_helper_funcs;
	}
...
}

However, the DRM subsystem in DirectFB2 adopts a policy of using an ARGB plane as the main plane if possible. This caused DirectFB2 to incorrectly select the CURSOR plane, instead of the PRIMARY plane, when running on semu with the virtio-gpu driver.

Currently, Linux DRM allows using multiplane overlay (MPO) model for power-saving. The plane types consist of PRIMARY, CURSOR, and OVERLAY. However, only the PRIMARY plane supporting operations like modesetting, flipping, etc. This is unlike the older DRM design, which treated all planes as universal.

This commit resolves the issue by introducing plane type checking, ensuring DirectFB2 does not mistakenly select non-PRIMARY planes as the main plane.

Reference:
[1] Multiplane Overlay (MPO) - The Linux kernel

[caramelli]

Interesting!

The "AR24" format used by default in the DRM/KMS system module is historical but the "XR24" format should probably be preferred (the primary layer is the bottom one in general so an opaque format might make more sense).