RGB no longer supported after firmware updates

[petemoore]

Describe the bug
Mailbox property tag 0x00048006 ineffective at setting RGB pixel order after firmware upgrade.

To reproduce
Run https://github.com/bztsrc/raspi3-tutorial/blob/master/09_framebuffer/kernel8.img on a RPi 3B with latest firmware
Repository https://github.com/petemoore/redscreen now contains a complete minimal assembly example

Expected behaviour
Previously this used to correctly display homer on a connected HDMI monitor

Actual behaviour
Homer is cyan rather than yellow, since framebuffer is using BGR instead of RGB

System
RPi 3B

Additional context
See also bztsrc/raspi3-tutorial#64.

I haven't bisected the firmware revision where the problem started occurring, but a couple of years ago the same kernel8.img worked correctly with the firmware of the time. Randomly on one upgrade, it broke.

Please note, the code that generates the mailbox request can be found in lfb.c. The response to the 0x00048006 mailbox property tag is not checked in the example code, but when I add a check for it, I see that mbox[24] is 0 after making the mailbox call, so for some reason the request is denied, although it was previously supported on older firmware.

[petemoore]

UPDATE: I attempted to bisect the firmware git revision history to discover which commit first introduced the issue, but to my surprise, I cannot find a revision where homer appears yellow, despite in the past having homer yellow when running the example.

So I have to conclude that the change of colour must be caused by some other state change. The only things I can think that could affect state are the OTP settings on my pi that might have changed, and possibly which GPIO pins have leads connected. Could either of these affect the "Set Pixel Order" mailbox call (0x00048006), to prevent RGB (state 0x1) from being supported?

Note I serve the files over tftp, and have no SD card connected. I'm pretty sure when things were working, I had the same setup, around a year ago when I last ran the demo.

The full set of files I am serving are:

pi@raspberrypi:/tftpboot $ md5sum *
156a886d5855e42a887f0dd9316fb4e3  bootcode.bin
25a8ace87b9ca7638e7ead37a55b7266  fixup.dat
3a61e1c3eb85a3a2b8cdc1d6878bd2bf  kernel8.img
c403841ff2837657b2ed8e5bb474ac8d  LICENCE.broadcom
623255ad0bce5f2b0acb837584acb3e2  Makefile
d8b397009283ce9bd784b8724ae0f549  start.elf
pi@raspberrypi:/tftpboot $ 

The Makefile is unused by kernel8.img and is just a utility I have for grabbing the latest firmware files that I run on the tftpserver to refresh the sibling files in the directory. The firmware files are on the latest revisions (as can be seen from the md5 values above) and the kernel8.img file is the latest binary linked to in the issue description.

Please note, no other files are served, e.g. there is no config.txt file. However, as I say, this used to work in the past, and homer does display, just in the wrong colour (BGR instead of RGB).

[popcornmix]

It won't be otp or gpio.

Is it possible that start.elf was only behaving as you desired between two updates, and too old and too new versions are not working and the bisecting missed a good version?
Can you narrow down the working date? (e.g. by build time of kernel8.img)

It is possible your code is relying on uninitialised variable? sdram could have had a favourable value in the past ybut no longer does.

[petemoore]

It won't be otp or gpio.

Many thanks, that's good to know.

Is it possible that start.elf was only behaving as you desired between two updates, and too old and too new versions are not working and the bisecting missed a good version?

I've randomly tested with quite a few different historical versions of the firmware now, and can't get it to work with any so far, so I'm starting to wonder if it might be a subtle change in the compiler toolchain or something else external that is causing it not to work. I would automate the process to test all the firmware versions of the last few years, but unfortunately testing requires manually inspecting the HDMI display and powering the rpi on and off, so I think I'll leave that until I've ruled out some of the other options.

I'll see if I can develop an absolutely minimal kernel8.img in aarch64 assembly to demonstrate the problem. If it doesn't exhibit the same problem, then I should be able to incrementally adopt the original code until it breaks and find the cause. If it does exhibit the same failure, we should have a simple test case to demonstrate the issue, or the code will demonstrate a misunderstanding of the interface, or contain false assumptions, which can be corrected.

Can you narrow down the working date? (e.g. by build time of kernel8.img)

This kernel8.img is from @bztsrc's framebuffer tutorial. I was running it in late November 2018, I would guess from revision 8a80ab88.

It is possible your code is relying on uninitialised variable? sdram could have had a favourable value in the past ybut no longer does.

I don't think so. The framebuffer request is defined here and it looks like the full memory block is initialised.

Note, I also tried creating a config.txt with gpu_mem=192 in case the GPU was too low on memory by default to support the display mode options, but that wasn't the case.

[petemoore]

I'll see if I can develop an absolutely minimal kernel8.img in aarch64 assembly to demonstrate the problem.

The minimal kernel exhibits the same issue. I've published it to https://github.com/petemoore/redscreen. The source code is a single assembly file and a shell script to build it. I've also published the binary in the repository together with the firmware.

[petemoore]

@popcornmix I realise there is a lot of information above, so here is a short summary. When submitting this mailbox request for allocating a framebuffer from a bare metal context, the Set Pixel Order tag requests RGB, but the response returns BGR. Technically this seems in keeping with the Mailbox Property Interface docs, since the docs say that the pixel order choice cannot be guaranteed, and that the response should be checked to get the pixel order.

However, previously the same mailbox request would accept the RGB pixel order choice, and the tag response would be set to RGB as requested.

So I'm not sure whether this is technically a bug or not, it is more a little strange, since:

1. Previously RGB pixel order was possible
2. I'm not sure the value of being able to request RGB/BGR if what you actually get may or may not be what you request. Perhaps it would be better for the interface to allow you to allocate a framebuffer, and the response tells you the pixel order, rather than you request something that may not be accepted. Or will the GPU code try to get you your preference, if possible, and only if not possible, provide the alternative encoding? The issue I see is that any calling code will need to be able to handle both encodings, so I'm not sure why the calling code would provide a preference that might not be accepted.

Note, independently from me, @bztsrc discovered the same issue.

In any case, at least on my raspberry pis, the issue is reproducible with this bare metal kernel containing around only 40 instructions, so I'm hoping this should help with diagnosing the issue. Please let me know if you need any further information etc.

Many thanks in advance.

[6by9]

I've just had a quick look at the firmware source code handling this. it looks like this have never been hooked up to actually change the RGB order. It is stored and returned, but never acted on.
Indeed there is a validation function that includes

   if (fb->pixel_order != 0)
      fb->pixel_order = 0;

The framebuffer_swap parameter in config.txt will swap the RGB order, but not the mailbox call.

I'll add it to the list of things to look at at some point. I guess the two parameters actually need to be combined into one, otherwise an application is none the wiser should framebuffer_swap be set, and the colours will be wrong again.

[petemoore]

I've just had a quick look at the firmware source code handling this. it looks like this have never been hooked up to actually change the RGB order. It is stored and returned, but never acted on. Indeed there is a validation function that includes

   if (fb->pixel_order != 0)
      fb->pixel_order = 0;

The framebuffer_swap parameter in config.txt will swap the RGB order, but not the mailbox call.

I'll add it to the list of things to look at at some point. I guess the two parameters actually need to be combined into one, otherwise an application is none the wiser should framebuffer_swap be set, and the colours will be wrong again.

@6by9 Did this get resolved in the end? Many thanks!

[6by9]

I haven't fixed it, and it doesn't look like anyone else has either.

I've created what looks like the right change, and will put up a test firmware here once CI has built it for me.

[6by9]

Attached is the CI build from my change.
If you could test and confirm it does what you want, then I'll get it merged.

Please note that either setting framebuffer_swap or setting the pixel_order field in the mailbox call will switch to RGB rather than BGR. Setting both will still set it to RGB, not flip it back to BGR. No longer true. See latest comments.

fw1320.zip

[petemoore]

Hi @6by9 many thanks for this! I really appreciate it! :-)
I've actually just moved house on Sunday, so all of my stuff is in boxes at the moment. I have a working example for rpi 3B here with older firmware: https://github.com/petemoore/redscreen/tree/master/dist. That should generate a blue screen, and after patching the firmware, should generate a red screen, if all goes to plan. It was written for my rpi 3B. Otherwise, as soon as I find my pis and have unpacked them, I can give it a try.
Thanks again!

[6by9]

Thanks, I've tried your test, fixed up the firmware again, and now have a red screen.

I've also verified with vcmailbox 0x00048006 4 4 0x01 when the vc4 driver is not enabled, and that switches the framebuffer RGB order as expected.

It turns out that the framebuffer_swap option in config.txt defaults to enabled, so I have made it that the mailbox pixel_order field will flip it with regard whatever framebuffer_swap sets the default order to.