Monitor serial console message during (e.g.) boot/shutdown

[hallerjulian]

Hi all,

we are using labgrid now for quite a while in BSP development and testing of different custom boards. As probably most users, we mainly use the serial console to control our targets. While executing commands works fine in our automated test cases, we often come to the point that we also need to monitor the serial console output while the device is booting. As an example, if I change something in the kernel or bootloader as a developer, the first thing I do on the DUT is to check the serial console output if I see anything unexpected during the boot process. Or sometimes you fix a bug and verify it by not seeing a specific kernel output anymore (because the actual effect of the bug is hard to measure).

There are probably many more cases where the "cheapest" indicator for presence or absence of errors are console messages printed during boot/shutdown. And exactly for those cases it would be good to have automated test cases for.

An approach could be to write a new driver, maybe called SerialScanner. This driver binds to a SerialDriver, just like a ShellDriver or UBootDriver. The driver then offers calls to check that a series of strings appears within a specified timeout, or that some specified strings do not appear. Both should be implementable using the expect method of the ConsoleProtocol.

Is this something that might be useful for other people as well? Or is this even possible already with labgrid and I have overlooked something?

[Bastian-Krause]

What about checking the kernel messages after the login with journalctl or dmesg? barebox understands dmesg, too. A persistent journal should also include shutdown messages on the next boot.

If the messages are really only visible on the console during boot/shutdown, I guess you could also use self.console.expect() in your strategy (assuming your bindings include "console": "ConsoleProtocol"):

• before activating the bootloader driver
• in the bootloader between boot() and await_boot()
• after await_boot() before activating the ShellDriver
• after running reboot or something similar in the shell

..to catch messages.

[hallerjulian]

Using journalctl or dmesg works under 2 conditions:

• As you said, a persistent journal is required, which is not always the case. Also, not everyone is using barebox, at least we don't.
• The test case must result in a booted system. But often we also want to test the negative case, e.g., that the system refuses to boot if a consistency check fails. This might cause the DUT to not reach the linux user space at all, depending on the exact test case.

The strategy approach could be used, I will check on that. The doubt I have here is that the test cases I have in mind often deviate from the "standard path" a strategy implements.

A very simple example test case where I see that neither the strategy nor the approach via the journalctl might work:
Test scope: Show that the bootloader refuses to boot a modified image (i.e. kernel image checksum is invalid)
Test steps:

1. Boot the system and modify a byte in the kernel image
2. Power cycle the device
3. Verify that the bootloader comes up and complains about the kernel image (by checking the well known console messages)
4. Verify that the bootloader resets the board afterwards within the specific timeout

[Emantor]

I agree that we are currently missing a good interface to retrieve the boot log messages from the kernel during system boot up. This all ties into a bigger issue with our current logging infrastructure, namely that serialization/deserialization is none existant and all log messages have arbitrary formats. We also currently only use them for debugging and have the steps information for internal data. I don't think we need a dedicated driver, but rather should rework the logging story for labgrid.

[hallerjulian]

If I get you right, the idea is to have the collected messages in a general log where I can look them up afterwards?

[Emantor]

Yes, something like that. Maybe not a general log but something that is bound to the specific SerialPort. Not sure what the architecture for this looks like yet.