x230: keyboard not working after reboot

[alex-nitrokey]

I experience regularly an issue that is causing the keyboard or the x230 to not function after reboot inside of heads. Killing heads via pressing the power button until power off and starting again fixes the problem.

This problem is happening often after reboot (again, reboots inside heads), but not always. I could not detect a pattern yet.

Anybody had this issue or can deduce a cause?

[paulmenzel]

Maybe related to #698?

[alex-nitrokey]

doesn't sound like it imho

[stmc-droid]

I have the same issue sometimes when rebooting inside heads

[paulmenzel]

Are you able to save the logs somewhere even without keyboard?

Please start the Linux (Heads) kernel with i8042.debug, and contact the Linux kernel developers with the failing logs.

[tlaurion]

Intuition here is a race condition with USB Security dongle being connected and considered a keyboard. Possible? Would need to test and isolate if behavior is there without USB Security dongle being connected at boot.

[alex-nitrokey]

Good idea! I try to test this...

[alex-nitrokey]

Can not confirm. I tried around just a bit, but while doing similar stuff like generating new HOTP secrets and reboot via Ctrl+Alt+Del I once got stuck with not working keyboard and once I didn't (in both cases I led the Nitrokey inserted).

[tlaurion]

@alex-nitrokey best point would be to see if the same behavior happens without the Nitrokey Inserted and why the Nitrokey is considered a USB keyboard in Heads use case. (new behavior)

[alex-nitrokey]

I don't think, it has anything to do with the Nitrokey, this was just a working hypothesis above.

I just tested rebooting without inserting the Nitrokey or any other USB device. The keyboard still freezes.

[alex-nitrokey]

(go to recovery shell without doing anything else and type reboot)

[tlaurion]

Just putting some notes, I will do some Heads pull request merging in the next days after testing them under a branch under CIs.

On current issue:
reboot under heads is a bash script:

user@x230-master:~/heads$ cat initrd/bin/reboot 
#!/bin/sh

# Sync all mounted filesystems
echo s > /proc/sysrq-trigger

# Remount all mounted filesystems in read-only mode
echo u > /proc/sysrq-trigger

# Immediately reboot the system, without unmounting or syncing filesystems
echo b > /proc/sysrq-trigger

Please detail your use case:
1- Call reboot inside of heads.
2- Have main Heads screen but not able to type anything even if no USB drive being present on rebooted system?
Is that correct?

@alex-nitrokey :

Please start the Linux (Heads) kernel with i8042.debug, and contact the Linux kernel developers with the failing logs.
That means putting that under coreboot config. Result?

[alex-nitrokey]

Please detail your use case:
1- Call reboot inside of heads.
2- Have main Heads screen but not able to type anything even if no USB drive being present on rebooted system?
Is that correct?

This is one test case, yes. But it happens in different reboot situations as well, like doing the oem-factory-reset (in which I of course needed to insert USB devices). The above use case is just proving that seemingly it has nothing to do with the USB drive as it is happening even without inserting any.

Will try next week with the kernel debug. Where do I find the logs after applying the kernel parameter to my build and booting it?

[tlaurion]

Confirm the behavior after flashing with USB storage in.

[tlaurion]

confirmed after flashing again with usb storage in after oem factory reset

[tlaurion]

infirmed with HOTP security dongle while calling reboot from command line

[tlaurion]

confirmed with HOTP security dongle + usb storage while calling reboot from command line

[tlaurion]

infirmed while flashing rom and removing usb storage prior of rebooting.
confirmed while rebooting having USB Security dongle in prior of reboot from command line
confirmed that no keyboard with no USB devices connected in on calling reboot from command line.

So issue is not USB related.

[tlaurion]

Could it be linked to s3 resume suspend codepath being broken with new cross-musl-make compiler switch? (#608?)

[alex-nitrokey]

Mh, dunno, I never experienced such issue as in #608 and I am not sure what exactly causes it, so it is difficult for me to clearly see any connection, but could be 🤷

[tlaurion]

Just saw at at... something something popping up prior of Heads banner about keyboard issue... Seems like to be coreboot related and linux not being able to initialize it...

@alex-nitrokey Have you tried putting the i8042.debug changes?

[alex-nitrokey]

Have you tried putting the i8042.debug changes?

Unfortunately, not. I hadn't as much time as I hoped to. I tried to fix/test some quick stuff today, but won't get to this keyboard issue before next week, I am sorry :(

[alex-nitrokey]

I played around with current master builds and did not experienced this issue anymore. Is it possible that it is gone by a change of the last weeks?

[alex-nitrokey]

Looking at the commits my best guess is the gnupg version

[alex-nitrokey]

Well I am not so sure anymore... I think they are more seldom with current master but I still get the keyboard freezing sometimes. It is strange...

[alex-nitrokey]

Have you tried putting the i8042.debug changes?

@tlaurion can you please go into detail how I would apply and use this command? I tried putting it in coreboot config CONFIG_LINUX_COMMAND_LINE, but I don't know if this is right nor how I could use it, especially once the keyboard freezes.

[tlaurion]

@alex-nitrokey confirmed you were lucky. Testing next PR merge here and confirms random keyboard unresponsiveness
https://app.circleci.com/pipelines/github/tlaurion/heads/273/workflows/3ce7f868-50fb-420d-a45a-ec3048e350f3/jobs/296/artifacts

[tlaurion]

Have you tried putting the i8042.debug changes?

@tlaurion can you please go into detail how I would apply and use this command? I tried putting it in coreboot config CONFIG_LINUX_COMMAND_LINE, but I don't know if this is right nor how I could use it, especially once the keyboard freezes.

Would need to get the kernel to be really verbose on that line too... best debug output is from qemu board linux and coreboot configs, which is what is the most verbose we have. Trick would be to have mount-usb called and output everything there and mount read only to catch what is happening when unresponsive....

@alex-nitrokey Trick would be to call those as a debug hack prior of calling gui-init I guess since its platform specific.
@MrChromebox you dont have this random issue on other platforms, right?

[tlaurion]

@alex-nitrokey : haaaaa, forgot I worked on producing a x230-debug config in the past

[MrChromebox]

@tlaurion no ps2k issues on librem devices of which I'm aware

[alex-nitrokey]

Just for the record: I had assumed here that the keyboard issue might got solved. It didn't. Additionally, I tested today a bit more with different coreboot/linux versions. None of them worked correctly regarding this issue.

I tested coreboot 4.8.1/4.12 and linux 4.14/4.19 repectively (resulting in four builds). So it seems that there is no connection in regard.

[tlaurion]

@alex-nitrokey : we are here in the troubleshooting path #765 (comment), which I haven't investigated yet, but confirm coreboot/heads giving error on screen just before GUI when keyboard is not recognized anymore. Logs would be helpful.

Suggested hack to facilitate log grabbing

[tlaurion]

@alex-nitrokey fixed under #1015

[tlaurion]

#1015 merged