Talos II under Heads rolling release: TPM module (revision prior of 1.1) : not detected #272
Labels
Comments
|
Hostboot+petitboot reports the same |
This was referenced Nov 23, 2022
|
coreboot config in Heads tree was not matching linuxboot/heads@9fc1f1c Coreboot config adjustment made under linuxboot/heads#1247 Reproducible testing trace: New observations: Unfortunately, the behavior is weird. It seems that having coreboot activate the TPM results in TCPA logs even though the TPM is not actually initialized properly nor made available currently. But the measurements are taken but waiting to be reported to the TPM. Output: A total of 2263 entrees like those shows in Let's resume output: |
|
Same roms as above test
Without TPM module connected. cbmem -L reports measurements without TPM Not normal: Also those messages are showed on screen BMC screen at boot, but not recorded neither in cbmem nor in dmesg: For good measure, to make sure my TPM module is not at fault, rebooting with TPM module connected to see if messages related to "No TPM registered/enabled" are present as well: Possible next steps:
|
|
@SergiiDmytruk I guess we would want STB logs recorded, since the most relevant information from logs While with TPM module connected/not connected (hope I do not have a faulty TPM module) as built from Heads and made easily reproducible from above traces: [last logs on screen] |
|
As a side note Heads points to dasharo/coreboot commit https://github.com/osresearch/heads/blob/3184bf7a8c6404551871e1353a391710cc52f426/modules/coreboot#L38-L41 TPM is enabled in config per PR https://github.com/osresearch/heads/pull/1247/files#diff-2cee4a5f5c9ee3d72c3fa7f916b9efd372a7e3f11b30b590f8fcac2d02cce425R3 Something else missing? |
|
@tlaurion have you tried full power cycle by cutting power for a while? BMC can be quirky when it comes to I2C, and we've seen cases where nothing but full power loss helped. We've had quite some of those issues while trying to enable I2C wrong way, and given what changes had to be made to fix booting on your platform I2C frequency was probably incorrectly calculated in earlier versions of Dasharo. |
|
Full power cycle happened in those tests yes. The platform was cut of power for 24h between the tests. (PSU power button turned off). Will redo. Orientation of board validation would be nice as well as consequences if it was connected wrong. |
Unfortunately I do not know. If you have a PR based on linuxboot/heads#1247 to propose so that can be tested, I would be willing to test. (PR is activating TPM, since past merged PRs were not containing required coreboot changes to activate TPM as discovered through community call last thursday with @miczyg1). Unfortunately, activating TPM doesn't discover bought TPM module, and the behavior is the same with/without TPM module connected, including TCPA logs produced without TPM connected, which is weird, to say the least, and where my TPM module seems dead, even though orientation was replicated (see #272 (comment)). We might be facing dead TPM module(waiting for instructions on how to proceed), while behavior below is still valid for investigation. Redoing test, starting with no electricity on motherboard, no led (PSU off), on artifacts produced by linuxboot/heads#1247 under https://app.circleci.com/pipelines/github/tlaurion/heads/1275/workflows/c1d5c877-f009-4a5b-9497-261de3035663/jobs/12661/artifacts Boot test: Cutting here since otherwise, buffer is bigger then 64kb which is limit of Qubes clipboard. cut.... resume on last: So:
|
|
If TPM is present on the bus, running
Dasharo/coreboot#185 (comment), but success of initialization can be checked (need to add a function for that). |
|
Regarding assert after dead loop, I've managed to reproduce it (at least I think so, though line numbers are different) and it seems to be caused by RNG initialization in chip.c. Hostboot was significantly slower than coreboot so it worked there, it also worked with more verbose output and/or with more RAM in coreboot so I haven't seen it earlier. I've left this
These would likely be messages produced by Skiboot. It should be exposed by kernel through |
|
Edit: that was from Heads pr pointed in this issue to have TPM enabled in coreboot (missing otherwise).
Is as of #192 (comment) Should I conclude that my TPM module is broken? |
Could be. It was verified before shipping, though. We will re-check on our side with the same set of the firmware binaries as you listed in the issue description to see if we can see similar problems on our side. If not, it is more likely that there is some platform difference or something else with the module is broken. @sulewskiprzemyslaw will report results of the test here. |
|
@tlaurion You can also list i2c buses via |
|
Interesting and possibly connected find in Skiboot. This is for HDATA (i.e. Hostboot) path, we skip it by passing FDT directly. We probably should add this workaround to coreboot. |
@SergiiDmytruk , note the error "[ 208.132623669,3] I2C: Initial error status 0x04011f0104000000" above? (All the above of course with TPM module connected). |
|
The BMC sees way more then Heads (above):
@SergiiDmytruk you want me to grab some output? |
No, I haven't figured out the mapping of the buses there and not sure I was able to see TPM from BMC in my tests. Interesting thing is that your Talos has fewer buses than ours (detection results differ too): Same output from Debian: Note that |
|
I have reproduced the problem, using the firmware components given in the issue Using the latest firmware components in version v0.6.0 from dasharo, |
|
@Pokisiekk we are talking about those artifacts, correct? #272 (comment) |
|
@tlaurion I used the following command to reproduce the issue: |
|
@Pokisiekk the main post of this issue was artifacts based on Heads master, which was not pointing to correct dasharo/coreboot Talos II config as documented under this reply of this issue: #272 (comment) To have TPM enabled under coreboot, as specified again in comment #272 (comment) the correct versioned artifacts are product of PR linuxboot/heads#1247 :
I'm glad I asked for clarifications. Upstream Heads is not having TPM support without that change in coreboot config.
Basically, you tested coreboot without TPM enablement. |
|
I just retested 0.6 release on my side with same results: my TPM module doesn't get detected: cut here. Resume at last repeated message: Skiboot logs: |
|
@tlaurion I used firmware components from #272 (comment) and the TPM seems to be working fine. Some command outputs: |
Ok. Retrying on top of linuxboot/heads#1313 Unfortunately, no success after having followed https://docs.dasharo.com/variants/talos_2/tpm-support/ with module provided at FOSDEM.... Please replicate: Main important output: complete console log provided in attachment. 3mdeb/talos-tpm-module#2 can be closed. |
|
As far as I know, I think my motherboard's TPM connector is now fried by having inverted connection prior :( @macpijan @SergiiDmytruk @krystian-hebel : Not sure what to do next. If everything TPM related is
Then my conclusion is fried TPM connector on mainboard. :/ |
|
TPM changes in coreboot may not be fully merged on the CPU frequency branch. At some point
Let's hope not. It may also be something related to different CPU revision or its frequencies. Register for setting up I2C frequency is differently described in documentation, comments and code, and value read back after Hostboot finishes is different than any of those. We ended up setting it by trial and error, but maybe there is a logic to it. |
|
@krystian-hebel well keep me posted. Cpu freq + fan speed seems fixed under linuxboot/heads#1313 You told here above to not merge under Heads, now I get why : different branches not on top of each other. Please provide another PR as for linuxboot/heads#1313 TPM is inserted properly this time, there is no TCPA log produced when no TPM discovered (cbmem -L), which is improvement from last remembered series of tests. Tag me when ready to test, ideally producing images from CircleCI that works on your side so I can test them on my side and report on expected working branch and corresponding PR. |
A lot of errors still for i2c transfer. Hopefully those are linked to my non-successful story with TPM! |
@krystian-hebel : you also get those i2c transfer errors under
? |
|
No such errors on our side. This may indicate different bus frequency, or any other problem. |
Just to make sure. Heads master points to This is what you want me to test with/ without TPM to provide logs? |
|
@tlaurion use the same version that was used to get logs in #272 (comment) |
#272 (comment) refers to testing for CPU and fan throttling which you said might not have TPM completeness in TPM. Going back to testing heads master which points to dasharo/coreboot 2207bbcccba31ad89cf21607b0d8d05d8dc47c03 |
Gives:
|
|
Wait.... what? On HARD reboot (poweroff, and then botting back): |
|
@tlaurion was BMC also power cycled? This may be using physical flash instead of mounted file, or vice versa. There may be different older versions that would have worked if it wasn't for broken module. |
Looking. Past reply
Which is 9676c79 which corresponds to at linuxboot/heads@9676c79 Lack of cmbem was also g9676c79 as can be seen under
Above. Let me know what I can do to narrow down or complement your tests. Where prior rom was from linuxboot/heads#1313 and flashed to BMC mem. I expect the firmware flashed internally from Heads to also be in BMC since mtd should abstract real physical presence. Also confirmed: I still have hostboot in pnor, never overwritten it in previous tests until now because scripts used. |
@krystian-hebel : No bmc was not power cycled. |
|
Then either OS overwrote coreboot tables or there is another problem with cbmem. Could you add comment about it to #69? |
|
Conclusion off-channel is that flashing internally doesn't do a coldboot with a normal OS reboot. Possibility here is to hack flash-gui.sh so that reboot/poweroff depends of board configs. In Talos case, poweroff would be required. |
|
Also needs a new release like v0.7.0 to close the issue |
|
@krystian-hebel @miczyg1 |
|
The ones just before |
@krystian-hebel from log provided under #272 (comment) ? |
|
Sorry, I've missed it, it doesn't appear on the second/third boot, only the first one. However, on those boots coreboot sees TPM but Skiboot complains, and Linux is totally unpredictable on that matter. This needs more investigation, but I still think this has to do with bus frequency. |
|
Might be related: open-power/hostboot#71 (comment) |
|
@tlaurion Can you provide some more logs on how you tested the TPM? @krystian-hebel Is concerned still about these I2C errors, and maybe some measurements in the log are missing. @krystian-hebel Please list commands output you need to judge whether it works ok, or not. |
We've tested TPM you returned to us and it behaves very strangely. It works in coreboot, but not in Skiboot and because of that also Heads. Another module works reliably on our platform, at least according to preliminary tests. |
|
Interesting find: if Skiboot fails to load kernel, it somehow tells BMC to drop mounted PNOR image and get back to real flash device. After that, |
|
I am not sure if another issue should be opened or this one bonified with more logs already posted at linuxboot/heads#1313 (comment) ? |
|
Continuation here: #415 Closing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Dasharo version
Heads running release. Command line download of required images, links obtained from CircleCI build of master's server board's artifacts:
user@talos-tests:~/QubesIncoming/heads-tests$ rm * && wget https://output.circle-artifacts.com/output/job/90adf917-76d4-4d26-be2a-e1f0cf121724/artifacts/0/build/ppc64/talos-2_server/heads-talos-2_server-v0.2.0-1296-g139ecb8-zImage.bundled https://output.circle-artifacts.com/output/job/90adf917-76d4-4d26-be2a-e1f0cf121724/artifacts/0/build/ppc64/talos-2_server/heads-talos-2_server-v0.2.0-1296-g139ecb8.bootblock https://output.circle-artifacts.com/output/job/90adf917-76d4-4d26-be2a-e1f0cf121724/artifacts/0/build/ppc64/talos-2_server/heads-talos-2_server-v0.2.0-1296-g139ecb8.romUpload:
rsync -ravczz --inplace --delete /home/user/QubesIncoming/heads-tests root@192.168.2.187:/tmp/images/BMC creation of test image (boot from BMC flash without actually flashing):
ssh -l root 192.168.2.187 "pflash -r /tmp/talos.pnor && cd /tmp/images/heads-tests && pflash -F ../../talos.pnor -f -P HBB -p *.bootblock && pflash -F ../../talos.pnor -f -P HBI -p *.rom && pflash -F ../../talos.pnor -f -P BOOTKERNEL -p *zImage.bundled && mboxctl --backend file:/tmp/talos.pnor"Boot from modified BMC image:
ssh -l root 192.168.2.187obmcutil poweron && obmc-console-clientDasharo variant
Server
Affected component(s) or functionality
TPM module, first revision is not detected
Brief summary
How reproducible
100%
How to reproduce
Test with above instructions with a TPM module connected
Expected behavior
TPM being detected from coreboot (cbmem -L) then detected under Heads and used to extend measurements
Actual behavior
TPM is not discovered
Screenshots
I replicated connection/orientation as shown under 3mdeb/talos-tpm-module#2 (comment), connected with and without jumper to obtain same undetected module from coreboot and Heads
Additional context
Solutions you've tried
None
The text was updated successfully, but these errors were encountered: