Lenovo x270: power-up delays, freeze on suspend

[je-nix]

Hi everyone

Inspired by your talk on the 34C3, I managed to flash my Lenovo x270 by creating a dump of the BIOS chip, successfully cleaned it with me_cleaner and then flashed that image. The good news: it seems to basically work, the device powers on and runs (no errors, no shutdown after 30 mins or something like that). Unfortunately, I can't verify that ME is disabled since /dev/mem/, which is required by intelmetool, does not exist on my Arch Linux (even when setting the iomen kernel parameter in grub). But that's no big deal for now.
The bad news: I experience a delay of about 20 seconds on power up until the boot screen appears (when powering on the device or restarting it) where the device is turned on, but does not seem to do anything. While I could live with that, the even worse part is that it freezes when I wake up the device from suspend. When I wake it up (push the power button), the device turns on (power LED, display, keyboard lights etc), but then immediately freezes and does not respond to any input (keyboard, acpi, etc). The only option is a forced power off.
Since this can have many different causes, it's hard to debug. But having nothing changed except the ME-free BIOS it kind of points toward this as cause. So I want to ask if this behaviour is already known and a work-around exists (or maybe I did something wrong?).

Thanks already for the support. Please let me know if I can provide additional information. I can also provide model specific information (type and location of the BIOS ROM for the x270 etc) and pictures if needed.

[c0d3z3r0]

Hi, is there any difference with the sleep issue with/without AC attached?
Just curious: does the x270 have SOIC8 or WSON package (like my x260)?

[je-nix]

Hi @c0d3z3r0
Thanks for your reply. I will try if there is a difference regarding AC power when suspending my device and then get back here.
Regarding the chip, it is a Winbond 25Q128FV which according to the data sheet can be used as SOIC or WSON. From the little information I could find about the differences, I believe it is a SOIC8. I uploaded some of the pictures I took while dismantling my x270, including one of the chip itself. So maybe you can verify that yourself: https://github.com/je-nix/x270-internals/raw/master/pictures/Winbond_25Q128FV_chip.png

[je-nix]

There is also another issue regarding the recognition of the nvme drive where I'm not sure if this is related to the ME free BIOS or my sloppiness during flashing. First I thought I just damaged the cable during dismantling or reassembly (it already was quite bend on one spot), but now I'm not sure. Even when totally untouched, my device will sometimes recognize the drive and sometimes fails to do so. Power cycling solves this (also it sometimes takes 2 or 3 attempts. I haven't found any facts what causes this, but I have the impression it's especially common when restarting the device. I'll investigate further and get back here.

[c0d3z3r0]

Thanks for the picture :-) Interesting... I'm wondering why they switched back to SOIC8. My x260 has WSON so I could not use my SOIC clip but had to solder the wires to the chip (and then bricked my mainboard somehow -.-).
I also had to remove the complete mainboard since the chip is on the other side.

[je-nix]

Maybe the SOIC allows for easier support when recovering faulty flashed chips :)
There is in fact another chip on the back side of the system board which I first mistook for the bios chip (it's a Winbond 25Q80BL SOIC). I have no idea what it's purpose is, but I'm also quite new to the electronics stuff (I'm from the System Engineering / Networking area).

[corna]

@je-nix Are you using the HAP bit or not?

[je-nix]

@corna I applied me_cleaner without additional parameters, so from my understanding HAP should be enabled. Or am I mistaken?

[c0d3z3r0]

@je-nix would you (privately) share a dump of the 25Q80BL? I could take a look at it.
HAP bit is only set when applying -S or -s.
What I find interesting is that you can apply me_cleaner in normal mode on an x270 which has Verified BootGuard enabled o.O That should not work at all if I am not mistaken. @corna Is my assumption correct?

[je-nix]

@c0d3z3r0 Sure, if you are interested I'll send you a dump of it. Unfortunately I already deleted the dumps I created, so I'll have to make another one. Seems I'll have to flash another image with HAP set anyway.
I can also share additional information (or the original dump of the chip) if needed.

It's possible that I've disabled BootGuard (if that is possible in the BIOS settings), as well as all UEFI stuff that you can disable there. My device is booting in "Legacy only" mode, but I'm happy to check / verify the settings.

[corna]

@je-nix
Nope, it must be activated with -s (HAP bit only, not recommended) or -S (HAP bit + code removal).

Boot Guard can't be disabled once enabled by the vendor (the activation state is saved in a write-once area in ME).

@c0d3z3r0
The interaction between Boot Guard and me_cleaner is not yet fully understood, see #6 and this, however in most of the cases it works correctly if the HAP bit is set (in addition to the code removal). From this blog post from Positive Technologies:

We also found some code in BUP that, when HAP mode is enabled, sets an additional bit in Boot Guard policies. Unfortunately, we have not succeeded in finding out what this bit controls.

[c0d3z3r0]

@corna aah. I remember reading this before... thanks!

[c0d3z3r0]

@je-nix if it's ok for you both flash dumps would be great :-) I'll share my findings of course

[je-nix]

@corna Thanks for your reply. I just ran me_cleaner again with the -S option to set the HAP bit and am currently flashing the new image. I'll get back to you when it's done about the result.

@c0d3z3r0 Sure, as far as I read there should be no critical information in the dump, correct? What would the OEM partition contain (if anything)? MAC addresses? Serial Numbers?
Also, how can I send you the dumps?

[c0d3z3r0]

@je-nix oh, just saw you deleted all previous dumps? do you have a backup of the original content?
Hm, there should be MAC (not sure), Serial, Model number, Windows Serial Key (if bought with Windows) and your BIOS/UEFI settings. You can send me an email to security(at)mniewoehner.de

[je-nix]

@c0d3z3r0 Sorry, I was a bit unclear there - I deleted the dumps of the second chip (the 25Q80BL), but of course I have kept a dump of the original content of the BIOS chip.
I'll send you the two dumps via mail, gladly I was able to purchase my device without any windows license (hooray for cheap special offers for students).

[je-nix]

Thanks for your support @corna and @c0d3z3r0. I created a new image with me_cleaner in which I set the HAP bit (via the -S option). After flashing that image, all the issues are gone. No more delays during power on and suspend works well again.
(The only option is the detection of my NVMe drive, but I guess that really boils down to an hardware issue with the cable. I'll look for a replacement part for that.)

May I suggest to expand the wiki entry for External Flashing (https://github.com/corna/me_cleaner/wiki/External-flashing). I would add some reference to the HAP bit options to the "Neutralize Intel ME" chapter. I initially followed this guide and thus missed the options which caused my issues in the first place.

I'll leave this issue open for now in case @c0d3z3r0 wants to share his findings for the 25Q80BL chip.

[corna]

I've not updated the Wiki in a while, I'm going to add the -S option as recommended in all the guides.

[c0d3z3r0]

@corna @je-nix I guess the remaining parts of ME do not work correctly after cleaning. The DISABLE_ME jumper on my P320 Mainboard seems to be equivalent to the HAP bit.
Since disabling ME in a very early stage by whatever means - HAP bit, AltMeDisable, jumper etc. - prevents running the problematic parts of it. Compare #154
In conclusion this means there is maybe something missing or maybe another bug (like #154) when cleaning the ME image that should be found and fixed. Setting -S is just a workaround and hides the problem. When cleaning works fine -S should be set nevertheless.

[corna]

Why do you think that -S is a workaround?

[c0d3z3r0]

@corna I'm sorry, that was not what I meant. -S itself is not a workaround. It's only a "workaround" in je-nix's case. IMHO it should even work without -S (when we want it to be "perfect").

[corna]

I don't think it's a workaround even in je-nix's case: both the HAP bit and the code removal execute more or less the same code, the only difference is:

• Boot Guard status
• returned error codes and replies on the HECI interface (or whatever it is called now)

We can't directly control these, the only way is through the signed Intel ME code.

With -S we have both the code removal (so we're sure that the non fundamental code is not executed) and the HAP (better return status), so I think it's a complete solution.

[c0d3z3r0]

@corna Ok, from this point of view it makes sense. Maybe this was just my perfectionism ;-)
@je-nix I've had a short look on the second dump but I have no clue what it is... looks a bit like some embedded code, not just data. Is there any uC in the near of the flash?

[je-nix]

@c0d3z3r0 Thanks for taking a look. There is a ti (texas instruments) uC next to it (model TPS65982DA) which could possibly be the USB C controller. So it probably belongs to it. I've uploaded a picture of the surrounding of the Winbond chip if you want to take a look yourself: https://github.com/je-nix/x270-internals/blob/master/pictures/Winbond_25Q80BL_surrounding.jpg

[c0d3z3r0]

@je-nix yep, you're right. Look at 8.4.5 Application Code in the datasheet:
http://www.ti.com/lit/ds/symlink/tps65982.pdf
Here is even more info about the firmware: http://www.ti.com/lit/ug/slvuah7b/slvuah7b.pdf

@corna I'm sorry for hijacking this issue for non-ME stuff. I think this can be closed now.

[je-nix]

@c0d3z3r0 Great, "mystery" solved. I'd close this issue since it is resolved if you don't have anything further. Thank you again for your great support.

[c0d3z3r0]

@je-nix yes, you can close the issue