FW16 Freeze then Reboot (FTR) S5_RESET_STATUS = 0x08000800 <- Sync Flood.

[jcdutton]

Device Information

System Model or SKU

[ ] Framework Laptop 16 (AMD Ryzen™ 7040 Series)
No dGPU.

BIOS VERSION

3.0.5

Windows:
N/A

Linux:
Open a terminal and run the following command
sudo dmidecode --string bios-version
03.05

DIY Edition information

Memory: Manufacture and SKU
Kingston Fury Impact: Part Number: KF556S40-32
2x making 64GB total.
Storage: Manufacture and SKU
Model Number: WD_BLACK SN850X 1000GB
Firmware Version: 620361WD

Port/Peripheral information

1. USB-C card, nothing plugged in.
2. Empty
3. Empty
4. Empty
5. USB-C card, FW16 PSU plugged in.
6. USB-A card, nothing plugged in.

Standalone Operation

Are you running your mainboard as a standalone device. Is standalone mode enabled in the BIOS?

• No

Describe the bug

S5_RESET_STATUS = 0x08000800 <- Sync Flood.

Occasionally, about once a month I get a random crash/freeze then about 20 seconds later a reboot.
There is some details in this community thread:
https://community.frame.work/t/frwk16-random-crash-then-reboots/62411/31

This issue is only for "Freeze then Reboot" issues. Not "Freeze then power off".

Steps To Reproduce

Steps to reproduce the behavior:

1. Start from a powered off laptop.
2. Power on laptop
3. Wait a random amount of time. Play videos, netflix, youtube etc.
4. System freezes for about 20 seconds and then reboots itself.

Note: I generally have the power plugged in most of the time. For all the FTR I have seen, the power was plugged in at the time. The PSU used is the FW provided one that comes with the FW16.
Note: The FW16 was not under high load. the cpu fans were not audibly running. I.e. I could not hear them above the netflix / youtube video playing.

Expected behavior

It should not randomly freeze then reboot. (FTR)

Screenshots

N/A

Operating System (please complete the following information):

• OS/Distribution: Linux/Ubuntu
• Version: 24.04
• Linux Kernel Version: uname -a 6.12.7 <- Mainline compiled kernel.

Additional context

Add any other context about the problem here.

[jcdutton]

There has been an interest in understanding the port80 codes.
On the FW16 AMD, these are 32bit values, and not 8bit values.
I have a version of the EC firmware that dumps the last 4096 PORT80 codes, and this is enough to survive a reboot of the CPU. During the CPU reboot, the EC does not reset, so the PORT80 codes should be retained.

I attach some files, that contain the PORT80 codes output during a normal reboot and a normal cold-boot. They have been sorted and de-duped to shorten the list of codes.

It would be helpful if FW support could identify what each code means.
When the problem next occurs, I will be capturing the PORT80 codes and posting them here.

port80-reboot-de-dupe1.txt
port80-cold-boot-de-dupe1.txt

[jcdutton]

People are seeing 3 different scenarios:

1. an instant power off. Needing power button to switch on.
2. a system pause/blank screen, for about 20 seconds, then a reboot.

In both cases, no logs or crash dumps to help diagnose the problem.

3. the screen goes blank, but does not power off and does not reboot.

All three are difficult to reproduce.
I see (2) about once a month.
If people are seeing (3), that is just a gpu driver bug, so gather crash traces and report to amd.
If people see (1), please raise a different Issue, and don't mix it with this one.

This issue is only for "Freeze then Reboot" issues. Not "Instant power off".

[jcdutton]

As an aid to people trying to understand the EC console output.
E.g. [66390.711600 HC 0x0115 err 1]
#define EC_CMD_PD_GET_LOG_ENTRY 0x0115
So this is the Host trying to get the log entries from the EC firmware, but on the FW16, this does not appear to be implemented, thus returning an err 1.

Please see attached file for the definition of each HC (Host command to EC)

ec_commands.txt

[jcdutton]

When comparing multiple port80 traces, from say, repeating a "reboot", and then comparing the sequence of port80 values. It appears to be that the capture of port80 values by the EC is not perfect. There a bit errors and missed values.
I.e. Sometimes the value is absent in the sequence compared to a different reboot.
Also, sometimes some bits are set in the 32bit value, and other times it is not set. This would hint at bit errors being introduced.
e.g.
ef008888
vs
00008888
vs
b0008888
vs
000b8888
The "ef" has bits set when maybe they should not have been set.
The "b0" has bits set when maybe they should not have been set.

I will see if there is some bug in the EC code or something else causing the problem.

I have now confirmed there is a problem.
I can write predictable values to port80 from user-space, and some of the output is lost.
The EC does actually read all values correctly, if the usleep(2000) is used between each outl()
The EC does actually read "a value" for each value written to the port80, if a reasonable usleep(100) is made between outl(), but the value is corrupted sometimes.
So, it appears the corruption happens if the outl() is made too quickly, without a delay between writes.

Further investigation:
I don't have the EC datasheet. I would welcome it is someone could send me one.
The EC receives 32bit port 80 values.
The EC only has a 4 32bit value hardware buffer. ( In some special cases it can handle 5 32 bit values.)
The EC splits the 32bit values into index/byte pairs. Meaning, to read 32bit value one actually has to read 8 bytes on the EC receiving side.
The EC is slow to respond to interrupts, so if one writes more than 4 values quickly, the port80 hardware buffer overflows and one looses port80 values.
The EC code for receiving port 80 values is here:
File: host_subs_npcx.c
static void host_port80_isr(const void *arg)

Side affects:
While experimenting, when the EC is busy doing something, it can slow the entire FW laptop down. (to a crawling pace)
Various parts of the EC comms stops working. E.g. detecting power plug/unplug.
I have not diagnosed why EC problems would slow the entire PC down, but will look into it later.
I did not believe the EC could slow the main CPU down until I saw it myself. So it is a possible new source of something to look at if users report slowness. Particularly of interest to Real Time Audio users.

[PureKrome]

[jcdutton]

During a normal boot (I.e. not a FTR):
Some of the port80 codes are interesting:
Here it is using lots of Port80 writes to say:
"Entering Vbios Boot Load%r"
Also, with what looks like two devices trying to write to port80 at the same time.
Notice the "%" instead of the "e"

00000145 | 5f535452 | _STR
00000146 | 0000000a | ....
00000147 | 00000045 | ...E
00000148 | 0000006e | ...n
00000149 | 00000074 | ...t
0000014a | 00000065 | ...e
0000014b | 00000072 | ...r
0000014c | 00000069 | ...i
0000014d | 0000006e | ...n
0000014e | 00000067 | ...g
0000014f | 00000020 | ...
00000150 | 00000056 | ...V
00000151 | 00000062 | ...b
00000152 | 00000069 | ...i
00000153 | 0000006f | ...o
00000154 | 00000073 | ...s
00000155 | 00000020 | ...
00000156 | 00000042 | ...B
00000157 | 0000006f | ...o
00000158 | 0000006f | ...o
00000159 | 00000074 | ...t
0000015a | 00000020 | ...
0000015b | 0000004c | ...L
0000015c | 0000006f | ...o
0000015d | 00000061 | ...a
0000015e | 00000064 | ...d
0000015f | 00000065 | ...e
00000160 | e825fe00 | .%..
00000161 | 00000072 | ...r
00000162 | 00000020 | ...
00000163 | 0000002e | ....
00000164 | 0000002e | ....
00000165 | 0000002e | ....
00000166 | 0000000a | ....
00000167 | 5f454e44 | _END

[jcdutton]

The following file contains the Port80 codes during a reboot of the Linux operating system normally. I.e. User clicked on "reboot". The purpose of this is to

1. Share them
2. Maybe discover what some of them mean.
3. There are some comments in the spreadsheet, of the values I know already.
4. Once I see a FTR, I can then compare that bad one with this one.
5. I have created some new Port80 codes, and they are commented in the spreadsheet.
   For example, I have modified my Linux kernel so that it outputs Port80 0xaaaa0001 while booting at the BIOS to OS transition point. i.e. start_start()
   Also Port80 0xaaaa0003 while rebooting at the OS to BIOS transition point.
   Also Port 80 0xbadbadXX when the EC controller sees a hardware overflow when receiving Port80 values.
   In this way, it is easier to detect whether Port80 values have been lost or not.

port80-reboot10.ods

[jcdutton]

The changes I have made to the EC and ectool that work with a FW16 AMD are here:
https://github.com/jcdutton/ectool/commits/main/
https://github.com/jcdutton/EmbeddedController/commits/fwk-lotus-azalea-19573
https://github.com/jcdutton/zephyr/commits/lotus-zephyr/

To get longer port80 history:
src/platform/ec/include/config.h:#define CONFIG_PORT80_HISTORY_LEN 4096

I have found that 4096 is about the largest value you can give it, otherwise the EC runs out of RAM and behaves badly.
A complete reboot uses about 3500 port80 records, so 4096 is adequate for our needs.
I have only ever tested writing my modified EC firmware to the RW section of the Flash, I have not overwritten the RO section of the Flash, so if anything fails in my modified EC firmware code, it just reboots back to the original FW provided RO section.

[jcdutton]

My updates to the ECTool.efi reflash program, to make it slightly safer to use.
It defaults to doing nothing unless the command line arguments are exactly right.
https://github.com/jcdutton/FrameworkHacksPkg/commits/main/

I only ever write to the RW section, never to the RO section, so that if something goes wrong, it falls back to the FW provided EC firmware.

[sinatosk]

Hi,

I'm using

Framework 16 AMD Ryzen 7 7840HS with Radeon 780M ( no dGPU )
RAM/Memory 32GB ( 2x16GB ) Framework
NVME 2280: Western Digital SN850X 2TB - Firmware 620361WD
NVME 2230: Western Digital SN770M 2TB - Firmware 731120WD
BIOS 3.05
Gentoo Linux 2.17 ( Linux 6.14-rc2 mainline realtime, compiled by clang 19.1.7 march and mtune set to znver4)
KDE Plasma 6.2.5 Wayland

1. USB-C - Framework 180W Power adapter
2. USB-C - External display ( it has a DisplayPort USB-C adapter that plugs into Framework's USB-C module )
3. Framework's audio module
4. USB-C - External storage or mouse ( mouse has a USB-A to USB-C adapter that plugs into Framework's USB-C module )
5. USB-C - generally don't use
6. USB-C - I don't use because of Linux "sudo lsusb -v" fails #5

I originally thought this issue was with amdgpu/drm as I don't use stable kernels anymore ( not that I intend to continue but I do it to grab amdgpu/drm issues ASAP ), since kernel 6.12-rc1 I've only used mainline ( Linus Torvalds branch )

This FTR is something I've been experiencing for sometime and since BIOS 3.05 it's increased ( I believe it happened 3 times when using BIOS 3.04 but I just assumed it was amdgpu/drm at the time )

what @jcdutton says here, point 2 is the one I'm experiencing

a system pause/blank screen, for about 20 seconds, then a reboot.

The FTR issue for me triggers more frequently ( from multiple times a day down to 1-3 days ) when the Framework 16 display panel refresh rate is at 165HZ ( with Panel Self Refresh/Panel Replay enabled ), also using an external display ( 3840x2160@60HZ ) and in DC power mode ( Framework battery )

I've also noticed FTR happens ( maybe coincidence ) when watching a video ( youtube on firefox for example ) on primary or secondary display but then again the issues happens when I'm just programming ( RustRover mostly ) or browsing the web but it's random like others have said and when in AC power mode ( Framework 180W power adapter ), it could be a week or so before it FTR.

This is all while the power profile ( set via KDE power devil ) is set to powersave whether I'm in AC or DC power mode.

With the display panel refresh rate is set to 60HZ, it still FTR's but not as frequently ( that's how it feels ).

For the past week-ish though with power profile set to balanced ( sometimes performance when compiling software ) and using AC power mode ( Framework 180W power adapter ), display panel refresh rate set to 165HZ, it hasn't FTR yet.

From my perspective, it's looking like something to do with when running in powersave and it's worst when in DC power mode

Since day one I've had the charge limit on the battery set to 80% and most of the time I'm using powersave.

What I'm using to change the power profiles ( GUI component ) with KDE, I think it's called power devil that's communicating with power profile daemon

Because I set the power profile via KDE power devil and not directly in the pseudo/proc files, the terms between KDE and AMD not fully expressed

KDE -> AMD

powersave -> powersave
balanced -> balanced_powersave ( if using DC )
balanced -> balanced_performance ( if using AC )
performance -> performance

also ( may not have anything to do with it ) when using power profile powersave, CPB ( Core Performance Boost ) is turned off and the iGPU performance level is set to low, in other power profiles CPB is enabled and iGPU performance level set to auto

I've only just learnt of this issue here today but been experiencing this FTR since last year

edit1: I forgot to add. Just like others after reboot, I check the kernel logs for warnings/errors and there is none

edit2: power profile daemon also adjust AMD's ABM ( Adaptive Backlight Management ) if you don't have the kernel parameter amdgpu.abmlevel set. Mine is set as amdgpu.abmlevel=2

[PureKrome]

@sinatosk just to confirm, you DON'T HAVE the AMD RX7700S in the extension port thiny? This is all occurring on your iGPU?

[sinatosk]

@sinatosk just to confirm, you DON'T HAVE the AMD RX7700S in the extension port thiny? This is all occurring on your iGPU?

correct