CY8CPROTO_062_4343W target failed on greentea watchdog test

[jamesbeyond]

Description of defect

In mbed OS CI we currently keep getting watchdog greentea test failures on CY8CPROTO_062_4343W target with watchdog tests.

as earlier mentioned in #11792 with @dustin-crossman, we keep seeing watchdog test failing on this target, and the issue can be reproduced locally. so raise this issue to track the progress.

Not sure if any special HW settings/modifications are required to enable the watchdog feature?

Target(s) affected by this defect ?

CY8CPROTO_062_4343W

Toolchain(s) (name and version) displaying this defect ?

ARMC6/GCC_ARM

What version of Mbed-os are you using (tag or sha) ?

latest master

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli and greentea

How is this defect reproduced ?

run command:

mbed test -t arm -m CY8CPROTO_062_4343W -n *watchdog*

and result on our side:
mbedgt: test suite report:

target	platform_name	test suite	result	elapsed_time (sec)	copy_method
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog	TIMEOUT	61.04	default
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog_reset	TIMEOUT	105.91	default
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	TIMEOUT	61.83	default
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_reset	TIMEOUT	106.74	default
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_timing	OK	41.75	default

mbedgt: test suite results: 4 TIMEOUT / 1 OK
mbedgt: test case report:

target	platform_name	test suite	test case	passed	failed	result	elapsed_time (sec)
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog	Restart multiple times	1	0	OK	0.05
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog	Start, 100 ms	0	0	ERROR	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog	Start, max_timeout	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog	Stop	1	0	OK	0.1
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog	max_timeout is valid	1	0	OK	0.05
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog_reset	Kicking the Watchdog prevents reset	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog_reset	Watchdog reset	0	0	ERROR	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog_reset	Watchdog reset in sleep mode	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_drivers-watchdog_reset	Watchdog started again	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	Init, 100 ms	0	0	ERROR	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	Init, max_timeout	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	Platform feature max_timeout is valid	1	0	OK	0.07
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	Stopped watchdog can be started again	1	0	OK	0.07
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	Update config with multiple init calls	1	0	OK	0.07
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog	Watchdog can be stopped	1	0	OK	0.12
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_reset	Kicking the Watchdog prevents reset	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_reset	Watchdog reset	0	0	ERROR	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_reset	Watchdog reset in sleep mode	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_reset	Watchdog started again	0	0	SKIPPED	0.0
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_timing	Timing, 1000 ms	1	0	OK	0.04
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_timing	Timing, 200 ms	1	0	OK	0.04
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_timing	Timing, 3000 ms	1	0	OK	0.05
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_timing	Timing, 500 ms	1	0	OK	0.04
CY8CPROTO_062_4343W-ARMC6	CY8CPROTO_062_4343W	tests-mbed_hal-watchdog_timing	timeout accuracy	1	0	OK	0.05

mbedgt: test case results: 12 OK / 4 ERROR / 8 SKIPPED
mbedgt: completed in 390.01 sec
mbedgt: exited with code 4

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOTRIAGE-2371

[dustin-crossman]

We're able to reproduce this issue now. It looks like the cause is that the default test spec (if no mbed_app.json and nothing provided to mbed test --test-spec) uses a baudrate of 9600. We weren't seeing this initially as our CI provides an mbed_app.json that specifies 115200 as the baudrate.

Can you confirm on your end that the tests will pass if the baudrate is set to 115200?

[jamesbeyond]

@dustin-crossman , yes oddly all tests passed in 115200 baudrate I can reproduce that locally.
but unfortunately. all of our CI test that runs under 9600. any ideas why baudrate impact the test results?

[dustin-crossman]

We're not sure yet but we are looking into it.

One interesting result we did see was in the tests-mbed_drivers-watchdog_reset in the test_simple_reset case. If the following lines

    Watchdog &watchdog = Watchdog::get_instance();
    TEST_ASSERT_FALSE(watchdog.is_running());
    TEST_ASSERT_TRUE(watchdog.start(TIMEOUT_MS));

are changed to

    Watchdog &watchdog = Watchdog::get_instance();
    TEST_ASSERT_FALSE(watchdog.is_running());
    wait_ms(20);
    TEST_ASSERT_TRUE(watchdog.start(TIMEOUT_MS));

the test will pass at 9600 baud. Perhaps all the data on the UART is not getting flushed out in time and that prevents the watchdog reset?

[dustin-crossman]

@jamesbeyond I believe I've resolved these failures. I noticed while watching tests run locally that the test runner was sometimes not receiving messages that the test should have sent before resetting due to the watchdog (e.g the reset notification message in mbed_drivers-watchdog_reset). It appears that the tests are simply not waiting long enough at 9600 baud for everything in the UART FIFO to be read out.

The tests define a SERIAL_FLUSH_TIME_MS value but the comment above it indicates that the value is calculated assuming a 16-byte Tx FIFO. The CY8CPROTO_062_4343W board has a 128-byte Tx FIFO so my fix was to simply increase the SERIAL_FLUSH_TIME_MS to 200 ms (from 20) and add a wait_ms to the end of the send_reset_notification functions in mbed_drivers-watchdog_reset and mbed_hal-watchdog_reset.

Here is the minimal changes required: dustin-crossman@53ad5b3

Do this approach sound like a reasonable approach to fix the watchdog tests?

Also the SERIAL_FLUSH_TIME_MS comment mentions that there is currently no mbed API to check if hardware buffers are empty. Do you know if there is any progress on that front? It seems like it would help a lot to prevent similar bugs on other hardware/tests.

[dustin-crossman]

@jamesbeyond Have you looked at these possible fixes?

[dustin-crossman]

Seems like this might have fallen off the radar with the holidays and everything. Has anyone taken a look at this?

[jamesbeyond]

Hi @dustin-crossman , sorry for the delay, we'll try your fix, see if it fixed the watchdog issue

[fkjagodzinski]

@dustin-crossman, thanks for the patch and a detailed explanation. 👍 I consider this approach to be correct since we do not have the new serial API yet.

@jamesbeyond, I'll prepare a fix for all tests that use SERIAL_FLUSH_TIME_MS based on @dustin-crossman's findings.

[dustin-crossman]

Great news! Thanks!