[Zepyhr v1.14.0] Unable to update FW with mcumgr over UART

[PhilippFinke]

I'm using a nrf52840 as an USB BLE dongle on our custom board. Thereby I'm using a UART interface to update the FW with mcumgr (based on the smp_svr sample). Everything worked like a charm until I switched from Zephyr v1.13 to v1.14. Now I always get a "NMP timeout" when I try to communicate with the chip using mcumgr. I also had a configuration for the nrf52840_pca10056 board that worked perfectly with v1.13 but not under v1.14.0.

Here's the config I use:

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048

CONFIG_MCUMGR=y
CONFIG_BOOTLOADER_MCUBOOT=y

CONFIG_MCUMGR_SMP_SHELL=y

CONFIG_MCUMGR_CMD_IMG_MGMT=y
CONFIG_MCUMGR_CMD_OS_MGMT=y

CONFIG_BT=y
CONFIG_BT_HCI_RAW=y

CONFIG_USB=y
CONFIG_USB_DEVICE_VID=0x2FE3
CONFIG_USB_DEVICE_PID=0x100
CONFIG_USB_DEVICE_MANUFACTURER="My company"
CONFIG_USB_DEVICE_PRODUCT="USB-BLE-HCI"
CONFIG_USB_DEVICE_SN="0.01"

CONFIG_UART_INTERRUPT_DRIVEN=y
CONFIG_FLASH=y
CONFIG_WATCHDOG=n

My main.c is basically the same as the one of the smp_svr sample.

To Reproduce
Steps to reproduce the behavior:

1. build an flash mcuboot
2. build and flash app with above config for nrf52840_pca10056
3. mcumgr --conntype serial --connstring dev=,baud=115200 echo bla

Expected behavior
output should be "bla" but NMP timeout is received

Impact
showstopper

Screenshots or console output
mcumgr --conntype serial --connstring dev=/dev/ttyACM0,baud=115200 echo bla
Error: NMP timeout

Environment (please complete the following information):

• OS: macos, Linux
• Toolchain GNU gcc
• Zephyr: v1.14.0-branch (cebe115)
• mcuboot: 2dc9f8f

Additional context
Add any other context about the problem here.

[carlescufi]

Just to clarify, this is using which physical UART on the chip? Is it a CDC ACM UART that goes over USB or do you have an actual physical UART connected?

[PhilippFinke]

On the nrf52840_pca10056 I'm using CDC ACM UART (I think there is no other way) and on my custom board I'm using a physical UART. In both cases I'm able to connect and use the UART console (e.g. with minicom) but unable to use mcumgr with the serial connection type.

[carlescufi]

@PhilippFinke

On the nrf52840_pca10056 I'm using CDC ACM UART (I think there is no other way)

I assume you mean the JLink CDC ACM UART right?

[carlescufi]

@PhilippFinke just to make sure, can you try reproducing in current master?

[PhilippFinke]

With CDC ACM UART I mean the micro USB interface that is also used to flash the board.

I'm gonna give it a try with the Zephyr master.

[carlescufi]

@PhilippFinke 2 more things to take into account:

1. Can you try without MCUboot to see if there's an issue there?
2. When using the devkit, can you please take into account this: nrf52840_pca10056: Leading spurious 0x00 byte in UART output #14753

[PhilippFinke]

I tried running it with Zephyr master with and without mcuboot, but the problem still exists.

[carlescufi]

I will give this a try as soon as I can @PhilippFinke

[PhilippFinke]

@carlescufi any news here or still all quiet on the western front?

[carlescufi]

@PhilippFinke I bisected this today to find the culprit, which is ce6be86. This needs to be fixed of course, but in the meantime if you want you can disable the shell and use the UART mcumgr transport instead, by using the following in the prj.conf:

CONFIG_MCUMGR_SMP_SHELL=n
CONFIG_SHELL=n
CONFIG_MCUMGR_SMP_UART=y

[carlescufi]

This patch fixes the issue when using the shell transport instead of the UART one but I am sure it is not correct as it stands:

diff --git a/subsys/shell/shell.c b/subsys/shell/shell.c
index a1044e5de0..863de7a565 100644
--- a/subsys/shell/shell.c
+++ b/subsys/shell/shell.c
@@ -1178,8 +1178,12 @@ void shell_thread(void *shell_handle, void *arg_log_backend,
        }

        while (true) {
+
+               if (shell->iface->api->update) {
+                       shell->iface->api->update(shell->iface);
+               }
                /* waiting for all signals except SHELL_SIGNAL_TXDONE */
-               err = k_poll(shell->ctx->events, SHELL_SIGNAL_TXDONE,
+               err = k_poll(shell->ctx->events, SHELL_SIGNALS,
                             K_FOREVER);

                k_mutex_lock(&shell->ctx->wr_mtx, K_FOREVER);
@@ -1190,9 +1194,6 @@ void shell_thread(void *shell_handle, void *arg_log_backend,
                        return;
                }

-               if (shell->iface->api->update) {
-                       shell->iface->api->update(shell->iface);
-               }

                shell_signal_handle(shell, SHELL_SIGNAL_KILL, kill_handler);
                shell_signal_handle(shell, SHELL_SIGNAL_RXRDY, shell_process);

[PhilippFinke]

@carlescufi I switched to the UART based implementation. This is totally sufficient to me, because I don't need a UART console anyway. Thanks!

[mfiumara]

@carlescufi Does this mean the CONFIG_MCUMGR_SMP_SHELL option should be working in Zephyr master? I am trying to get it working but having similar problems as @PhilippFinke. The UART based implementation works correctly but when I try to use the mcumgr serial backend I get no response. Unfortunately for me it is not sufficient to use the UART based implementation and also need shell over UART. Should I create a separate issue for this?

[carlescufi]

@carlescufi Does this mean the CONFIG_MCUMGR_SMP_SHELL option should be working in Zephyr master? I am trying to get it working but having similar problems as @PhilippFinke. The UART based implementation works correctly but when I try to use the mcumgr serial backend I get no response. Unfortunately for me it is not sufficient to use the UART based implementation and also need shell over UART. Should I create a separate issue for this?

Yes, the SHELL option should be working on master. @nordic-krch fixed it a while back actually. You can create a separate issue but make sure that you are testing this correctly and that it works with the UART backend first, because it could be the mcumgr application not being correctly configured.

[nvlsianpu]

I checked that if you selected progressive erase it starts working CONFIG_IMG_ERASE_PROGRESSIVELY=y (it should work without it as well). @de-nordic I believe you can help us.

[nvlsianpu]

@carlescufi ^^ this is under investigation by @de-nordic. Let's wait till Monday for more results.

[de-nordic]

When shell is enabled, shell based mcumgr shall work correctly. Both should work correctly, but not at the same time, which is not problem with Zephyr but the problem comes from the fact that UART does have one buffer on host side, your PC side, which means that if you have terminal connected to uart and run mcumgr at the same time, then it is up to scheduling which process will collect data from RC buffer; if terminal collects data first, then MCUMGR will stuck thinking that there is problem with transmission.

Additionally, the transfer via UART may be quite slow and completion time may differ depending on size of the binary being transferred. For that reason you may want to add "-t " switch to mcumgr invocation, where is timeout in seconds.