esp_modem simple_cmux_client doesn't work with SIM7670G (IDFGH-13823) #669
Comments
github-actions
bot
changed the title
|
Hi @morgana2313 I currently don't have any SIM7670 device with me, but the CMUX related command should work the same way as 7600 or 7672, which have been tested with cmux protocol. Could you please enable verbose level and share the log when you're setting up the connection? I'm mainly interested in the raw cmux protocol data. There're also some CMUX related options in the component Kconfig. This one could be relevant for the initial setup: esp-protocols/components/esp_modem/Kconfig Lines 30 to 37 in 8c4f392 (you can try to set it to say 250ms and see if it helps) |
|
Hello David, Thank you for your response!
Here is the verbose log: Searching for the bytestring f9 03 3f 01 1c f9 leads to:
Relevant config: |
|
I think that the code in BTW: the esp_modem/examples/modem_console doesn't support diff --git a/components/esp_modem/examples/modem_console/main/modem_console_main.cpp b/components/esp_modem/examples/modem_console/main/modem_console_main.cpp
index ac4b8f6..65fc899 100644
--- a/components/esp_modem/examples/modem_console/main/modem_console_main.cpp
+++ b/components/esp_modem/examples/modem_console/main/modem_console_main.cpp
@@ -205,8 +256,21 @@ extern "C" void app_main(void)
// init console REPL environment
esp_console_repl_config_t repl_config = ESP_CONSOLE_REPL_CONFIG_DEFAULT();
+
+#if defined(CONFIG_ESP_CONSOLE_UART_DEFAULT) || defined(CONFIG_ESP_CONSOLE_UART_CUSTOM)
esp_console_dev_uart_config_t uart_config = ESP_CONSOLE_DEV_UART_CONFIG_DEFAULT();
ESP_ERROR_CHECK(esp_console_new_repl_uart(&uart_config, &repl_config, &s_repl));
+#elif defined(CONFIG_ESP_CONSOLE_USB_CDC)
+ esp_console_dev_usb_cdc_config_t hw_config = ESP_CONSOLE_DEV_CDC_CONFIG_DEFAULT();
+ ESP_ERROR_CHECK(esp_console_new_repl_usb_cdc(&hw_config, &repl_config, &s_repl));
+#elif defined(CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG)
+ esp_console_dev_usb_serial_jtag_config_t hw_config = ESP_CONSOLE_DEV_USB_SERIAL_JTAG_CONFIG_DEFAULT();
+ ESP_ERROR_CHECK(esp_console_new_repl_usb_serial_jtag(&hw_config, &repl_config, &s_repl));
+#else
+#error Unsupported console type
+#endif
+
+
switch (esp_sleep_get_wakeup_cause()) {
case ESP_SLEEP_WAKEUP_TIMER: |
|
Thanks for testing this again and sharing the logs!
Right, the device seems to send SABM CMD (requests), rather then acknowledges our requests ( It looks like the problem could either be a conflict of roles (both parties are trying to be the initiator, not the module), or some low level protocol mismatch. Could you please try to send the CMUX AT command and check what the device sends after the --- a/components/esp_modem/src/esp_modem_uart.cpp
+++ b/components/esp_modem/src/esp_modem_uart.cpp
@@ -175,7 +175,9 @@ int UartTerminal::read(uint8_t *data, size_t len)
uart_get_buffered_data_len(uart.port, &length);
length = std::min(len, length);
if (length > 0) {
- return uart_read_bytes(uart.port, data, length, portMAX_DELAY);
+ int read_len = uart_read_bytes(uart.port, data, length, portMAX_DELAY);
+ ESP_LOG_BUFFER_HEXDUMP("read", data, read_len, ESP_LOG_INFO);
+ return read_len;
}
return 0;In case you won't see any data on UART after the
(applying the first one would likely help overcome the protocol error in the log:
|
Thanks for bringing this up! if you feel like fixing it yourself, we'll be happy to accept your PR, btw:-) |
morgana2313
changed the title
|
I've added all three patches. I've disabled the ESP_LOG_BUFFER_HEXDUMP in esp_modem_uart.cpp and added a TX one in I would expect that After If I turn off echo mode ( Then I started a new session (+modem reset) to get esp_modem out of CMUX mode: My logic analyser connected to the TX/RX ports confirms this: 
I think it's very weird that thd SIM7670G doesn't send any CMUX frames, but keeps responding to I've found a couple of curious hints:
|
|
Thanks for your extensive testing and investigation!
Also seen some references to that "user guide", but couldn't find one.
I'm afraid that's related to a different device, but still, we're sending the SABM's immediately after getting the |
I tried |
|
Two commits were added to the LilyGo ticket containing:
|
|
With this windows (!) software CMUX appears to work with the SIM7670G on a LilyGo T-SIM7670G s3, so it should be possible to analyse this further and see what the windows driver is doing differently. |
|
Nice, thanks for sharing! Noticed there's also a source in plain C (for linux) called gsm0710, suggesting it complies with standard CMUX spec. |
|
We noticed Some other questions:
|
I'd noticed this bug some times ago, but refrained from fixin it, finding it useful to enter CMUX mode while in CMUX already (AT command would fail, but the subsequent CMUX request(s) would re-enter CMUX). But I'll fix it in the end, after merging #612
esp-protocols/components/esp_modem/src/esp_modem_cmux.cpp Lines 140 to 142 in fbd296f (and your experiments suggest so, as you saw SABM request echo's but CMUX entry failed on missing SABM replies)
shouldn't have any reply buffer limit.
just checking the expected pattern, for |
|
|
Ad 3) if the pattern is empty, we just print out the first chunk are return or help for more details |
|
With the windows software LilyGo provided CMUX doesn't work with the default SIM767XM5A_B01V03_231207 firmware. However after a upgrade to SIM767XM5_B03V02_240914 CMUX works with windows. Tomorrow I'll try it with esp_modem. |
|
With modem firmware |
|
@morgana2313 sorry for not responding timely, have you resolved this issue? I was just about to look into it, as it seemed like we're almost there. Only looks like the device's sending the actual data before we initialize the CMUX completely. This change should fix the warnings: --- a/components/esp_modem/src/esp_modem_cmux.cpp
+++ b/components/esp_modem/src/esp_modem_cmux.cpp
@@ -123,7 +123,12 @@ bool CMux::data_available(uint8_t *data, size_t len)
{
if (data && (type & FT_UIH) == FT_UIH && len > 0 && dlci > 0) { // valid payload on a virtual term
int virtual_term = dlci - 1;
- if (virtual_term < MAX_TERMINALS_NUM && read_cb[virtual_term]) {
+ if (virtual_term < MAX_TERMINALS_NUM) {
+ if (read_cb[virtual_term] == nullptr) {
+ // ignore all virtual terminal's data before
+ ESP_LOG_BUFFER_HEXDUMP("CMUX Rx before init", data, len, ESP_LOG_DEBUG);
+ return true;
+ }
// Post partial data (or defragment to post on CMUX footer)
#ifdef DEFRAGMENT_CMUX_PAYLOAD |
|
No Problem! I managed to detect the This fix doesn't either: diff --git a/components/esp_modem/src/esp_modem_cmux.cpp b/components/esp_modem/src/esp_modem_cmux.cpp
index 9179101..39e824e 100644
--- a/components/esp_modem/src/esp_modem_cmux.cpp
+++ b/components/esp_modem/src/esp_modem_cmux.cpp
@@ -121,12 +121,29 @@ struct CMux::CMuxFrame {
bool CMux::data_available(uint8_t *data, size_t len)
{
if (data && (type & FT_UIH) == FT_UIH && len > 0 && dlci > 0) { // valid payload on a virtual term
int virtual_term = dlci - 1;
- if (virtual_term < MAX_TERMINALS_NUM && read_cb[virtual_term]) {
+ // if (virtual_term < MAX_TERMINALS_NUM && read_cb[virtual_term]) {
+
+ ESP_LOGI("DA","len:%d MAX_TERMINALS_NUM:%d read_cb[virtual_term=%d]%p", len, MAX_TERMINALS_NUM, virtual_term, (void*) &read_cb[virtual_term]);
+ ESP_LOG_BUFFER_HEXDUMP("DA", data, len, ESP_LOG_INFO);
+
+ const char QCRDY[10] = "\r\nQCRDY\r\n";
+ ESP_LOG_BUFFER_HEXDUMP("QC", QCRDY, 9, ESP_LOG_INFO);
+ if ((len == 9) && (strncmp((char *) data, QCRDY, std::min((size_t) 9, len)) == 0)) {
+ ESP_LOGI("DA","Ignored QCRDY");
+ return false;
+ }
+
+ if (virtual_term < MAX_TERMINALS_NUM) {
+ if (read_cb[virtual_term] == nullptr) {
+ // ignore all virtual terminal's data before
+ ESP_LOG_BUFFER_HEXDUMP("CMUX Rx before init", data, len, ESP_LOG_DEBUG);
+ return true;
+ }
// Post partial data (or defragment to post on CMUX footer)
#ifdef DEFRAGMENT_CMUX_PAYLOAD
if (payload_start == nullptr) {I experience another problem; after a day or so using a CMUX connection the modem loses it's connection. I have some questions:
I'm considering how to debug this pronlem to see what's going on... |
|
Need to return true here: ESP_LOG_BUFFER_HEXDUMP("QC", QCRDY, 9, ESP_LOG_INFO);
if ((len == 9) && (strncmp((char *) data, QCRDY, std::min((size_t) 9, len)) == 0)) {
ESP_LOGI("DA","Ignored QCRDY");
return true;
}This is causing the warning IMO, but I don't understand why the patch posted here didn't work. Note that
This works the same way as standard command and data mode, there's no auto retry. need to be performed on application layer with backoff/retry and checking connections. you can enable PPP keep alive and perhaps TCP keepalive or some means to periodically check connectivity. There're also two events, IP lost event from netif and connection drop from PPP.
This is device specific, but most devices drop the connection on CMUX exit. It's also possible to use CMUX manual modes and deal with modes on each virtual terminal separately. |
|
Ah, I was having a but of trouble understanding how to work with I removed my return after |
Can I just drop the CMUX instance and create a new one? Or tickle the existing instance into restarting?
Shouldn't |
It should, this: means The problem is that this needs to be ignored around the CMUX footer: diff --git a/components/esp_modem/src/esp_modem_cmux.cpp b/components/esp_modem/src/esp_modem_cmux.cpp
index c47e13b9..1612b727 100644
--- a/components/esp_modem/src/esp_modem_cmux.cpp
+++ b/components/esp_modem/src/esp_modem_cmux.cpp
@@ -142,7 +142,12 @@ bool CMux::data_available(uint8_t *data, size_t len)
sabm_ack = dlci;
} else if (data == nullptr && dlci > 0) {
int virtual_term = dlci - 1;
- if (virtual_term < MAX_TERMINALS_NUM && read_cb[virtual_term]) {
+ if (virtual_term < MAX_TERMINALS_NUM) {
+ if (read_cb[virtual_term] == nullptr) {
+ // ignore all virtual terminal's data before
+ ESP_LOG_BUFFER_HEXDUMP("CMUX Rx before init (footer)", data, len, ESP_LOG_DEBUG);
+ return true;
+ }
#ifdef DEFRAGMENT_CMUX_PAYLOAD
read_cb[virtual_term](payload_start, total_payload_size);
#endifThese two changes should remove the warnings.
CMUX exit is already part of that example, does it work for you?
I would need to see the logs. Haven't seen a device that sends SABM's periodically, but that's still possible I think. I'm also using SAMB frames to detect modes, maybe they could be used to implement some keepalive mechanism? |
|
Yes, that did it; no more diff --git a/components/esp_modem/src/esp_modem_cmux.cpp b/components/esp_modem/src/esp_modem_cmux.cpp
index 9179101..201b987 100644
--- a/components/esp_modem/src/esp_modem_cmux.cpp
+++ b/components/esp_modem/src/esp_modem_cmux.cpp
@@ -121,12 +121,17 @@ struct CMux::CMuxFrame {
bool CMux::data_available(uint8_t *data, size_t len)
{
if (data && (type & FT_UIH) == FT_UIH && len > 0 && dlci > 0) { // valid payload on a virtual term
int virtual_term = dlci - 1;
- if (virtual_term < MAX_TERMINALS_NUM && read_cb[virtual_term]) {
+
+ if (virtual_term < MAX_TERMINALS_NUM) {
+ if (read_cb[virtual_term] == nullptr) {
+ // ignore all virtual terminal's data before
+ ESP_LOG_BUFFER_HEXDUMP("CMUX Rx before init", data, len, ESP_LOG_INFO);
+ return true;
+ }
// Post partial data (or defragment to post on CMUX footer)
#ifdef DEFRAGMENT_CMUX_PAYLOAD
if (payload_start == nullptr) {
@@ -146,15 +151,25 @@ bool CMux::data_available(uint8_t *data, size_t len)
} else if (data == nullptr && dlci > 0) {
int virtual_term = dlci - 1;
+ #ifdef skip
if (virtual_term < MAX_TERMINALS_NUM && read_cb[virtual_term]) {
+ #else
+ if (virtual_term < MAX_TERMINALS_NUM) {
+ if (read_cb[virtual_term] == nullptr) {
+ // ignore all virtual terminal's data before
+ ESP_LOG_BUFFER_HEXDUMP("CMUX Rx before init (footer)", data, len, ESP_LOG_INFO);
+ return true;
+ }
+ #endif
#ifdef DEFRAGMENT_CMUX_PAYLOAD
read_cb[virtual_term](payload_start, total_payload_size);
#endif
You mean the part with I've setup a wat to log the problem with the lost connection. Now I have to wait until it occurs. |
the reason is that on the "footer" side we log data which is null -- sorry, just blindly copied the previous condition. here we should do something like: if (virtual_term < MAX_TERMINALS_NUM) {
if (read_cb[virtual_term] == nullptr) {
// ignore all virtual terminal's data before
ESP_LOGD("CMUX Rx before init (footer)", "payload of this frame has been ignored");
return true;
}
agree, just referencing another example (as this is more like a system level, it doens't belong to esp-modem) https://github.com/espressif/esp-protocols/blob/master/examples/esp_netif/multiple_netifs/main/ppp_connect_esp_modem.c
I usually disconnect the antenna to simulate this event. |
|
Ah, of course, your're absolutely right: |
|
What categoric reasons are there for a non-functioning CMUX ip uplink?
I think in all cases a way to re-establish the IP link is:
Do you agree? |
|
Here is the log of a moment when the ip uplink stopped functioning; a |
I wouldn't say non-functioning CMUX, rather non-functioning PPP, as it's the PPP session that would get dropped, CMUX just creates virtual terminals. (not important if the session is closed on virtual or a real physical terminal).
I'd rather use PPP keepalive (LCP timeouts) here. But I do agree overall with potential reasons and actions you mentioned above. |
I enabled these:
Like |
|
I think I got a functioning ppp watchdog with minimal but adequate actions. In pseudo code: static void on_ppp_changed(void *arg, esp_event_base_t event_base, int32_t event_id, void *event_data){
last_ppp_status = event_id;
}
#define WATCHDOG_TRY_INTERVAL 60 * NSEC_PER_SEC
#define WATCHDOG_MODEM_OK_TIMEOUT 2 * WD_INTERVAL + (5 * NSEC_PER_SEC)
cmux_task {
esp_event_handler_register(NETIF_PPP_STATUS, ESP_EVENT_ANY_ID, &on_ppp_changed, NULL);
reset_hq_init_cmux(); // simple_cmux_exanple
int32_t prev_ppp_status = last_ppp_status;
while(true){
if (dce->get_signal_quality(rssi, ber) == command_result::OK){
} else {
last_modem_ok_nsec = uptime_nsec();
if (rssi < 99){
// connected to network.
int32_t ppp_status = last_ppp_status;
if ((ppp_status != NETIF_PPP_PHASE_RUNNING) && (ppp_status == prev_ppp_status)){
ESP_LOGE(TAG, "PPP not OK; restart CMUX");
set_mode(COMMAND_MODE);
set_mode(CMUX_MODE);
} else
{ // test (http) connection.
elapsed_nsec = now_nsec - last_watchdog_try_fetch_nsec;
if (elapsed_nsec > WATCHDOG_TRY_INTERVAL)
fetch_watchdog_url(); // updates last_watchdog_try_fetch_nsec
}
prev_ppp_status = ppp_status;
} else ESP_LOGI(TAG,"Waiting for network signal.");
} else
{ // no communication with modem => modem dropped out of cmux mode?
elapsed_nsec = now_nsec - last_modem_ok_nsec;
ESP_LOGI(TAG,"Modem not responding: %.1f", elapsed_nsec / 1e9);
if (elapsed_nsec > WATCHDOG_MODEM_OK_TIMEOUT) {
ESP_LOGE(TAG,"WATCHDOG_MODEM_OK_TIMEOUT => reset_init_cmux");
vTaskDelay(pdMS_TO_TICKS(1 * MSEC_PER_SEC));
reset_init_cmux(); // includes mode => cmd => cmux
last_modem_ok_nsec = now_nsec;
}
}
elapsed_nsec = now_nsec - last_watchdog_fetch_ok_nsec;
#define WATCHDOG_FETCH_TIMEOUT 600 * NSEC_PER_SEC
if (elapsed_nsec > WATCHDOG_FETCH_TIMEOUT) {
ESP_LOGE(TAG,"WATCHDOG_FETCH_TIMEOUT => esp_restart");
vTaskDelay(pdMS_TO_TICKS(1 * MSEC_PER_SEC));
esp_restart();
}
vTaskDelay(pdMS_TO_TICKS(WD_INTERVAL * MSEC_PER_SEC));
}
} |
Nice, thanks for sharing! Do you think you can make it a real example (with the real actual code) and post a PR? I'll have to test it with various devices so would take some time to accept, but would be really helpful? |
espressif-bot
added
Status: Done
Resolution: Done
and removed
Status: Opened
|
I see two reasons for the How can I tell CMUX to just stop without informing the modem? Or set the timeout lower? |
1.3.0 Features - Add mode detection to the example (18f196f) - Support for pausing network in C-API (1db83cd) - Add support for pausing netif (247f168, espressif#699) Bug Fixes - Minor cleanup of pppos example (5e92990) - Fix PPP mode detection to accept LCP/conf (c989c6a) - Refine mode switch data->command (8b6ea33, espressif#692) - Detect serial ports properly (0cb59ff) - Fix CMUX enter to ignore URC before transition (1284f66, espressif#669)
Answers checklist.
What component are you using? If you choose Other, provide details in More Information.
esp_modem
component version
v1.1.0
IDF version.
v5.3.1-244-g4d0db7045d
More Information.
The
simple_cmux_clientdoesn't work:Failed to configure multiplexed command mode... exiting. Using printf statements I found thatcmux_term->init()inDTE::setup_cmux()returns false because of a timeout inCMux::init().I'm using a LilyGo LilyGo T SIM7670G S3 board and added some code to reset and PWR the SIM7670G. The same code + uart config works with
pppos_clientand get an ip number. If I skip thedce->set_modecode the correctOperator nameis displayed; so the uart is configured correctly.BTW: I think this issue category should be named "Issue: Installation, build or bug report"
The text was updated successfully, but these errors were encountered: