UART locks and stops receiving data

[gonzabrusco]

Board

ESP32 Dev Module

Device Description

ESP32 Dev Module connected with USB directly to the PC.

Hardware Configuration

Standard Serial(0) configuration.

Version

latest master

IDE Name

Platformio IO

Operating System

Windows 10

Flash frequency

40 MHz

PSRAM enabled

no

Upload speed

115200

Description

Hello. I think I found a bug in HardwareSerial.

When you fill the receive buffer very fast when the code is locked in some other process, the serial stops receiving. It just stops working for RX (TX keeps working). It looks like UART RX is getting locked.

I must say that I'm using Platformio IO with Tasmota dev fork @Jason2866 (https://github.com/Jason2866/platform-espressif32) because Platformio still does not support Arduino 2.0/ IDF 4.4. But the platform-packages are being used directly from master like this:

platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.2.2/platform-tasmota-espressif32-2.0.2.zip
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git

Another thing that called my attention, if the default rx buffer is 256 bytes, why does the serial loopbacks MORE than 256 bytes? Where is it storing the extra characters? Are we looking at a buffer overflow problem?

If you send this (255 chars):
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has sur
It responds ok everythime.

If you send this (366 chars!!!)
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
It still responds ok everythime when It should truncate to 256 because the default RX buffer is 256.... (or I'm getting something wrong here?)

But if you send this (574 chars!!)
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
The Serial RX just locks forever.

I saw this because in my application I'm using the UART for communication with another microcontroller. And I noticed that after a long blocking part of the code, the Serial stopped working.

Sketch

#include <Arduino.h>

uint32_t startTime;

void setup() {
    Serial.begin(115200);

    startTime = millis();
}

void loop() {
    while(millis() - startTime < 5000) {
        // blocking loop
    }
    Serial.println();
    Serial.println("Finished blocking loop. Now printing everything received");

    while(Serial.available()) Serial.print((char)Serial.read());
    Serial.println();
    startTime = millis();
}

Debug Message

Run sketch and send the texts described before.

Other Steps to Reproduce

No.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[gonzabrusco]

With debug enabled. I get this: "UART0 FIFO Overflow. Flushing data. Consider adding Flow Control to your Application." but only when I send the text with 574 chars. Not the one with 366 chars (although the buffer is 256 chars).

Reviewing the code I wonder, why are you flushing all the input data when you have an overflow? Let my program decide what to do with that data. Why flush it? The buffer may be full but there still could be useful information in it.

[gonzabrusco]

More debugging info. If you change the buffer size to 1024.

Then you can send this text upto 3 times without problems (574x3 = 1722 chars!!).
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

But the forth time locks the UART forever. The curious part is that if you try to send the text again (just one time), the error log about the FIFO overflow appears multiple times.

It looks like:

1. The overflow error it not being resetted properly.
2. After the overflow the Serial locks forever.
3. After the overflow error, the FIFO size gets reduced? (the log appears with a lower number of chars sent).
4. DO NOT discard the information in the FIFO if it overflows. Let the user decide with that information received.

[Jason2866]

@gonzabrusco can you try with my latest build? There are changes regarding uart since 2.0.2

platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.2.2/platform-tasmota-espressif32-2.0.2.zip

[Jason2866]

Bleeding edge version, includes S3 support

platform = https://github.com/Jason2866/platform-espressif32.git#IDF44/ESP32-S3
platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/674/framework-arduinoespressif32-v4.4_dev-e8af343d93.tar.gz

[gonzabrusco]

@gonzabrusco can you try with my latest build? There are changes regarding uart since 2.0.2

platform = https://github.com/tasmota/platform-espressif32/releases/download/v2.0.2.2/platform-tasmota-espressif32-2.0.2.zip

That is what I'm using. I also tried commenting the line
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git but this still happens.

[gonzabrusco]

Bleeding edge version, includes S3 support

platform = https://github.com/Jason2866/platform-espressif32.git#IDF44/ESP32-S3
platform_packages = framework-arduinoespressif32 @ https://github.com/Jason2866/esp32-arduino-lib-builder/releases/download/674/framework-arduinoespressif32-v4.4_dev-e8af343d93.tar.gz

I'm using an ordinary ESP32 devkit. Not a new chip. Just the plain old ESP32

[gonzabrusco]

Ok. More findings: the size of the buffer is multiplied by 2 inside uartBegin() in esp32-hal-uart.c. That's why it wasn't overflowing with the expected amount of characters (by default the RX buffer is 256, but it overflows with 513 bytes). Is this an error or on purpose?

ESP_ERROR_CHECK(uart_driver_install(uart_nr, 2*queueLen, 0, 20, &(uart->uart_event_queue), 0));

If I comment the lines uart_flush_input(uart->num); inside uart_event_task(void *args) inside esp32-hal-uart.c, the Serial does not lock!

But what calls my attention is that even if I send more characters than the buffer size (256x2 = 512), the loopback now sends all the characters (although it overflowed). Is the internal IDF RX buffer bigger than 512? I don't get why the UART is not discarding the latest characters after the buffer overflow....

[SuGlider]

@gonzabrusco V2.0.2 has a bug in its UART implementation.
Please apply PR #6133 and this issue will be gone.

[gonzabrusco]

@SuGlider I'm using master. This issue happens with that PR applied.

[SuGlider]

Ok. I'll try to reproduce it.

[SuGlider]

@gonzabrusco Issue confimed. Thanks for reporting and investigating it!

I'll work on a fix that removes UART flushing. It is really causing issues.
I'll also add __weak__ functions for the events and allow the user to set its own functions if desired.
Default functions will just print error message, and do not flush any data.
This will allow the user to define what to do in UART each failure case.

Maybe I can work also on the std::function for onReceive(), also letting the user to overload error functions instead of using C __weak___. Not sure yet if I'll have the time to do all of it. Maybe in two steps. First fix the issue itself, then improve it.

Ok. More findings: the size of the buffer is multiplied by 2 inside uartBegin() in esp32-hal-uart.c. That's why it wasn't overflowing with the expected amount of characters (by default the RX buffer is 256, but it overflows with 513 bytes). Is this an error or on purpose?

It is on purpose. This is a general recommendation from IDF UART driver examples.
This is the size of the IDF ringBuffer (software buffer) that receives data fro UART ISR.
ESP32 has an internal FIFO with 127 bytes (hardware buffer)

The first error message "UART0 Buffer Full. Flushing data. Consider encreasing your buffer size of your Application." has to do with the software buffer (IDF RX ringBuffer).
The error message "UART0 FIFO Overflow. Flushing data. Consider adding Flow Control to your Application." has to do with the hardware buffer (ESP32 FIFO).

The way it works is that any data received on the UART is buffered by IDF UART ISR to the internal ringBuffer, which the user can set its size in advance (calling :setRxBufferSize() before calling Serial.begin()).
After the internal buffer is full, Arduino Event Task will issue the first error message "UART0 Buffer Full.".
Then it will populate the hardware FIFO up to 127 bytes (its limit) and issue the error message "UART0 FIFO Overflow."

[mrengineer7777]

@SuGlider Possibly related:
espressif/esp-idf#8445
espressif/esp-idf@708e3b6

[gonzabrusco]

Thanks for the help @SuGlider

One more thing I would like to say about the UART behaviour. When I commented those flush uart lines, I saw another weird behaviour in UART. It seems that the UART buffer starts to drop the lastest characters to arrive after it's full (expected behaviour, at least in my book haha), but the UART FIFO seems to work like a RingBuffer and drops the oldest characters and stores the newest characters. The result of this inconsistency is that when you read the serial buffer, you receive the oldest and the newest arrived characters, but you lose the characters "in the middle".

For example, If I send this content:
A B C D E F G H I J K
and the buffer overflows, you will read:
A B C D I J K

In my opinion, the UART FIFO should also drop the newest characters so it is consistent with the UART buffer. That way you would read
A B C D E F G

Let me know what you think.