Recover from serial errors when flashing (ESPTOOL-123)

[projectgus]

At high baud rates or in noisy environments, data is sometimes corrupted over the serial link when flashing. At the moment, this leads to fatal errors like the following:

Failed to write compressed data to flash after seq 0 (result was C100)

(0xC1 is ESP_BAD_DATA_CHECKSUM)

Reflashing usually succeeds, but esptool needs to be run again.

Ideally, esptool.py will be able to recover from this error automatically by re-sending the bad data block.

[rspenc29]

I am also having this issue very frequently. Baud rate doesn't seem to affect it. I've tried many common baud rates in the range of 9600 - 115200 and it happens on all of them. Sometimes I have to reflash like 10 times before it's finally successful. I've checked all the voltages as well and they are all what they should be.

Although I don't think this is the problem, I should also mention that I'm flashing it through a teensy 3.2 using the following code.

#define ESP8266 Serial1
#define ESP_RESET 4
#define ESP_FLASH 6
#define ESP_FLASHING true

void setup() {
    pinMode(ESP_RESET, OUTPUT);
    pinMode(ESP_FLASH, OUTPUT);
    Serial.begin(115200);
    ESP8266.begin(115200);

    if (ESP_FLASHING) {
        digitalWrite(ESP_FLASH, LOW);
        digitalWrite(ESP_RESET, LOW);
        delay(1000);
        digitalWrite(ESP_RESET, HIGH);
    } else {
        digitalWrite(ESP_FLASH, HIGH);
        digitalWrite(ESP_RESET, HIGH);
    }
}

void loop() {
    if (ESP8266.available()) {
         Serial.write(ESP8266.read());
    }

    if (Serial.available()) {
        ESP8266.write(Serial.read());
    }
}

[adamglowacki]

I am in a situation similar to @rspenc29. I need to be able flash ESP-WROOM-02 via another device. But sometimes a single byte gets lost (every ~10kB or even less often) and it stops the flashing operation. I tried to slightly modify the application to retry if a command failed but it's not as easy as I hoped. During stub uploading (cmd=0x07) a single byte got lost. ESP module responded with 0x0107 status bytes. So esptool.py (modified by me) sent the same packet again. This time no byte got lost. But the ESP module keeps responding with 0x0105 status. Could you tell me what these status bytes mean? I would also be very thankful for any hints/suggestions on how to make esptool.py retry some of the commands if they failed.

[adamglowacki]

I modified the "bridge device" to interpret the bootloader packets as they go from the PC to the ESP module and drop those that are too short. And I made my fork of esptool.py retry the commands if they failed. It is a temporary solution for me and I hope that soon we can have esptool.py recover from errors autonomously. The modification of esptool.py is available here: adamglowacki/esptool@440828a

[donpdonp]

@adamglowacki do you have the arduino code to drop the short packets? im using a teensy and get an upload failure at some point after a successful startup (chip id is good, stub upload is good), and I cannot complete an entire write of the flash ram.

[adamglowacki]

I don't have the Arduino code - the device I mentioned is based on a custom board with a microcontroller, software is written in plain C. But the algorithm itself is quite simple. I parse all the incoming packets and forward them to the ESP module only if their real length is the same as the one declared inside the packet.

I process every incoming byte one by one - detect the packet boundaries (See SLIP on Wikipedia) and at the same time unescape all the bytes inside the packet. When finally I get the unescaped packet, I compare the length field with the length of what I have just unescaped. I don't bother with computing the checksum or interpreting the command codes. The length-check is enough for me.
If the length field value seems to be valid, I forward the whole escaped packet to the ESP module. Out of laziness I collect the raw (escaped) bytes when unescaping so that I don't have to escape the bytes again when the packet has been "validated" by me. But you can escape the bytes "on-the-fly" when transmitting - this way you can save a little bit of RAM.

I limited the size of the unescaped packets to 1024B.

[dominikzaq]

I have a question. I created retransmission packets in esptool. Program(esptool) respond correct frame, but the problem is that md5 isn't correct .
Maybe someone knows why esploader cannot handle with retransmission data. (retransmission means=if the data will not be sent, I will send it again.)

Example:
send: [192, 0, 3, 16, 64, 200, 0, 0, 0, 0, 64,...
not receive correct data, so i send this same frame one more time
send: [192, 0, 3, 16, 64, 200, 0, 0, 0, 0, 64,...
respond correct [192, 1, 3, 2, 0, 0, 0, 0, 0, 0, 0, 192]
next data
...

after programming md5 does not match: /

md5
[192, 0, 19, 16, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 13, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192]
[192, 1, 3, 2, 0, 0, 0, 0, 0, 0, 0, 192]

[projectgus]

@dominikzaq I didn't see this post until just now. Continuing this discussion on the other issue you opened, #428.

[curlyz]

=)) If you are using a DIY board, this problem is due to the lack of a capacitor near the 3.3V of the module ( 10uf + 0.1uF ) , the reason being the flash chip when erase a page will cause a spike of voltage drop because the cap inside the module didn't handle it very well.

[adamglowacki]

@curlyz In our custom PCB there are parallel capacitors 10uF+100nF on 3.3V, nonetheless there are problems with communication. At least some of them (or all, I don't know) happen between the PC and the "bridge" device. Therefore correct handling of errors due to unreliable connections by esptool.py would be great for me.

[jakub-vesely]

I had the same issue. I flashed esp from VirtualBox (Debian on Windows). I tried to use different esp or restart the VirtualBox and finally helped to restart Windows.

[shivharis]

It would be highly desired to have an automatic retry of the failed block. Simply reducing the BAUD rate is not a solution since the operation time at lower baud rate is very high.