ESP8266HttpClient never seems to finish large files when using lwip2

[schlaegerz]

Basic Infos

Hardware

Hardware: NodeMCU 1.0 (ESP-12E)
Core Version: 2.4

Description

I am trying to download a file that is around 10MB, and on lwip2 the file download seems to get around 1MB done and then just stops. If i go back to 1.4 then it finishes the download, but takes about 4min.

Settings in IDE

Module: NodeMCU 1.0 (ESP-12E)
Flash Size: 4M
CPU Frequency: 160Mhz
Flash Mode:
Upload Using: Serial

Code:

while (success &&
http.connected() &&
(downloadRemaining > 0 || downloadRemaining == -1)
) {
// get available data size

    auto size = stream->available();
    if (size > 0) {
        auto c = stream->readBytes(cur_buffer, ((size > CHUNK_SIZE) ? CHUNK_SIZE : size));
        cur_buffer = cur_buffer + c;
        auto totalSize = cur_buffer - buff;
        if (totalSize + 2* CHUNK_SIZE > buffSize)
        {
            cur_buffer = buff;
        }
         
        hasDownload = true;
        success &= c > 0;
        if (downloadRemaining > 0) {
            downloadRemaining -= c;
        }
        lastDownloadTime = millis();
    }
    yield();
    //delay(1);
    static int i = 0;
    i++;
    if (i % 2000 == 0)
    {
        Serial.println(downloadRemaining);
    }
}

[d-a-v]

Could you please post a full sketch so the issue can be reproduced ?

[schlaegerz]

This sketch downloaded 2Mb for me and then stalled.

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>

#define USE_SERIAL Serial

ESP8266WiFiMulti WiFiMulti;


#define CHUNK_SIZE HTTP_TCP_BUFFER_SIZE/2
uint8_t *buff;
const int buffSize = 4096;
void setup() {

    USE_SERIAL.begin(115200);
    // USE_SERIAL.setDebugOutput(true);

    USE_SERIAL.println();
    USE_SERIAL.println();
    USE_SERIAL.println();

    for (uint8_t t = 4; t > 0; t--) {
        USE_SERIAL.printf("[SETUP] WAIT %d...\n", t);
        USE_SERIAL.flush();
        delay(1000);
    }

    WiFi.mode(WIFI_STA);
    WiFiMulti.addAP("SSID", "password");
    buff = new uint8_t[buffSize];

}



void loop() {
    // wait for WiFi connection
    if ((WiFiMulti.run() == WL_CONNECTED)) {

        HTTPClient http;

        USE_SERIAL.print("[HTTP] begin...\n");

        // configure server and url
        http.begin("http://ipv4.download.thinkbroadband.com/20MB.zip");
        //http.begin("192.168.1.12", 80, "/test.html");

        USE_SERIAL.print("[HTTP] GET...\n");
        // start connection and send HTTP header
        int httpCode = http.GET();
        if (httpCode > 0) {
            // HTTP header has been send and Server response header has been handled
            USE_SERIAL.printf("[HTTP] GET... code: %d\n", httpCode);

            // file found at server
            if (httpCode == HTTP_CODE_OK) {

                // get lenght of document (is -1 when Server sends no Content-Length header)
                int downloadRemaining = http.getSize();
                // create buffer for read
                uint8_t * cur_buffer = buff;
                // get tcp stream
                WiFiClient * stream = http.getStreamPtr();
                bool success = true;
                // read all data from server
                while (success &&
                    http.connected() &&
                    (downloadRemaining > 0 || downloadRemaining == -1)
                    ) {
                    // get available data size

                    auto size = stream->available();
                    if (size > 0) {
                        auto c = stream->readBytes(cur_buffer, ((size > CHUNK_SIZE) ? CHUNK_SIZE : size));
                        cur_buffer = cur_buffer + c;
                        auto totalSize = cur_buffer - buff;
                        if (totalSize + 2 * CHUNK_SIZE > buffSize)
                        {
                            cur_buffer = buff;
                        }

                        success &= c > 0;
                        if (downloadRemaining > 0) {
                            downloadRemaining -= c;
                        }
                    }
                    yield();
                    //delay(1);
                    static int i = 0;
                    i++;
                    if (i % 2000 == 0)
                    {
                        Serial.println(downloadRemaining);
                    }
                }


                USE_SERIAL.println();
                USE_SERIAL.print("[HTTP] connection closed or file end.\n");

            }
        }
        else {
            USE_SERIAL.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
        }

        http.end();
    }
    delay(1000);
}

This stalls with lwip2 but not lwip1.4

[d-a-v]

I have made my tests, no stall.

• lwip2/536: 307s@545kbits/s
• lwip2/1460: 152s@1100kbits/s
• lwip1/1460: 134s@1245kbits/s (<-- you have ~240s).

My downlink BW is ~20Mbits/s.
Did you try lwip2/1460 ?
How long did you wait before giving up the run (there are timers retriggering transfers)?
Could you make a tcp dump of the traffic ?

[schlaegerz]

I just ran it for 20 min until the http connection automatically closed, it downloaded less than half of the file.

When I tried lwip2/1460 it took about 670s

So it seems like the problem is more specifically with lwip2/536

What is the best way to get a tcp dump for this?

Also side note, is there something I can do to speed up the download I get nowhere near your speeds. On the same router my computer gets 30Mb/s

[d-a-v]

The website is 13 hops far from me. Try with

tcptraceroute ipv4.download.thinkbroadband.com 80

The farther peer is, the more chance you get lost packets (among tons of other reasons to get lost packets). TCP has to retransmit them. TCP has its limits, Designers set up tcp stacks with good parameters so it can just work. Under linux and I guess in all regular TCP stacks, the TCP window size (= the receive buffer) is at least 64KB.
With our ESP, it is ~6000 with MSS=1460 and ~2000 with MSS=536. I personally have no doubt about the ESP stability with lwIP-v2 (up to MSS=896) - one always may be wrong - and we cannot dedicate all RAM for tcp.
You still can tweak TCP constants and buffer size in lwipopts.h to try to make it more efficient (TCP_WND, TCP_SND_BUF, ...). Peer's tcp stack also has to be considered: for example a while ago when I was testing lwip2, I realized that there was an issue with linuxkernel-4.4 (I could try and trigger it again) in the fast retransmit algorithm. 2-way traffic just stopped with linux failing to honour the resend request.

In this issue, the slowness could be due to ESP's inability to store many / large holes so lots of retransmissions has to happen on a path with higher probability for lost packets. In these cases, the retransmit-timeout TCP algorithm is more often triggered which dramatically decreases performances.

If we want to precisely understand what is happening in this issue, tcp dumps are needed, but I'm afraid there's nothing more we can do than understand and/or dedicate all the RAM for tcp.

For tcp dumps, a way is to install and run tcpdump on your router (if it can), or wireshark on your PC if you can get the ESP traffic go through it (via an usb-wlan in AP mode).

[schlaegerz]

I'm 15 hops away, do you think that could account for the difference between our downloads? Me getting 5 times slower download seems pretty bad, is there anything else you did differently?

I'll work on getting a tcp dump to see if I can get any more info, but i think I'll just stick lwip1.4 for now since it seems like the faster lwip2 may be unstable?

[d-a-v]

I did nothing different, I can't. Everybody would do if it was possible.
I would consider lwip2/mss=1460 less unstable than lwip1 until I'm proven wrong, the "unstable"ness applies for both.
The average TCP BW can vary from run to run in same conditions.
What you could do is recompile lwip with different values for TCP_WND and report back here.

[schlaegerz]

OK, I can try that, what makes these unstable? Just so I know what to look out for, is it just lower RAM?

[d-a-v]

Can you please retry your sketch with replacing HTTP_TCP_BUFFER_SIZE (always 1460) by TCP_MSS (536 with lower memory or 1460 with higher bandwidth and lwip-1.4 ) ?

One other question about that constants: Is it a bug that HTTP_TCP_BUFFER_SIZE does not always match TCP_MSS?

[d-a-v]

Yes and no. It was initially set to MSS, and was not changed since MSS changed.
This is not required and should work with any large enough value.
What to do has not been discussed yet

• keep TCP_MSS for the buffer length (then add TCP_MSS definition when lwip1.4 is selected) and use it instead of 1460.
• decide what is "large enough" and use it
• use an on-demand buffer length

[d-a-v]

Regarding my previous comment, MSS=1460 is not at all unstable (I was responsible for saying that and I've proven myself wrong).
And after reading ESP8266HTTPClient.cpp again, there is absolutely no need to limit the buffer size to any fixed value. If the required len is mallocable, then we can use it even if it is greater than any MSS. We are dealing here with TCP (unbounded stream) not UDP (packet bounded).