HTTPClient connection unexpectedly closed by underlying WiFiClient #5379

Jeroen88 · 2018-11-25T21:01:18Z

Basic Infos

This issue complies with the issue POLICY doc.
I have read the documentation at readthedocs and the issue is not addressed there.
I have tested that the issue is present in current master branch (aka latest git).
I have searched the issue tracker for a similar issue.
If there is a stack dump, I have decoded it.
I have filled out all fields below.

Platform

Hardware: [ESP-8266]
Core Version: [latest git]
Development Env: [Arduino IDE]
Operating System: [Ubuntu]

Settings in IDE

Module: [Nodemcu]
Flash Mode: [qio|dio|other]
Flash Size: [4MB/1MB]
lwip Variant: [v2 Higher Bandwidth]
Reset Method: [nodemcu]
Flash Frequency: [40Mhz]
CPU Frequency: [160MHz]
Upload Using: [SERIAL]
Upload Speed: [460800] (serial upload only)

Problem Description

I use ESP8266HTTPClient to POST data to my NodeJS Express server. I ::setReuse(true) to keep the connection alive. Even if I check if the client is ::connected() before writing to the client, I get a HTTPC_ERROR_SEND_HEADER_FAILED or a HTTPC_ERROR_SEND_PAYLOAD_FAILED on the 'even' calls.
I already found out that the 'odd' calls succeed, because if the state is 'disconnected' the client reconnects automatically. I also found out that a work around is not using ::setReuse(true) at all. If I ::begin() every time, the data is posted without any issues.
I already checked that ESP8266HTTPClient is not disconnecting (_client->stop() is never called), so it must be the underlying _client disconnecting.
I am using WiFiClientSecureBearSSL for transport.

Until recently reusing the connection worked fine.

Maybe issue #5375 is related?

MCVE Sketch

The sketch is too big to add

Debug Messages

Debug messages go here

The text was updated successfully, but these errors were encountered:

devyte · 2018-11-25T21:20:06Z

@Jeroen88 can you please try to reproduce with normal WiFiClient instead of the secure one?

Jeroen88 · 2018-11-25T21:44:36Z

@devyte that is somewhat complicated. My cloud server is set up not to accept insecure calls, and if I disable that my sensor network will fail to connect. Any test I could do on a secure connection?

Jeroen88 · 2018-11-25T22:04:42Z

@devyte I created an insecure route on my cloud server and sent data repeatedly reusing the connection with a normal WiFiClient without any problem!

devyte · 2018-11-25T22:04:51Z

I'm wondering if the disconnect happens at WiFiClient level or below, or above at ssl level. It's entirely different places to look at.
Axtls or bssl?
Can you reroduce the problem with an MCVE sketch that is standalone? E.g.:dummy cloud or test server that the sketch can POST to.

Jeroen88 · 2018-11-25T22:07:34Z

I will try...

Jeroen88 · 2018-11-25T22:58:25Z

@devyte It is not my most beautiful code (quick and dirty) but this shows the same behavior, even without reusing the connection! It is on a public url, so you can try it yourself.
The example uses a fingerprint for the connection, but I kept the certificate in for maybe other testing...

/**
   BasicHTTPSClient.ino
    Created on: 20.08.2018
*/

#include <Arduino.h>
#include <time.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>

#include <WiFiClientSecure.h>
// Fingerprint for demo URL, expires on June 2, 2019, needs to be updated well before this date
const uint8_t fingerprint[20] = {0x5A, 0xCF, 0xFE, 0xF0, 0xF1, 0xA6, 0xF4, 0x5F, 0xD2, 0x11, 0x11, 0xC6, 0x1D, 0x2F, 0x0E, 0xBC, 0x39, 0x8D, 0x50, 0xE0};

const char* rootCABuff = \
"-----BEGIN CERTIFICATE-----\n" \
"MIIF6TCCA9GgAwIBAgIQBeTcO5Q4qzuFl8umoZhQ4zANBgkqhkiG9w0BAQwFADCB\n" \
"iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \
"cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV\n" \
"BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQw\n" \
"OTEyMDAwMDAwWhcNMjQwOTExMjM1OTU5WjBfMQswCQYDVQQGEwJGUjEOMAwGA1UE\n" \
"CBMFUGFyaXMxDjAMBgNVBAcTBVBhcmlzMQ4wDAYDVQQKEwVHYW5kaTEgMB4GA1UE\n" \
"AxMXR2FuZGkgU3RhbmRhcmQgU1NMIENBIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IB\n" \
"DwAwggEKAoIBAQCUBC2meZV0/9UAPPWu2JSxKXzAjwsLibmCg5duNyj1ohrP0pIL\n" \
"m6jTh5RzhBCf3DXLwi2SrCG5yzv8QMHBgyHwv/j2nPqcghDA0I5O5Q1MsJFckLSk\n" \
"QFEW2uSEEi0FXKEfFxkkUap66uEHG4aNAXLy59SDIzme4OFMH2sio7QQZrDtgpbX\n" \
"bmq08j+1QvzdirWrui0dOnWbMdw+naxb00ENbLAb9Tr1eeohovj0M1JLJC0epJmx\n" \
"bUi8uBL+cnB89/sCdfSN3tbawKAyGlLfOGsuRTg/PwSWAP2h9KK71RfWJ3wbWFmV\n" \
"XooS/ZyrgT5SKEhRhWvzkbKGPym1bgNi7tYFAgMBAAGjggF1MIIBcTAfBgNVHSME\n" \
"GDAWgBRTeb9aqitKz1SA4dibwJ3ysgNmyzAdBgNVHQ4EFgQUs5Cn2MmvTs1hPJ98\n" \
"rV1/Qf1pMOowDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYD\n" \
"VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCIGA1UdIAQbMBkwDQYLKwYBBAGy\n" \
"MQECAhowCAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNl\n" \
"cnRydXN0LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNy\n" \
"bDB2BggrBgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRy\n" \
"dXN0LmNvbS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZ\n" \
"aHR0cDovL29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAWGf9\n" \
"crJq13xhlhl+2UNG0SZ9yFP6ZrBrLafTqlb3OojQO3LJUP33WbKqaPWMcwO7lWUX\n" \
"zi8c3ZgTopHJ7qFAbjyY1lzzsiI8Le4bpOHeICQW8owRc5E69vrOJAKHypPstLbI\n" \
"FhfFcvwnQPYT/pOmnVHvPCvYd1ebjGU6NSU2t7WKY28HJ5OxYI2A25bUeo8tqxyI\n" \
"yW5+1mUfr13KFj8oRtygNeX56eXVlogMT8a3d2dIhCe2H7Bo26y/d7CQuKLJHDJd\n" \
"ArolQ4FCR7vY4Y8MDEZf7kYzawMUgtN+zY+vkNaOJH1AQrRqahfGlZfh8jjNp+20\n" \
"J0CT33KpuMZmYzc4ZCIwojvxuch7yPspOqsactIGEk72gtQjbz7Dk+XYtsDe3CMW\n" \
"1hMwt6CaDixVBgBwAc/qOR2A24j3pSC4W/0xJmmPLQphgzpHphNULB7j7UTKvGof\n" \
"KA5R2d4On3XNDgOVyvnFqSot/kGkoUeuDcL5OWYzSlvhhChZbH2UF3bkRYKtcCD9\n" \
"0m9jqNf6oDP6N8v3smWe2lBvP+Sn845dWDKXcCMu5/3EFZucJ48y7RetWIExKREa\n" \
"m9T8bJUox04FB6b9HbwZ4ui3uRGKLXASUoWNjDNKD/yZkuBjcNqllEdjB+dYxzFf\n" \
"BT02Vf6Dsuimrdfp5gJ0iHRc2jTbkNJtUQoj1iM=\n" \
"-----END CERTIFICATE-----\n" \
"-----BEGIN CERTIFICATE-----\n" \
"MIIFdzCCBF+gAwIBAgIQE+oocFv07O0MNmMJgGFDNjANBgkqhkiG9w0BAQwFADBv\n" \
"MQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFk\n" \
"ZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBF\n" \
"eHRlcm5hbCBDQSBSb290MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFow\n" \
"gYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtK\n" \
"ZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYD\n" \
"VQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIICIjAN\n" \
"BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgBJlFzYOw9sIs9CsVw127c0n00yt\n" \
"UINh4qogTQktZAnczomfzD2p7PbPwdzx07HWezcoEStH2jnGvDoZtF+mvX2do2NC\n" \
"tnbyqTsrkfjib9DsFiCQCT7i6HTJGLSR1GJk23+jBvGIGGqQIjy8/hPwhxR79uQf\n" \
"jtTkUcYRZ0YIUcuGFFQ/vDP+fmyc/xadGL1RjjWmp2bIcmfbIWax1Jt4A8BQOujM\n" \
"8Ny8nkz+rwWWNR9XWrf/zvk9tyy29lTdyOcSOk2uTIq3XJq0tyA9yn8iNK5+O2hm\n" \
"AUTnAU5GU5szYPeUvlM3kHND8zLDU+/bqv50TmnHa4xgk97Exwzf4TKuzJM7UXiV\n" \
"Z4vuPVb+DNBpDxsP8yUmazNt925H+nND5X4OpWaxKXwyhGNVicQNwZNUMBkTrNN9\n" \
"N6frXTpsNVzbQdcS2qlJC9/YgIoJk2KOtWbPJYjNhLixP6Q5D9kCnusSTJV882sF\n" \
"qV4Wg8y4Z+LoE53MW4LTTLPtW//e5XOsIzstAL81VXQJSdhJWBp/kjbmUZIO8yZ9\n" \
"HE0XvMnsQybQv0FfQKlERPSZ51eHnlAfV1SoPv10Yy+xUGUJ5lhCLkMaTLTwJUdZ\n" \
"+gQek9QmRkpQgbLevni3/GcV4clXhB4PY9bpYrrWX1Uu6lzGKAgEJTm4Diup8kyX\n" \
"HAc/DVL17e8vgg8CAwEAAaOB9DCB8TAfBgNVHSMEGDAWgBStvZh6NLQm9/rEJlTv\n" \
"A73gJMtUGjAdBgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/\n" \
"BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wEQYDVR0gBAowCDAGBgRVHSAAMEQGA1Ud\n" \
"HwQ9MDswOaA3oDWGM2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9BZGRUcnVzdEV4\n" \
"dGVybmFsQ0FSb290LmNybDA1BggrBgEFBQcBAQQpMCcwJQYIKwYBBQUHMAGGGWh0\n" \
"dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggEBAJNl9jeD\n" \
"lQ9ew4IcH9Z35zyKwKoJ8OkLJvHgwmp1ocd5yblSYMgpEg7wrQPWCcR23+WmgZWn\n" \
"RtqCV6mVksW2jwMibDN3wXsyF24HzloUQToFJBv2FAY7qCUkDrvMKnXduXBBP3zQ\n" \
"YzYhBx9G/2CkkeFnvN4ffhkUyWNnkepnB2u0j4vAbkN9w6GAbLIevFOFfdyQoaS8\n" \
"Le9Gclc1Bb+7RrtubTeZtv8jkpHGbkD4jylW6l/VXxRTrPBPYer3IsynVgviuDQf\n" \
"Jtl7GQVoP7o81DgGotPmjw7jtHFtQELFhLRAlSv0ZaBIefYdgWOWnU914Ph85I6p\n" \
"0fKtirOMxyHNwu8=\n" \
"-----END CERTIFICATE-----\n";

void setClock() {
  configTime(0, 0, "pool.ntp.org", "time.nist.gov");

  Serial.print(F("Waiting for NTP time sync: "));
  time_t nowSecs = time(nullptr);
  while (nowSecs < 8 * 3600 * 2) {
    delay(500);
    Serial.print(F("."));
    yield();
    nowSecs = time(nullptr);
  }

  Serial.println();
  struct tm timeinfo;
  gmtime_r(&nowSecs, &timeinfo);
  Serial.print(F("Current time: "));
  Serial.print(asctime(&timeinfo));
}


ESP8266WiFiMulti WiFiMulti;

BearSSL::WiFiClientSecure client;
HTTPClient https;


void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("SSID", "PASSWORD");

  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {
    setClock();
  } else {
    Serial.println("Can not connect to WiFi");
  }


  client.setFingerprint(fingerprint);
  
//  https.setReuse(true);

  Serial.print("[HTTPS] begin...\n");
  if (https.begin(client, "https://jigsaw.w3.org/HTTP/connection.html")) {  // HTTPS
    Serial.println("[HTTPS] client began");
  } else {
    Serial.printf("[HTTPS] Unable to connect\n");
  }
}

void loop() {
  client.setFingerprint(fingerprint);
  if (https.begin(client, "https://jigsaw.w3.org/HTTP/connection.html")) {  // HTTPS
    Serial.println("[HTTPS] client began");
  } else {
    Serial.printf("[HTTPS] Unable to connect\n");
    delay(10000);
    return;
  }

  Serial.print("[HTTPS] GET...\n");
  // start connection and send HTTP header
  int httpCode = https.GET();

  // httpCode will be negative on error
  if (httpCode > 0) {
    // HTTP header has been send and Server response header has been handled
    Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

    // file found at server
    if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
      String payload = https.getString();
      Serial.println(payload);
    }
  } else {
    Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
  }

  delay(3000);
}

This gives the following output:

[HTTPS] begin...
[HTTPS] client began
[HTTPS] client began
[HTTPS] GET...
[HTTPS] GET... code: 200
<HTML>
<HEAD>
  <!-- Created with AOLpress/2.0 -->
  <TITLE>Connection Header</TITLE>
</HEAD>
<BODY>
<P>
<IMG ALT="Jigsaw" BORDER="0" WIDTH="212" HEIGHT="49" SRC="/icons/jigsaw">
<H1>
  The <I>Connection</I> header
</H1>
<P>
This page will be served to you with the following headers:
<P>
<CODE>ExtensionHeader: ExtensionValue<BR>
Connection: ExtensionHeader</CODE>
<P>
If you're getting this page through a proxy, you should <I>not</I> see the
<I>ExtensionHeader</I> !
<P>
  <HR>
<BR>
<A HREF="mailto:jigsaw@w3.org">jigsaw@w3.org</A>
</BODY></HTML>

[HTTPS] client began
[HTTPS] GET...
[HTTPS] GET... failed, error: connection refused
[HTTPS] client began
[HTTPS] GET...
[HTTPS] GET... code: 200
<HTML>
<HEAD>
  <!-- Created with AOLpress/2.0 -->
  <TITLE>Connection Header</TITLE>
</HEAD>
<BODY>
<P>
<IMG ALT="Jigsaw" BORDER="0" WIDTH="212" HEIGHT="49" SRC="/icons/jigsaw">
<H1>
  The <I>Connection</I> header
</H1>
<P>
This page will be served to you with the following headers:
<P>
<CODE>ExtensionHeader: ExtensionValue<BR>
Connection: ExtensionHeader</CODE>
<P>
If you're getting this page through a proxy, you should <I>not</I> see the
<I>ExtensionHeader</I> !
<P>
  <HR>
<BR>
<A HREF="mailto:jigsaw@w3.org">jigsaw@w3.org</A>
</BODY></HTML>

[HTTPS] client began
[HTTPS] GET...
[HTTPS] GET... failed, error: connection refused

And this sequence is repeated

devyte · 2018-11-26T00:29:21Z

@earlephilhower is this enough for you to make an assessment?

Jeroen88 · 2018-11-26T06:48:22Z

I saw that I forgot to call https.end() at the end of the loop (just before delay()). I added it, but this does not make a difference.

earlephilhower · 2018-11-28T23:01:07Z

Debugging shows the problem clearly. Thanks, @d-a-v for adding the simple line.

BSSL: Connection *will* fail, no authentication method is set up.

This stems from what I believe to be a bad design choice, namely clearing all authentication options from the BSSL client object on a close. HTTPClient et. al. can no longer than use the client object after a ::stop has been called. I'll open a PR to undo that change.

Many objects now expect a WiFiClient* object to be passed to them and potentially re-used multiple times (HTTPClient, others). Clearing the authentication options on a ::stop means they can never reconnect. Remove the option clearing in ::stop Fixes esp8266#5379

earlephilhower · 2018-11-28T23:09:42Z

@Jeroen88 please try the PR. It's 1 line you can hand-apply if needed. The example you sent is fixed with it. This would be unrelated to #5375 because this was purely an issue in the BSSL client.

Many objects now expect a WiFiClient* object to be passed to them and potentially re-used multiple times (HTTPClient, others). Clearing the authentication options on a ::stop means they can never reconnect. Remove the option clearing in ::stop Fixes #5379

Jeroen88 · 2018-11-29T07:08:27Z

@earlephilhower thnx for the PR, I will try it next weekend!

Jeroen88 · 2018-12-02T10:53:07Z

@earlephilhower I can confirm reusing the connection is also fixed in my sketch. Thank you!

devyte added the waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. label Nov 25, 2018

devyte assigned earlephilhower Nov 26, 2018

earlephilhower mentioned this issue Nov 28, 2018

Don't clear authentication options on a ::stop #5386

Merged

earlephilhower added this to the 2.5.0 milestone Nov 28, 2018

earlephilhower closed this as completed in #5386 Nov 29, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPClient connection unexpectedly closed by underlying WiFiClient #5379

HTTPClient connection unexpectedly closed by underlying WiFiClient #5379

Jeroen88 commented Nov 25, 2018

devyte commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

devyte commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

devyte commented Nov 26, 2018

Jeroen88 commented Nov 26, 2018 •

edited

Loading

earlephilhower commented Nov 28, 2018

earlephilhower commented Nov 28, 2018

Jeroen88 commented Nov 29, 2018

Jeroen88 commented Dec 2, 2018

HTTPClient connection unexpectedly closed by underlying WiFiClient #5379

HTTPClient connection unexpectedly closed by underlying WiFiClient #5379

Comments

Jeroen88 commented Nov 25, 2018

Basic Infos

Platform

Settings in IDE

Problem Description

MCVE Sketch

Debug Messages

devyte commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

devyte commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

Jeroen88 commented Nov 25, 2018

devyte commented Nov 26, 2018

Jeroen88 commented Nov 26, 2018 • edited Loading

earlephilhower commented Nov 28, 2018

earlephilhower commented Nov 28, 2018

Jeroen88 commented Nov 29, 2018

Jeroen88 commented Dec 2, 2018

Jeroen88 commented Nov 26, 2018 •

edited

Loading