Error Invalid Handshake with websites other than api.github.com using HTTPSRequestCACert

[jtmoderate876]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [latest git hash or date]
• Development Env: [Arduino IDE]
• Operating System: [Windows]

Settings in IDE

Debug level set to everything, SSL TLS_MEM+...

Problem Description

Cannot connect to some sites with HTTPSRequestCACert.
With debug on I see Invalid Handshake when it tries to Receive Certificate (see debug log below).

I can get it to work with some sites such as api.github.com
But not online.wonderware.com
I've also tried and had trouble with others.

It sounds similar to this log:
#3661
Where that user recommends a change to axtls but I don't have the skills.

Sketch

#include <time.h>
#include <ESP8266WiFi.h>

const char* ssid = "myssid";
const char* password = "mypassword";

const int httpsPort = 443;
const char* host = "online.wonderware.com";
//note: i modified CACert with linux xxd -i DigiCertHighAssuranceEVRootCA.crt.der >cacert.h
// Root certificate used by api.github.com.
// Defined in "CACert" tab.
extern const unsigned char caCert[] PROGMEM;
extern const unsigned int caCertLen;

WiFiClientSecure client;

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Synchronize time useing SNTP. This is necessary to verify that
  // the TLS certificates offered by the server are currently valid.
  Serial.print("Setting time using SNTP");
  configTime(8 * 3600, 0, "pool.ntp.org", "time.nist.gov");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));

  // Load root certificate in DER format into WiFiClientSecure object
  bool res = client.setCACert_P(caCert, caCertLen);
  if (!res) {
    Serial.println("Failed to load root CA certificate!");
    while (true) {
      yield();
    }
  }
}

void loop() {
  // Connect to remote server
  Serial.print("connecting to ");
  Serial.println(host);
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  // Verify validity of server's certificate
  if (client.verifyCertChain(host)) {
    Serial.println("Server certificate verified");
  } else {
    Serial.println("ERROR: certificate verification failed!");
    return;
  }

  String url = "/repos/esp8266/Arduino/commits/master/status";
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');
  if (line.startsWith("{\"state\":\"success\"")) {
    Serial.println("esp8266/Arduino CI successfull!");
  } else {
    Serial.println("esp8266/Arduino CI has failed");
  }
  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println();

  static int repeat = 0;
  if (++repeat == 3) {
    Serial.println("Done");
    while (true) {
      delay(1000);
    }
  }
  delay(60000);
}

and the CACert for this site is:

CACert

const unsigned char caCert[] PROGMEM = {
  0x30, 0x82, 0x03, 0xfe, 0x30, 0x82, 0x02, 0xe6, 0xa0, 0x03, 0x02, 0x01,
  0x02, 0x02, 0x10, 0x15, 0xac, 0x6e, 0x94, 0x19, 0xb2, 0x79, 0x4b, 0x41,
  0xf6, 0x27, 0xa9, 0xc3, 0x18, 0x0f, 0x1f, 0x30, 0x0d, 0x06, 0x09, 0x2a,
  0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b, 0x05, 0x00, 0x30, 0x81,
  0x98, 0x31, 0x0b, 0x30, 0x09, 0x06, 0x03, 0x55, 0x04, 0x06, 0x13, 0x02,
  0x55, 0x53, 0x31, 0x16, 0x30, 0x14, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x13,
  0x0d, 0x47, 0x65, 0x6f, 0x54, 0x72, 0x75, 0x73, 0x74, 0x20, 0x49, 0x6e,
  0x63, 0x2e, 0x31, 0x39, 0x30, 0x37, 0x06, 0x03, 0x55, 0x04, 0x0b, 0x13,
  0x30, 0x28, 0x63, 0x29, 0x20, 0x32, 0x30, 0x30, 0x38, 0x20, 0x47, 0x65,
  0x6f, 0x54, 0x72, 0x75, 0x73, 0x74, 0x20, 0x49, 0x6e, 0x63, 0x2e, 0x20,
  0x2d, 0x20, 0x46, 0x6f, 0x72, 0x20, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72,
  0x69, 0x7a, 0x65, 0x64, 0x20, 0x75, 0x73, 0x65, 0x20, 0x6f, 0x6e, 0x6c,
  0x79, 0x31, 0x36, 0x30, 0x34, 0x06, 0x03, 0x55, 0x04, 0x03, 0x13, 0x2d,
  0x47, 0x65, 0x6f, 0x54, 0x72, 0x75, 0x73, 0x74, 0x20, 0x50, 0x72, 0x69,
  0x6d, 0x61, 0x72, 0x79, 0x20, 0x43, 0x65, 0x72, 0x74, 0x69, 0x66, 0x69,
  0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x20, 0x41, 0x75, 0x74, 0x68, 0x6f,
  0x72, 0x69, 0x74, 0x79, 0x20, 0x2d, 0x20, 0x47, 0x33, 0x30, 0x1e, 0x17,
  0x0d, 0x30, 0x38, 0x30, 0x34, 0x30, 0x32, 0x30, 0x30, 0x30, 0x30, 0x30,
  0x30, 0x5a, 0x17, 0x0d, 0x33, 0x37, 0x31, 0x32, 0x30, 0x31, 0x32, 0x33,
  0x35, 0x39, 0x35, 0x39, 0x5a, 0x30, 0x81, 0x98, 0x31, 0x0b, 0x30, 0x09,
  0x06, 0x03, 0x55, 0x04, 0x06, 0x13, 0x02, 0x55, 0x53, 0x31, 0x16, 0x30,
  0x14, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x13, 0x0d, 0x47, 0x65, 0x6f, 0x54,
  0x72, 0x75, 0x73, 0x74, 0x20, 0x49, 0x6e, 0x63, 0x2e, 0x31, 0x39, 0x30,
  0x37, 0x06, 0x03, 0x55, 0x04, 0x0b, 0x13, 0x30, 0x28, 0x63, 0x29, 0x20,
  0x32, 0x30, 0x30, 0x38, 0x20, 0x47, 0x65, 0x6f, 0x54, 0x72, 0x75, 0x73,
  0x74, 0x20, 0x49, 0x6e, 0x63, 0x2e, 0x20, 0x2d, 0x20, 0x46, 0x6f, 0x72,
  0x20, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x65, 0x64, 0x20,
  0x75, 0x73, 0x65, 0x20, 0x6f, 0x6e, 0x6c, 0x79, 0x31, 0x36, 0x30, 0x34,
  0x06, 0x03, 0x55, 0x04, 0x03, 0x13, 0x2d, 0x47, 0x65, 0x6f, 0x54, 0x72,
  0x75, 0x73, 0x74, 0x20, 0x50, 0x72, 0x69, 0x6d, 0x61, 0x72, 0x79, 0x20,
  0x43, 0x65, 0x72, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f,
  0x6e, 0x20, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x74, 0x79, 0x20,
  0x2d, 0x20, 0x47, 0x33, 0x30, 0x82, 0x01, 0x22, 0x30, 0x0d, 0x06, 0x09,
  0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01, 0x05, 0x00, 0x03,
  0x82, 0x01, 0x0f, 0x00, 0x30, 0x82, 0x01, 0x0a, 0x02, 0x82, 0x01, 0x01,
  0x00, 0xdc, 0xe2, 0x5e, 0x62, 0x58, 0x1d, 0x33, 0x57, 0x39, 0x32, 0x33,
  0xfa, 0xeb, 0xcb, 0x87, 0x8c, 0xa7, 0xd4, 0x4a, 0xdd, 0x06, 0x88, 0xea,
  0x64, 0x8e, 0x31, 0x98, 0xa5, 0x38, 0x90, 0x1e, 0x98, 0xcf, 0x2e, 0x63,
  0x2b, 0xf0, 0x46, 0xbc, 0x44, 0xb2, 0x89, 0xa1, 0xc0, 0x28, 0x0c, 0x49,
  0x70, 0x21, 0x95, 0x9f, 0x64, 0xc0, 0xa6, 0x93, 0x12, 0x02, 0x65, 0x26,
  0x86, 0xc6, 0xa5, 0x89, 0xf0, 0xfa, 0xd7, 0x84, 0xa0, 0x70, 0xaf, 0x4f,
  0x1a, 0x97, 0x3f, 0x06, 0x44, 0xd5, 0xc9, 0xeb, 0x72, 0x10, 0x7d, 0xe4,
  0x31, 0x28, 0xfb, 0x1c, 0x61, 0xe6, 0x28, 0x07, 0x44, 0x73, 0x92, 0x22,
  0x69, 0xa7, 0x03, 0x88, 0x6c, 0x9d, 0x63, 0xc8, 0x52, 0xda, 0x98, 0x27,
  0xe7, 0x08, 0x4c, 0x70, 0x3e, 0xb4, 0xc9, 0x12, 0xc1, 0xc5, 0x67, 0x83,
  0x5d, 0x33, 0xf3, 0x03, 0x11, 0xec, 0x6a, 0xd0, 0x53, 0xe2, 0xd1, 0xba,
  0x36, 0x60, 0x94, 0x80, 0xbb, 0x61, 0x63, 0x6c, 0x5b, 0x17, 0x7e, 0xdf,
  0x40, 0x94, 0x1e, 0xab, 0x0d, 0xc2, 0x21, 0x28, 0x70, 0x88, 0xff, 0xd6,
  0x26, 0x6c, 0x6c, 0x60, 0x04, 0x25, 0x4e, 0x55, 0x7e, 0x7d, 0xef, 0xbf,
  0x94, 0x48, 0xde, 0xb7, 0x1d, 0xdd, 0x70, 0x8d, 0x05, 0x5f, 0x88, 0xa5,
  0x9b, 0xf2, 0xc2, 0xee, 0xea, 0xd1, 0x40, 0x41, 0x6d, 0x62, 0x38, 0x1d,
  0x56, 0x06, 0xc5, 0x03, 0x47, 0x51, 0x20, 0x19, 0xfc, 0x7b, 0x10, 0x0b,
  0x0e, 0x62, 0xae, 0x76, 0x55, 0xbf, 0x5f, 0x77, 0xbe, 0x3e, 0x49, 0x01,
  0x53, 0x3d, 0x98, 0x25, 0x03, 0x76, 0x24, 0x5a, 0x1d, 0xb4, 0xdb, 0x89,
  0xea, 0x79, 0xe5, 0xb6, 0xb3, 0x3b, 0x3f, 0xba, 0x4c, 0x28, 0x41, 0x7f,
  0x06, 0xac, 0x6a, 0x8e, 0xc1, 0xd0, 0xf6, 0x05, 0x1d, 0x7d, 0xe6, 0x42,
  0x86, 0xe3, 0xa5, 0xd5, 0x47, 0x02, 0x03, 0x01, 0x00, 0x01, 0xa3, 0x42,
  0x30, 0x40, 0x30, 0x0f, 0x06, 0x03, 0x55, 0x1d, 0x13, 0x01, 0x01, 0xff,
  0x04, 0x05, 0x30, 0x03, 0x01, 0x01, 0xff, 0x30, 0x0e, 0x06, 0x03, 0x55,
  0x1d, 0x0f, 0x01, 0x01, 0xff, 0x04, 0x04, 0x03, 0x02, 0x01, 0x06, 0x30,
  0x1d, 0x06, 0x03, 0x55, 0x1d, 0x0e, 0x04, 0x16, 0x04, 0x14, 0xc4, 0x79,
  0xca, 0x8e, 0xa1, 0x4e, 0x03, 0x1d, 0x1c, 0xdc, 0x6b, 0xdb, 0x31, 0x5b,
  0x94, 0x3e, 0x3f, 0x30, 0x7f, 0x2d, 0x30, 0x0d, 0x06, 0x09, 0x2a, 0x86,
  0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b, 0x05, 0x00, 0x03, 0x82, 0x01,
  0x01, 0x00, 0x2d, 0xc5, 0x13, 0xcf, 0x56, 0x80, 0x7b, 0x7a, 0x78, 0xbd,
  0x9f, 0xae, 0x2c, 0x99, 0xe7, 0xef, 0xda, 0xdf, 0x94, 0x5e, 0x09, 0x69,
  0xa7, 0xe7, 0x6e, 0x68, 0x8c, 0xbd, 0x72, 0xbe, 0x47, 0xa9, 0x0e, 0x97,
  0x12, 0xb8, 0x4a, 0xf1, 0x64, 0xd3, 0x39, 0xdf, 0x25, 0x34, 0xd4, 0xc1,
  0xcd, 0x4e, 0x81, 0xf0, 0x0f, 0x04, 0xc4, 0x24, 0xb3, 0x34, 0x96, 0xc6,
  0xa6, 0xaa, 0x30, 0xdf, 0x68, 0x61, 0x73, 0xd7, 0xf9, 0x8e, 0x85, 0x89,
  0xef, 0x0e, 0x5e, 0x95, 0x28, 0x4a, 0x2a, 0x27, 0x8f, 0x10, 0x8e, 0x2e,
  0x7c, 0x86, 0xc4, 0x02, 0x9e, 0xda, 0x0c, 0x77, 0x65, 0x0e, 0x44, 0x0d,
  0x92, 0xfd, 0xfd, 0xb3, 0x16, 0x36, 0xfa, 0x11, 0x0d, 0x1d, 0x8c, 0x0e,
  0x07, 0x89, 0x6a, 0x29, 0x56, 0xf7, 0x72, 0xf4, 0xdd, 0x15, 0x9c, 0x77,
  0x35, 0x66, 0x57, 0xab, 0x13, 0x53, 0xd8, 0x8e, 0xc1, 0x40, 0xc5, 0xd7,
  0x13, 0x16, 0x5a, 0x72, 0xc7, 0xb7, 0x69, 0x01, 0xc4, 0x7a, 0xb1, 0x83,
  0x01, 0x68, 0x7d, 0x8d, 0x41, 0xa1, 0x94, 0x18, 0xc1, 0x25, 0x5c, 0xfc,
  0xf0, 0xfe, 0x83, 0x02, 0x87, 0x7c, 0x0d, 0x0d, 0xcf, 0x2e, 0x08, 0x5c,
  0x4a, 0x40, 0x0d, 0x3e, 0xec, 0x81, 0x61, 0xe6, 0x24, 0xdb, 0xca, 0xe0,
  0x0e, 0x2d, 0x07, 0xb2, 0x3e, 0x56, 0xdc, 0x8d, 0xf5, 0x41, 0x85, 0x07,
  0x48, 0x9b, 0x0c, 0x0b, 0xcb, 0x49, 0x3f, 0x7d, 0xec, 0xb7, 0xfd, 0xcb,
  0x8d, 0x67, 0x89, 0x1a, 0xab, 0xed, 0xbb, 0x1e, 0xa3, 0x00, 0x08, 0x08,
  0x17, 0x2a, 0x82, 0x5c, 0x31, 0x5d, 0x46, 0x8a, 0x2d, 0x0f, 0x86, 0x9b,
  0x74, 0xd9, 0x45, 0xfb, 0xd4, 0x40, 0xb1, 0x7a, 0xaa, 0x68, 0x2d, 0x86,
  0xb2, 0x99, 0x22, 0xe1, 0xc1, 0x2b, 0xc7, 0x9c, 0xf8, 0xf3, 0x5f, 0xa8,
  0x82, 0x12, 0xeb, 0x19, 0x11, 0x2d
};
const unsigned int caCertLen = 1026;

Debug Messages

..............wifi evt: 1
STA disconnect: 204
wifi evt: 0
wifi evt: 3
.
WiFi connected
IP address: 
192.168.1.146
Setting time using SNTP.
Current time: Mon Mar 12 04:35:03 2018
connecting to online.wonderware.com
[hostByName] request IP for: online.wonderware.com
[hostByName] Host: online.wonderware.com IP: 104.40.63.98
:ref 1
State:	sending Client Hello (1)
:wr 102 102 0
:wrc 102 102 0
:sent 102
:rn 536
:rd 5, 536, 0
:rdi 536, 5
:rd 531, 536, 5
:rdi 531, 531
:c0 531, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 255
:rd 255, 255, 0
:rdi 255, 255
:c0 255, 255
State:	receiving Server Hello (2)
State:	receiving Certificate (11)
Error: invalid handshake
:wr 7 7 0
:wrc 7 7 0
Alert: unexpected message
Error: invalid handshake
:wr 7 7 0
:wrc 7 7 0
Alert: close notify
connection failed
connecting to online.wonderware.com


then it repeats

[jtmoderate876]

@igrr Are you too busy to look at this? (trying to HTTPSRequestCACert with sites other than api.github.com such as:
online.wonderware.com (above)
www.gsmcounters.com (see issue #3661)

I've found another and could document look for more.

[jtmoderate876]

Anyone having HTTPSRequestCACert (or similar sketch) working with a different website please list what it works with:

Works with:
api.github.com (verified by jt and others)

Doesn't work with:
online.wonderware.com (verified by jt)

[JiriBilek]

Hi,
I tested your sketch on iot.bilek.info (verified by Let's Encrypt) and it works.
I run GET /test.html http request.

Not being an expert in TLS I can't tell it would be useful to you, though...

connecting to iot.bilek.info
Alert: close notify
State:	sending Client Hello (1)
State:	receiving Server Hello (2)
State:	receiving Certificate (11)
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		www.bilek.info
Organization (O):		<Not Part Of Certificate>
Basic Constraints:		critical, CA:FALSE, pathlen:10000
Key Usage:			critical, Digital Signature, Key Encipherment
Subject Alt Name:		iot.bilek.info test.bilek.info www.bilek.info 
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		Let's Encrypt Authority X3
Organization (O):		Let's Encrypt
Country (C):			US
Not Before:			Thu Feb 15 23:21:12 2018
Not After:			Wed May 16 23:21:12 2018
RSA bitsize:			2048
Sig Type:			SHA256
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		Let's Encrypt Authority X3
Organization (O):		Let's Encrypt
Country (C):			US
Basic Constraints:		critical, CA:TRUE, pathlen:0
Key Usage:			critical, Digital Signature, Key Cert Sign, CRL Sign
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DST Root CA X3
Organization (O):		Digital Signature Trust Co.
Not Before:			Thu Mar 17 16:40:46 2016
Not After:			Wed Mar 17 16:40:46 2021
RSA bitsize:			2048
Sig Type:			SHA256
State:	receiving Server Hello Done (14)
State:	sending Client Key Exchange (16)
State:	sending Finished (16)
State:	receiving Finished (16)
Server certificate verified
requesting URL: /test.html
request sent
headers received
esp8266/Arduino CI has failed
reply was:
==========
<!doctype html>

==========

Done

[jtmoderate876]

@JiriBilek Thank you for doing that - I'm sure it was a hassle with the DER CACert thing. I appreciate it.
You are connecting and getting past the "Server Hello" stage that we others are failing at with Invalid Handshake - possibly having something to do with axtls.

So HTTPSRequestCACert.ino example works with some websites:
api.github.com (verified by jt and others)
www.bilek.info verified by @JiriBilek

Doesn't work with with others:
online.wonderware.com (verified by jt)
www.gsmcounters.com (see issue #3661)

@earlephilhower - thanks for additional clues. I'll keep an eye over there too.

[jtmoderate876]

Thankfully, @sukretniy took a version of his fix, changing axtls:
uint8_t *buf = &ssl->bm_data[ssl->dc->bm_proc_index];
replace with:
uint8_t *buf = &ssl->bm_data[ssl->dc->bm_proc_index > 0 ? ssl->dc->bm_proc_index + 6 : ssl->dc->bm_proc_index];

and recompiled it for me because I don't know how and he was kind enough to do so (see his issue #3661) .
And his fix works for online.wonderware.com too!
It would be good to get this fix into the main code:
igrr/axtls-8266

[earlephilhower]

Closing this as a duplicate of #3661. Hopefully @sukretniy can share his source changes so it can go into the esp8266 axtls repo.