Skip to content

Error Invalid Handshake with websites other than api.github.com using HTTPSRequestCACert #4501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
6 tasks done
jtmoderate876 opened this issue Mar 11, 2018 · 6 comments
Closed
6 tasks done

Error Invalid Handshake with websites other than api.github.com using HTTPSRequestCACert #4501

jtmoderate876 opened this issue Mar 11, 2018 · 6 comments

Comments

@jtmoderate876
Copy link

Basic Infos

  • This issue complies with the issue POLICY doc.
  • I have read the documentation at readthedocs and the issue is not addressed there.
  • I have tested that the issue is present in current master branch (aka latest git).
  • I have searched the issue tracker for a similar issue.
  • If there is a stack dump, I have decoded it.
  • I have filled out all fields below.

Platform

  • Hardware: [ESP-12]
  • Core Version: [latest git hash or date]
  • Development Env: [Arduino IDE]
  • Operating System: [Windows]

Settings in IDE

Debug level set to everything, SSL TLS_MEM+...

Problem Description

Cannot connect to some sites with HTTPSRequestCACert.
With debug on I see Invalid Handshake when it tries to Receive Certificate (see debug log below).

I can get it to work with some sites such as api.github.com
But not online.wonderware.com
I've also tried and had trouble with others.

It sounds similar to this log:
#3661
Where that user recommends a change to axtls but I don't have the skills.

Sketch

#include <time.h>
#include <ESP8266WiFi.h>

const char* ssid = "myssid";
const char* password = "mypassword";

const int httpsPort = 443;
const char* host = "online.wonderware.com";
//note: i modified CACert with linux xxd -i DigiCertHighAssuranceEVRootCA.crt.der >cacert.h
// Root certificate used by api.github.com.
// Defined in "CACert" tab.
extern const unsigned char caCert[] PROGMEM;
extern const unsigned int caCertLen;

WiFiClientSecure client;

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Synchronize time useing SNTP. This is necessary to verify that
  // the TLS certificates offered by the server are currently valid.
  Serial.print("Setting time using SNTP");
  configTime(8 * 3600, 0, "pool.ntp.org", "time.nist.gov");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));

  // Load root certificate in DER format into WiFiClientSecure object
  bool res = client.setCACert_P(caCert, caCertLen);
  if (!res) {
    Serial.println("Failed to load root CA certificate!");
    while (true) {
      yield();
    }
  }
}

void loop() {
  // Connect to remote server
  Serial.print("connecting to ");
  Serial.println(host);
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  // Verify validity of server's certificate
  if (client.verifyCertChain(host)) {
    Serial.println("Server certificate verified");
  } else {
    Serial.println("ERROR: certificate verification failed!");
    return;
  }

  String url = "/repos/esp8266/Arduino/commits/master/status";
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');
  if (line.startsWith("{\"state\":\"success\"")) {
    Serial.println("esp8266/Arduino CI successfull!");
  } else {
    Serial.println("esp8266/Arduino CI has failed");
  }
  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println();

  static int repeat = 0;
  if (++repeat == 3) {
    Serial.println("Done");
    while (true) {
      delay(1000);
    }
  }
  delay(60000);
}

and the CACert for this site is:

CACert

const unsigned char caCert[] PROGMEM = {
  0x30, 0x82, 0x03, 0xfe, 0x30, 0x82, 0x02, 0xe6, 0xa0, 0x03, 0x02, 0x01,
  0x02, 0x02, 0x10, 0x15, 0xac, 0x6e, 0x94, 0x19, 0xb2, 0x79, 0x4b, 0x41,
  0xf6, 0x27, 0xa9, 0xc3, 0x18, 0x0f, 0x1f, 0x30, 0x0d, 0x06, 0x09, 0x2a,
  0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b, 0x05, 0x00, 0x30, 0x81,
  0x98, 0x31, 0x0b, 0x30, 0x09, 0x06, 0x03, 0x55, 0x04, 0x06, 0x13, 0x02,
  0x55, 0x53, 0x31, 0x16, 0x30, 0x14, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x13,
  0x0d, 0x47, 0x65, 0x6f, 0x54, 0x72, 0x75, 0x73, 0x74, 0x20, 0x49, 0x6e,
  0x63, 0x2e, 0x31, 0x39, 0x30, 0x37, 0x06, 0x03, 0x55, 0x04, 0x0b, 0x13,
  0x30, 0x28, 0x63, 0x29, 0x20, 0x32, 0x30, 0x30, 0x38, 0x20, 0x47, 0x65,
  0x6f, 0x54, 0x72, 0x75, 0x73, 0x74, 0x20, 0x49, 0x6e, 0x63, 0x2e, 0x20,
  0x2d, 0x20, 0x46, 0x6f, 0x72, 0x20, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72,
  0x69, 0x7a, 0x65, 0x64, 0x20, 0x75, 0x73, 0x65, 0x20, 0x6f, 0x6e, 0x6c,
  0x79, 0x31, 0x36, 0x30, 0x34, 0x06, 0x03, 0x55, 0x04, 0x03, 0x13, 0x2d,
  0x47, 0x65, 0x6f, 0x54, 0x72, 0x75, 0x73, 0x74, 0x20, 0x50, 0x72, 0x69,
  0x6d, 0x61, 0x72, 0x79, 0x20, 0x43, 0x65, 0x72, 0x74, 0x69, 0x66, 0x69,
  0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x20, 0x41, 0x75, 0x74, 0x68, 0x6f,
  0x72, 0x69, 0x74, 0x79, 0x20, 0x2d, 0x20, 0x47, 0x33, 0x30, 0x1e, 0x17,
  0x0d, 0x30, 0x38, 0x30, 0x34, 0x30, 0x32, 0x30, 0x30, 0x30, 0x30, 0x30,
  0x30, 0x5a, 0x17, 0x0d, 0x33, 0x37, 0x31, 0x32, 0x30, 0x31, 0x32, 0x33,
  0x35, 0x39, 0x35, 0x39, 0x5a, 0x30, 0x81, 0x98, 0x31, 0x0b, 0x30, 0x09,
  0x06, 0x03, 0x55, 0x04, 0x06, 0x13, 0x02, 0x55, 0x53, 0x31, 0x16, 0x30,
  0x14, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x13, 0x0d, 0x47, 0x65, 0x6f, 0x54,
  0x72, 0x75, 0x73, 0x74, 0x20, 0x49, 0x6e, 0x63, 0x2e, 0x31, 0x39, 0x30,
  0x37, 0x06, 0x03, 0x55, 0x04, 0x0b, 0x13, 0x30, 0x28, 0x63, 0x29, 0x20,
  0x32, 0x30, 0x30, 0x38, 0x20, 0x47, 0x65, 0x6f, 0x54, 0x72, 0x75, 0x73,
  0x74, 0x20, 0x49, 0x6e, 0x63, 0x2e, 0x20, 0x2d, 0x20, 0x46, 0x6f, 0x72,
  0x20, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x65, 0x64, 0x20,
  0x75, 0x73, 0x65, 0x20, 0x6f, 0x6e, 0x6c, 0x79, 0x31, 0x36, 0x30, 0x34,
  0x06, 0x03, 0x55, 0x04, 0x03, 0x13, 0x2d, 0x47, 0x65, 0x6f, 0x54, 0x72,
  0x75, 0x73, 0x74, 0x20, 0x50, 0x72, 0x69, 0x6d, 0x61, 0x72, 0x79, 0x20,
  0x43, 0x65, 0x72, 0x74, 0x69, 0x66, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f,
  0x6e, 0x20, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x74, 0x79, 0x20,
  0x2d, 0x20, 0x47, 0x33, 0x30, 0x82, 0x01, 0x22, 0x30, 0x0d, 0x06, 0x09,
  0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01, 0x05, 0x00, 0x03,
  0x82, 0x01, 0x0f, 0x00, 0x30, 0x82, 0x01, 0x0a, 0x02, 0x82, 0x01, 0x01,
  0x00, 0xdc, 0xe2, 0x5e, 0x62, 0x58, 0x1d, 0x33, 0x57, 0x39, 0x32, 0x33,
  0xfa, 0xeb, 0xcb, 0x87, 0x8c, 0xa7, 0xd4, 0x4a, 0xdd, 0x06, 0x88, 0xea,
  0x64, 0x8e, 0x31, 0x98, 0xa5, 0x38, 0x90, 0x1e, 0x98, 0xcf, 0x2e, 0x63,
  0x2b, 0xf0, 0x46, 0xbc, 0x44, 0xb2, 0x89, 0xa1, 0xc0, 0x28, 0x0c, 0x49,
  0x70, 0x21, 0x95, 0x9f, 0x64, 0xc0, 0xa6, 0x93, 0x12, 0x02, 0x65, 0x26,
  0x86, 0xc6, 0xa5, 0x89, 0xf0, 0xfa, 0xd7, 0x84, 0xa0, 0x70, 0xaf, 0x4f,
  0x1a, 0x97, 0x3f, 0x06, 0x44, 0xd5, 0xc9, 0xeb, 0x72, 0x10, 0x7d, 0xe4,
  0x31, 0x28, 0xfb, 0x1c, 0x61, 0xe6, 0x28, 0x07, 0x44, 0x73, 0x92, 0x22,
  0x69, 0xa7, 0x03, 0x88, 0x6c, 0x9d, 0x63, 0xc8, 0x52, 0xda, 0x98, 0x27,
  0xe7, 0x08, 0x4c, 0x70, 0x3e, 0xb4, 0xc9, 0x12, 0xc1, 0xc5, 0x67, 0x83,
  0x5d, 0x33, 0xf3, 0x03, 0x11, 0xec, 0x6a, 0xd0, 0x53, 0xe2, 0xd1, 0xba,
  0x36, 0x60, 0x94, 0x80, 0xbb, 0x61, 0x63, 0x6c, 0x5b, 0x17, 0x7e, 0xdf,
  0x40, 0x94, 0x1e, 0xab, 0x0d, 0xc2, 0x21, 0x28, 0x70, 0x88, 0xff, 0xd6,
  0x26, 0x6c, 0x6c, 0x60, 0x04, 0x25, 0x4e, 0x55, 0x7e, 0x7d, 0xef, 0xbf,
  0x94, 0x48, 0xde, 0xb7, 0x1d, 0xdd, 0x70, 0x8d, 0x05, 0x5f, 0x88, 0xa5,
  0x9b, 0xf2, 0xc2, 0xee, 0xea, 0xd1, 0x40, 0x41, 0x6d, 0x62, 0x38, 0x1d,
  0x56, 0x06, 0xc5, 0x03, 0x47, 0x51, 0x20, 0x19, 0xfc, 0x7b, 0x10, 0x0b,
  0x0e, 0x62, 0xae, 0x76, 0x55, 0xbf, 0x5f, 0x77, 0xbe, 0x3e, 0x49, 0x01,
  0x53, 0x3d, 0x98, 0x25, 0x03, 0x76, 0x24, 0x5a, 0x1d, 0xb4, 0xdb, 0x89,
  0xea, 0x79, 0xe5, 0xb6, 0xb3, 0x3b, 0x3f, 0xba, 0x4c, 0x28, 0x41, 0x7f,
  0x06, 0xac, 0x6a, 0x8e, 0xc1, 0xd0, 0xf6, 0x05, 0x1d, 0x7d, 0xe6, 0x42,
  0x86, 0xe3, 0xa5, 0xd5, 0x47, 0x02, 0x03, 0x01, 0x00, 0x01, 0xa3, 0x42,
  0x30, 0x40, 0x30, 0x0f, 0x06, 0x03, 0x55, 0x1d, 0x13, 0x01, 0x01, 0xff,
  0x04, 0x05, 0x30, 0x03, 0x01, 0x01, 0xff, 0x30, 0x0e, 0x06, 0x03, 0x55,
  0x1d, 0x0f, 0x01, 0x01, 0xff, 0x04, 0x04, 0x03, 0x02, 0x01, 0x06, 0x30,
  0x1d, 0x06, 0x03, 0x55, 0x1d, 0x0e, 0x04, 0x16, 0x04, 0x14, 0xc4, 0x79,
  0xca, 0x8e, 0xa1, 0x4e, 0x03, 0x1d, 0x1c, 0xdc, 0x6b, 0xdb, 0x31, 0x5b,
  0x94, 0x3e, 0x3f, 0x30, 0x7f, 0x2d, 0x30, 0x0d, 0x06, 0x09, 0x2a, 0x86,
  0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b, 0x05, 0x00, 0x03, 0x82, 0x01,
  0x01, 0x00, 0x2d, 0xc5, 0x13, 0xcf, 0x56, 0x80, 0x7b, 0x7a, 0x78, 0xbd,
  0x9f, 0xae, 0x2c, 0x99, 0xe7, 0xef, 0xda, 0xdf, 0x94, 0x5e, 0x09, 0x69,
  0xa7, 0xe7, 0x6e, 0x68, 0x8c, 0xbd, 0x72, 0xbe, 0x47, 0xa9, 0x0e, 0x97,
  0x12, 0xb8, 0x4a, 0xf1, 0x64, 0xd3, 0x39, 0xdf, 0x25, 0x34, 0xd4, 0xc1,
  0xcd, 0x4e, 0x81, 0xf0, 0x0f, 0x04, 0xc4, 0x24, 0xb3, 0x34, 0x96, 0xc6,
  0xa6, 0xaa, 0x30, 0xdf, 0x68, 0x61, 0x73, 0xd7, 0xf9, 0x8e, 0x85, 0x89,
  0xef, 0x0e, 0x5e, 0x95, 0x28, 0x4a, 0x2a, 0x27, 0x8f, 0x10, 0x8e, 0x2e,
  0x7c, 0x86, 0xc4, 0x02, 0x9e, 0xda, 0x0c, 0x77, 0x65, 0x0e, 0x44, 0x0d,
  0x92, 0xfd, 0xfd, 0xb3, 0x16, 0x36, 0xfa, 0x11, 0x0d, 0x1d, 0x8c, 0x0e,
  0x07, 0x89, 0x6a, 0x29, 0x56, 0xf7, 0x72, 0xf4, 0xdd, 0x15, 0x9c, 0x77,
  0x35, 0x66, 0x57, 0xab, 0x13, 0x53, 0xd8, 0x8e, 0xc1, 0x40, 0xc5, 0xd7,
  0x13, 0x16, 0x5a, 0x72, 0xc7, 0xb7, 0x69, 0x01, 0xc4, 0x7a, 0xb1, 0x83,
  0x01, 0x68, 0x7d, 0x8d, 0x41, 0xa1, 0x94, 0x18, 0xc1, 0x25, 0x5c, 0xfc,
  0xf0, 0xfe, 0x83, 0x02, 0x87, 0x7c, 0x0d, 0x0d, 0xcf, 0x2e, 0x08, 0x5c,
  0x4a, 0x40, 0x0d, 0x3e, 0xec, 0x81, 0x61, 0xe6, 0x24, 0xdb, 0xca, 0xe0,
  0x0e, 0x2d, 0x07, 0xb2, 0x3e, 0x56, 0xdc, 0x8d, 0xf5, 0x41, 0x85, 0x07,
  0x48, 0x9b, 0x0c, 0x0b, 0xcb, 0x49, 0x3f, 0x7d, 0xec, 0xb7, 0xfd, 0xcb,
  0x8d, 0x67, 0x89, 0x1a, 0xab, 0xed, 0xbb, 0x1e, 0xa3, 0x00, 0x08, 0x08,
  0x17, 0x2a, 0x82, 0x5c, 0x31, 0x5d, 0x46, 0x8a, 0x2d, 0x0f, 0x86, 0x9b,
  0x74, 0xd9, 0x45, 0xfb, 0xd4, 0x40, 0xb1, 0x7a, 0xaa, 0x68, 0x2d, 0x86,
  0xb2, 0x99, 0x22, 0xe1, 0xc1, 0x2b, 0xc7, 0x9c, 0xf8, 0xf3, 0x5f, 0xa8,
  0x82, 0x12, 0xeb, 0x19, 0x11, 0x2d
};
const unsigned int caCertLen = 1026;

Debug Messages

..............wifi evt: 1
STA disconnect: 204
wifi evt: 0
wifi evt: 3
.
WiFi connected
IP address: 
192.168.1.146
Setting time using SNTP.
Current time: Mon Mar 12 04:35:03 2018
connecting to online.wonderware.com
[hostByName] request IP for: online.wonderware.com
[hostByName] Host: online.wonderware.com IP: 104.40.63.98
:ref 1
State:	sending Client Hello (1)
:wr 102 102 0
:wrc 102 102 0
:sent 102
:rn 536
:rd 5, 536, 0
:rdi 536, 5
:rd 531, 536, 5
:rdi 531, 531
:c0 531, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 536
:rd 536, 536, 0
:rdi 536, 536
:c0 536, 536
:rn 255
:rd 255, 255, 0
:rdi 255, 255
:c0 255, 255
State:	receiving Server Hello (2)
State:	receiving Certificate (11)
Error: invalid handshake
:wr 7 7 0
:wrc 7 7 0
Alert: unexpected message
Error: invalid handshake
:wr 7 7 0
:wrc 7 7 0
Alert: close notify
connection failed
connecting to online.wonderware.com


then it repeats
@jtmoderate876
Copy link
Author

jtmoderate876 commented Mar 15, 2018 •
edited
Loading

@igrr Are you too busy to look at this? (trying to HTTPSRequestCACert with sites other than api.github.com such as:
online.wonderware.com (above)
www.gsmcounters.com (see issue #3661)

I've found another and could document look for more.

@jtmoderate876 jtmoderate876 changed the title Error Invalid Handshake with SOME websites using HTTPSRequestCACert Error Invalid Handshake with websites other than api.github.com using HTTPSRequestCACert Mar 15, 2018
@jtmoderate876
Copy link
Author

Anyone having HTTPSRequestCACert (or similar sketch) working with a different website please list what it works with:

Works with:
api.github.com (verified by jt and others)

Doesn't work with:
online.wonderware.com (verified by jt)

@JiriBilek
Copy link
Contributor

JiriBilek commented Mar 18, 2018 •
edited
Loading

Hi,
I tested your sketch on iot.bilek.info (verified by Let's Encrypt) and it works.
I run GET /test.html http request.

Not being an expert in TLS I can't tell it would be useful to you, though...

connecting to iot.bilek.info
Alert: close notify
State:	sending Client Hello (1)
State:	receiving Server Hello (2)
State:	receiving Certificate (11)
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		www.bilek.info
Organization (O):		<Not Part Of Certificate>
Basic Constraints:		critical, CA:FALSE, pathlen:10000
Key Usage:			critical, Digital Signature, Key Encipherment
Subject Alt Name:		iot.bilek.info test.bilek.info www.bilek.info 
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		Let's Encrypt Authority X3
Organization (O):		Let's Encrypt
Country (C):			US
Not Before:			Thu Feb 15 23:21:12 2018
Not After:			Wed May 16 23:21:12 2018
RSA bitsize:			2048
Sig Type:			SHA256
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		Let's Encrypt Authority X3
Organization (O):		Let's Encrypt
Country (C):			US
Basic Constraints:		critical, CA:TRUE, pathlen:0
Key Usage:			critical, Digital Signature, Key Cert Sign, CRL Sign
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DST Root CA X3
Organization (O):		Digital Signature Trust Co.
Not Before:			Thu Mar 17 16:40:46 2016
Not After:			Wed Mar 17 16:40:46 2021
RSA bitsize:			2048
Sig Type:			SHA256
State:	receiving Server Hello Done (14)
State:	sending Client Key Exchange (16)
State:	sending Finished (16)
State:	receiving Finished (16)
Server certificate verified
requesting URL: /test.html
request sent
headers received
esp8266/Arduino CI has failed
reply was:
==========
<!doctype html>

==========

Done

@earlephilhower earlephilhower mentioned this issue Mar 19, 2018
Closed
@jtmoderate876
Copy link
Author

jtmoderate876 commented Mar 20, 2018 •
edited
Loading

@JiriBilek Thank you for doing that - I'm sure it was a hassle with the DER CACert thing. I appreciate it.
You are connecting and getting past the "Server Hello" stage that we others are failing at with Invalid Handshake - possibly having something to do with axtls.

So HTTPSRequestCACert.ino example works with some websites:
api.github.com (verified by jt and others)
www.bilek.info verified by @JiriBilek

Doesn't work with with others:
online.wonderware.com (verified by jt)
www.gsmcounters.com (see issue #3661)

@earlephilhower - thanks for additional clues. I'll keep an eye over there too.

@jtmoderate876
Copy link
Author

jtmoderate876 commented Apr 7, 2018 •
edited
Loading

Thankfully, @sukretniy took a version of his fix, changing axtls:
uint8_t *buf = &ssl->bm_data[ssl->dc->bm_proc_index];
replace with:
uint8_t *buf = &ssl->bm_data[ssl->dc->bm_proc_index > 0 ? ssl->dc->bm_proc_index + 6 : ssl->dc->bm_proc_index];

and recompiled it for me because I don't know how and he was kind enough to do so (see his issue #3661) .
And his fix works for online.wonderware.com too!
It would be good to get this fix into the main code:
igrr/axtls-8266

@earlephilhower
Copy link
Collaborator

Closing this as a duplicate of #3661. Hopefully @sukretniy can share his source changes so it can go into the esp8266 axtls repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@earlephilhower @JiriBilek @jtmoderate876