WiFiClientSecure doesn't handle large certificates

[nikant]

Hardware

Hardware: NodeMCU v1.0 ESP-12E
Core Version: 2.1.0

Description

Well.. if someone could offer some enlightenment.. can't find the error in the following (modified host from the original of course..)

It hangs at client.connect for some time.. and then connection fails.
the same happens in "google.com" or "www.google.com"

other hosts work and reply normally

Is it my network?
I've come to believe that it has something to do with dns resolving (or not..)
Of course I've tried setting various DNS and a static IP -commented sections in code- but no result.

Settings in IDE

Module: NodeMCU v1.0
Flash Size: 4MB
CPU Frequency: 80Mhz
Flash Mode: ?qio?
Flash Frequency: ?40Mhz?
Upload Using: SERIAL
Reset Method: ?ck / nodemcu?

Sketch

/*
 *  HTTP over TLS (HTTPS) example sketch
 *
 *  This example demonstrates how to use
 *  WiFiClientSecure class to access HTTPS API.
 *  We fetch and display the status of
 *  esp8266/Arduino project continuous integration
 *  build.
 *
 *  Created by Ivan Grokhotkov, 2015.
 *  This example is in public domain.
 */

#include <ESP8266WiFi.h>

// Replace with your network credentials
const char* ssid = ".....";
const char* password = ".........";

const char* host = "calendar.google.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
//const char* fingerprint = "CF 05 98 89 CA FF 8E D8 5E 5C E0 C2 E4 F7 E6 C3 C7 50 DD 5C";
const char* fingerprint = "99 70 D6 7C B8 42 71 18 DC A6 88 DB DC C8 69 66 96 9D 51 88"; //google

/*
// Update these with values suitable for your network.
IPAddress ip(192,168,1,229);  //Node static IP
IPAddress gateway(192,168,1,1);
IPAddress subnet(255,255,255,0);
IPAddress dns1(208,67,222,222);
IPAddress dns2(208,67,220,220);
*/

// Use WiFiClientSecure class to create TLS connection
WiFiClientSecure client;

void setup() {
  Serial.begin(57600);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);

  //WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  //WiFi.config(ip, gateway, subnet, dns1, dns2);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  Serial.print("connecting to ");
  Serial.println(host);

  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }


  String url = "/";
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: ESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');

  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println("closing connection");
}

void loop() {
}

Debug Messages

connecting to ssid
..
WiFi connected
IP address:
192.168.1.11
connecting to calendar.google.com
connection failed

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[Links2004]

enable the debug out for Core + WiFi
https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/debugging.md

then you will see if the resolve is the problem.

Note: for http/https existing a class then you not need to handle the header and encoding yourself
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/examples/BasicHttpClient/BasicHttpClient.ino

[nikant]

@Links2004 thank you for the second link. 👍 I'll take a look. Sorry I'm new to all this.. :/

Also debug menu is not available in the IDE if you have NodeMCU board selected and I don't have a bare esp.

---

edit: I tried the the HttpClient method with a link in my google calendar (my private link that allows to donwload the ical) which is https://calendar.google.com/calendar/..........
and the result was

WiFi connected
IP address: 
192.168.1.229
connection refused

---

edit2: ok I enabled debug
At HTTPClient+SSL it gives the following:

[HTTP] GET...
[HTTP-Client] connect https...
please start sntp first !
State:  sending Client Hello (1)
State:  receiving Server Hello (2)
ssl->need_bytes=10149 > 6859
Error: invalid protocol message
Alert: handshake failure
Alert: close notify
[HTTP-Client] failed connect to calendar.google.com:443
[HTTP-Client][returnError] error(-1): connection refused
[HTTP] GET... failed, error: connection refused
[HTTP-Client][end] tcp is closed
pm open,type:2 0

[Links2004]

the generic board work with all board you only need to set the settings for flash and reset method right.
please set the debug too Core + WiFi then you can see if it a DNS problem.

but ssl->need_bytes=10149 > 6859
looks like the ssl cert is to big

[igrr]

It's more likely that connection is failing during ssl handshake due to the certificate size. I do have a patch for this, maybe will push it on the weekend.

[nikant]

I set debug to Core + WiFi

[HTTP] begin...
[HTTP] GET...
[hostByName] request IP for: calendar.google.com
[hostByName] Host: calendar.google.com IP: 193.92.133.45
:ref 1
please start sntp first !
:wr
:sent 52
:ww
:rn 1452
:rd 5, 1452, 0
:rdi 1452, 5
:rd 74, 1452, 5
:rdi 1447, 74
:rd 5, 1452, 79
:rdi 1373, 5
ssl->need_bytes=10149 > 6859
:wr
:rch 1452, 596
:ww
:wr
:rch 2048, 1452
:rch 3500, 1452
:sent 7
:ww
:wr
:er -9 7 1
:ww
[HTTP] GET... failed, error: connection refused
:ur 1
:del
pm open,type:2 0

something noticed is that IP of calendar.google.com returned is 193.92.133.45 which is
canonical name *cache.google.com *

while calendar.google.com is
canonical name www3.l.google.com.
aliases calendar.google.com
some other IP here.
but I guess Google knows better..

also fingerprint changed from what it was yesterday (given from https://www.grc.com/fingerprints.htm)
(am I crazy??)
(really sorry I'm reporting anything in my ignorance and you probably try to figure out where is the problem..)

here is the info with the fingerprint

[HTTP] GET...
[hostByName] request IP for: calendar.google.com
[hostByName] Host: calendar.google.com IP: 216.58.209.206
:ref 1
please start sntp first !
:wr
:sent 52
:rn 1430
:ww
:rd 5, 1430, 0
:rdi 1430, 5
:rd 74, 1430, 5
:rdi 1425, 74
:rd 5, 1430, 79
:rdi 1351, 5
:rd 1346, 1430, 84:rch 1430, 1430
:rch 2860, 1135

:rdi 1346, 1346
:c 1346, 1430, 3995
:rd 2556, 2565, 0
:rdi 1430, 1430
:c 1430, 1430, 2565
:rdi 1135, 1126
:rd 5, 1135, 1126
:rdi 9, 5
:rd 4, 1135, 1131
:rdi 4, 4
:c0 4, 1135
:wr
:sent 267
:ww
:wr
:sent 6
:ww
:wr
:sent 69
:ww
:rn 75
:rd 5, 75, 0
:rdi 75, 5
:rd 1, 75, 5
:rdi 70, 1
:rd 5, 75, 6
:rdi 69, 5
:rd 64, 75, 11
:rdi 64, 64
:c0 64, 75
cert FP: 2E BC 3F 9B F9 19 4D 89 49 65 B2 24 AB A8 B1 8B 37 7B DB 20 
test FP: 3A 88 A0 FC 7A BF 9F 0F 37 D9 A1 99 95 7F D9 46 CE 15 7C 41 
fingerprint doesn't match
:wr
:sent 53
:rcl
:abort
:ww
:ur 1
:del
[HTTP] GET... failed, error: connection refused

[scottjgibson]

@igrr you mentioned you have a patch for this; I would be interested in trying it if it is in a usable state. If it is a work in progress I would still be interested in seeing how you are planning on addressing it.

[igrr]

This issue should be resolved with c8a1507.

[electronicsguy]

Ok the boards manager told me there's an update (v2.2.0) for esp8266 so I updated that. @igrr is the patch included in this version?

I then tried @nikant's script. This is what I get. Is this "working"? Sorry I am very new to using esp8266 and ssl in general. (Using ESP-01)

wifi evt: 0
.wifi evt: 3
.
WiFi connected
IP address: 
192.168.1.105
connecting to calendar.google.com
[hostByName] request IP for: calendar.google.com
[hostByName] Host: calendar.google.com IP: 216.58.216.110
certificate doesn't match
requesting URL: /
request sent
reply was:
==========
HTTP/1.1 301 Moved Permanently

==========
closing connection
pm open,type:2 0

[electronicsguy]

@nikant Were you using esp8266 with nodemcu installed on it, through Arduino IDE? Is that possible?

[igrr]

Well the connection was established, and server replied with 301 code, so yes, this seems to be working. There still is an issue with certificate doesn't match, but i suppose for some reason the certificate i see in the web browser is not the same as the one ESP receives, hence the verification error. This is a separate issue, though.

[electronicsguy]

@igrr Well it initially connects, but you cannot actually do anything useful. I generated a new fingerprint from "grc.com", which is different, but I get the same error as above.

One thing I wanted to ask, which you may know the answer to: If you try using Hurl to make https requests to, say, a Google Apps script that you've written, which has a doGet() function, it works perfectly.

However, if you set the "follow redirects" to OFF on that website, it'll do the initial connection and then report the same error. (something like "moved") and not hit your google script's doGet(). Google seems to be redirecting the requests to "script.googleusercontent.com/macros...".

Is the "follow redirects" some kind of a flag that can be included in the header for esp8266 to make it work? Or is it more complex than that. Example below.

==== Google Sheet
Link to sheet here: sheet

The script has a simple doGet() like this:

// doGet needs the spreadsheet ID, it has no concept of "active spreadsheet" if the sheet isn't open.
var doc = SpreadsheetApp.openById('1j5UX_r9JBG_qLsKYpLnlgqdZgXSkF1VC8L_mt7iAhgI');
var ss = doc.getSheetByName('Sheet1');

function doGet(e){
  var val = e.parameter.value;

  var r = ss.getRange('A1');
  r.setValue(val);

  return ContentService.createTextOutput("Success!")
}

The script is active. All it needs is a single parameter "value" and it should put that into the first cell. This is the syntax to be used:
script-url

If you put this into hurl and run it'll work as long as "follow redirects" is set to ON.

[nikant]

@electronicsguy

to my understanding the problem is that big companies like Google use multiple servers for the same domain name (load balancing, caching etc.) so one domain doesn't have one IP only and thus not one specific certificate.
So a forwarding problem occurs (I don't know how esp8266 https code handles redirects: that's one thing) and also in which of the Google servers it will end up so we can compare the correct certificate.

[electronicsguy]

@nikant Right. Apart from the certificate error, I think that it's the inability to follow redirects, which is causing it to make the initial connection, but not do what is needed. Someone will more knowledge on this would hopefully chip in. I say this because the exact behaviour can be simulated using the hurl.it service, AFAICS.

[electronicsguy]

@igrr and @nikant This page outlines how to handle redirects: python-urllib2. It says that 'urllib2' automatically handles the redirect using the Location: header info in the data returned by the server. Is this something that can be implemented on the esp8266?

[electronicsguy]

@igrr and @nikant Ok, some success:) I am able to get the partial response and have my data passed on as a variable be posted to Google Sheets. This is the partial response, which contains the 'location' field in the header that needs to be followed, to get the final response.

response

Any ideas how to proceed with parsing this, getting the 'location' URL and following that?

[electronicsguy]

@igrr and @nikant ok, success :) I am able to use esp8226 to make the initial request to Google, get the redirect location and make the second request to the redirected URL, getting the final response. Thanks for your help! With this, I can directly post data into Google spreadsheets and read sheet information.

[nikant]

@electronicsguy hi! great! do you have a sample code for that?

[electronicsguy]

@nikant Yes of course I'll put it up.

[electronicsguy]

@nikant Code and example posted here: HTTPSRedirect. You can write/read to google spreadsheets, fetch calendar events for the next 1 week and do a google 'chat' with the ESP8266.

I also fixed the certificate mismatch problem. I generate the fingerprint using openssl as described in the library.

[DavidTruyens]

@electronicsguy, I've been using your code for a while now. Sometimes there was a certificate missmatch, but that could be because I'm redirected to a different server.

But now I can't get it working. I'm using the openssl as you suggested, but I can't get it running anymore. Does anybody has the same issue?

[electronicsguy]

hi David when you say it doesn't work anymore, what happens exactly? does only certificate verification fail or also the actual reading/writing ṭo google sheets fail? if so what is the error message? some people have reported problems due to their specific DNS config. Could you try on a different network and check?

…

On Tue, Dec 20, 2016 at 8:07 PM, David Truyens ***@***.***> wrote: @electronicsguy <https://github.com/electronicsguy>, I've been using your code for a while now. Sometimes there was a certificate missmatch, but that could be because I'm redirected to a different server. But now I can't get it working. I'm using the openssl as you suggested, but I can't get it running anymore. Does anybody has the same issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1816 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFh9vVbYEd3X9p_BB3vMb5Ns2iRHNg2cks5rJ-g0gaJpZM4H4ocE> .

[DavidTruyens]

Hi Sujay,

I couldn't even connect to the google script host anymore. But now it works again... i got rid of the certificate as it works well without them (is it right that you only need that if you do a get post?).

I guess it could be something with my network. Didn't had the chance to test it on another network yet..

[electronicsguy]

@DavidTruyens You only need certificate check for security purposes. (in case you want to make sure you're not subjected to say a MITM attack). However, Google seems to keep changing their server fingerprint by time and location, so it's hard to keep up with it. In case it's a non-issue for you, you could disregard the fingerprint check.

Please let me know if things still stop working. Also try on a different network if you get the chance.

[kiralikbeyin]

@electronicsguy
I tried these fingerprints but Certificate mis-match

//const char* fingerprint = "F0 5C 74 77 3F 6B 25 D7 3B 66 4D 43 2F 7E BC 5B E9 28 86 AD";
//const char* fingerprint2 = "94 64 D8 75 DE 5D 3A E6 3B A7 B6 15 52 72 CC 51 7A BA 2B BE";
const char* fingerprint3 = "13 2C 74 E3 CB 4D 30 8F 4C 18 42 2E DC 8C 95 92 E7 E6 B5 A0";

fingerprint3 must be the latest but it didn't work.. I got this from https://www.grc.com/fingerprints.htm
and also checked from chrome as @igrr said
// Use web browser to view and copy
// SHA1 fingerprint of the certificate

debug with your example https://github.com/electronicsguy/ESP8266/tree/master/HTTPSRedirect

#include <ESP8266WiFi.h>
#include "HTTPSRedirect.h"

WiFi connected
IP address: 
192.168.0.11
Connecting to script.google.com
please start sntp first !
**cert FP: 0D 9A 55 12 4E A0 73 BE DD 1C 02 36 B5 D1 BA 91 66 A6 42 39** 
test FP: 94 2F 19 F7 A8 B4 5B 09 90 34 36 B2 2A C4 7F 17 06 AC 6A 2E 
Certificate mis-match

This fp works, i got this from debug so how can i use it when my const char* fingerprint is wrong?
cert FP: 0D 9A 55 12 4E A0 73 BE DD 1C 02 36 B5 D1 BA 91 66 A6 42 39

[pieman64]

@electronicsguy loving your work Sujay.
Below is a screenshot of data updated every minute from 3 sensors attached to an ESP8266.
Row 2, in yellow, shows the latest data and is a duplication of the last row in the sheet.

All working fine, but like others, there is an issue with the fingerprint and I see 'Certificate mis-match'
error in Serial Monitor.
To be honest I'm not really expecting anyone to carry out a man in the middle attack on my Google Sheet but I am interested to know why the fingerprints are not working. I have tried all the combinations I can think of for fingerprint and fingerprint2 but not joy.
Any idea what we are doing wrong?

[pieman64]

OK I found the correct fingerprints for Google in our part of the world (edge of Europe / Asia).

const char* fingerprint =  "0D 9A 55 12 4E A0 73 BE DD 1C 02 36 B5 D1 BA 91 66 A6 42 39";    // certificates match for script.google.com OK
const char* fingerprint2 = "67 36 7F EA 24 86 98 D3 F5 30 95 49 2B CB 41 60 22 76 07 08";    // certificates match for script.googleusercontent.com OK

Serial Monitor now states "Certificate match."

[electronicsguy]

@kiralikbeyin As mentioned in the main project file, www.grc.com doesn't seem to get the right fingerprint. You must use the shell command that is in the file (which uses openssl). I updated my fingerprint right now and it matches correctly. Please do that and check.

@pieman64 Google keeps changing the server endpoints (and hence certificates) and it seems they also change based on where in the world you access it from. So there's no way to really have a certificate work for everyone, forever (at least, I don't know how to do that). But please use the technique I mentioned above and extract the certificate fingerprint. It'll should then work correctly, at least for some time, from your location. Sorry, this certificate business is not really my field of expertise. Maybe @igrr could explain it better.

I'm busy re-writing the code. I'll upload a newer version soon, making some things easier for developers.

Cheers.

[kiralikbeyin]

@electronicsguy Thanks, please keep here updated when you are done. I hope you are adding auto-match fp.

[Daemach]

I've been working with Igrr's test code here: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

When I use api.github.com as the host it seems to work. When I use gb.synaptrix.com as the host, it fails - it doesn't even get to the fingerprint verification stage. I have debugging turned on and set to "all" and don't see any kind of messaging for the https handshake. This is the output:

wifi evt: 0
.....wifi evt: 3
.
WiFi connected
IP address:
192.168.1.177
connecting to gb.synaptrix.com
[hostByName] request IP for: gb.synaptrix.com
[hostByName] Host: gb.synaptrix.com IP: 52.38.244.165
connection failed
wifi evt: 7

How do I troubleshoot this?

[Daemach]

Suggestions from Igrr solved my problem. Hopefully this helps someone else:

Can you please enable 'Core + SSL' debug level and add Serial.setDebugOutput(true); to the setup function? This may produce a more verbose log.

IIRC, the two cipher suites supported in 2.3.0 are TLS_RSA_WITH_AES_128_CBC_SHA and TLS_RSA_WITH_AES_256_CBC_SHA. Latest git version adds a couple other cipher suites, but also not the ones listed for this website (axTLS only does RSA).

You need to configure the webserver to allow one of these two cipher suites, or set up an HTTPS termination proxy if this server is outside of your control.

[electronicsguy]

@DavidTruyens @kiralikbeyin @pieman64 V2.0 uploaded. Core completely re-written. Please check it out.