HttpClient: HTTPS request without specifying certificate fingerprint?

[tuxedo0801]

Hi there,

I have to push some data from ESP8266 to an IoT Cloud system. But this requires HTTPS.

In the examples I found, I always need to specify the fingerprint of the https-certficate.

This binds my source directly to the cloud system. If certificate changes, the request fails I guess...
So I have to update my sketch with every certificate-change, which is not that easy for my use-case.

Is there a possibility to accept whatever certificate is running behind https?

For my use-case it would be okay to use http instead of https, but the cloud system does only provide https. So I have to stick to it.

br,
Alex

[SnakeDoc]

Not the best solution, but could you proxy the requests through a service/server you control? I suppose you'd have the same issue once your service/server's certificate expires though...

[tuxedo0801]

This issue relates to #1851 ...

Maybe I add a php-script inbetween to decouple https-request from esp8266, but this is more a workaround than a real solution.

[tuxedo0801]

Found this:

https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequestCACert/HTTPSRequestCACert.ino

Have to check if this is a valid solution for my use-case.

[tuxedo0801]

Currently I'm testing the solution provided in #1851, but I would like to see, that I can make https-calls, without specifying a certificate-finger-print and without providing the root-ca-certificate to do certificate validation. Just "accept all certificates and ignore the complete certificate-check", like you can do on any other desktop-programming-language ...

Is this in general possible? Or is this technically on the esp8266 not possible?

[JamesGKent]

you might be able to do it if you set the methods that check the cert to return as if they matched at all times, from a brief look you need to:

• replace the return with a return true at line 485 in wificlientsecure.cpp
• replace return with return true at line 518 in wificlientsecure.cpp
• replace return with return true at line 546 in wificlientsecure.cpp

I would suggest that to do what you want you'll want to move the returns closer to the start of each function, but after checking that you have an ssl context, so i'd suggest just adding return true before each of these lines:

• 465
• 494
• 524
• 554 (needed because call to verify cert chain will fail)

can always give this a try.

[tuxedo0801]

Sounds like copy&paste&modify of Wificlientsecure.cpp/.h to create a quick'n'easy workaround...

Did someone test this already?

I just had the same problem. Just ignore the result of the client.verify() method, or just dont run it at all. It's up to you what to do if the certificate verification fails.
Works for me. No need to change the source code of WiFiClientSecure.cpp

[tuxedo0801]

Finally, I have some feedback:

When I use this sample as a starting point: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

and just ignore lines 52-56:

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

then https communication is working in principle. My Code:

void sendMessageHTTPS(String msg) {

  
  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to : '");
  Serial.print(host);
  Serial.println("'");
  
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

//  if (client.verify(fingerprint, host)) {
//    Serial.println("certificate matches");
//  } else {
//    Serial.println("certificate doesn't match");
//  }

  Serial.print("requesting URL: '");
  Serial.print(url);
  Serial.println("'");

  client.print(String("POST ") + url + " HTTP/1.1\r\n" +
          "Host: " + host + "\r\n" +
          "Connection: close\r\n" +
          "Accept: */*\r\n" +
          "User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n" +
          "Content-Type: application/json;charset=utf-8\r\n" +
          "Authorization: Bearer "+cloudDeviceToken+"\r\n" + 
          "Content-Length: "+msg.length()+"\r\n" +
          "\r\n" + 
          msg + "\r\n");

  Serial.println("request sent");
  unsigned long timeout = millis();
  while (client.available() == 0) {
    if (millis() - timeout > 5000) {
      Serial.println(">>> Client Timeout !");
      client.stop();
      return;
    }
  }
  
  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
    String line = client.readStringUntil('\r');
    Serial.print(line);
  }
}

but I face some ESP crashes after a few https requests. Still need to investigate what goes wrong.

[tuxedo0801]

decoded stacktrace:

Exception 29: StoreProhibited: A store referenced a page mapped with an attribute that does not permit stores
Decoding 29 results
0x402167d2: memset at /Users/igrokhotkov/e/newlib-xtensa/xtensa-lx106-elf/newlib/libc/machine/xtensa/../../../../.././newlib/libc/machine/xtensa/memset.S line 133
0x402231d6: more_comps at crypto/bigint.c line 672
0x40223368: alloc at crypto/bigint.c line 672
0x402235e9: regular_multiply at crypto/bigint.c line 672
0x4020eaa8: esp_yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 87
0x40223368: alloc at crypto/bigint.c line 672
0x4022409f: bi_barrett at crypto/bigint.c line 1289
0x402242f4: bi_mod_power at crypto/bigint.c line 1414
0x40223238: trim at crypto/bigint.c line 672
0x40225425: RSA_public at crypto/rsa.c line 252
:  (inlined by) RSA_encrypt at crypto/rsa.c line 286
0x40220e9c: send_client_key_xchg at ssl/tls1_clnt.c line 409
0x40221331: do_clnt_handshake at ssl/tls1_clnt.c line 123
0x40208200: ClientContext::_consume(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) ClientContext::read(char*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src\include/ClientContext.h line 253
0x4020eb20: __yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 96
0x40220ca4: do_handshake at ssl/tls1.c line 2005
:  (inlined by) basic_read at ssl/tls1.c line 1481
0x40221080: do_client_connect at ssl/tls1_clnt.c line 168
0x401004d8: malloc at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266\umm_malloc/umm_malloc.c line 1664
0x40220e0c: ssl_read at ssl/tls1.c line 2005
0x4020868d: SSLContext::connect(ClientContext*, char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) WiFiClientSecure::_connectSSL(char const*) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 321
0x40207cf2: WiFiClient::connect(IPAddress, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClient.cpp line 324
0x402088ad: WiFiClientSecure::connect(char const*, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 313
0x402038c4: sendMessageHTTPS(String) at C:\dev\git-repos\TestFW\Test/Test.ino line 489
0x4020dec3: String::changeBuffer(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df0f: String::reserve(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df41: String::copy(char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020e048: String::operator=(String const&) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x40203c74: loop at C:\dev\git-repos\TestFW\Test/Test.ino line 584
0x4020eaf4: loop_wrapper at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 124
0x4010070c: cont_norm at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/cont.S line 109

Hmm ...

[tuxedo0801]

@igrr
Any ideas what's going wrong?

[tuxedo0801]

Initially, the request was sent every 5 sec. It fails every 5-8 times, then I get stacktrace, device restarts.

I changed interval from 5sec to 60sec. This did not change anything. Crashed after 5 requests.

Beside the https-request, the device is serving a small webpage (which has not been accessed during my tests), queries NTP time and reads power-measurement data from HLW8012 chip via interrupt.
Everything works unless I activate HTTPS.
HTTP worked quite well, was extremly stable (no crash so far). And according to the stacktrace, it has something to do with https/rsa/crypto ...

Would be great if someone can shed some light on this issue.

[tuxedo0801]

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

[cimba007]

Any chance to get this working with HTTPClient? My responses could become lager and I don't want to deal with chunked responses manually...

[odelot]

a noob question: this fingerprint changes over time?

I know you are trying not to use them, but I have a valid one and I was thinking to hardcode it on my device.