Help with HTTPS

[torntrousers]

I'm trying to use the new https support but can't get it to work so wonder if someone could see what the problem is (sorry if this is leaping ahead, i know the https code hasn't been released yet). I've built the latest git code from today (1f8c14d) and started with the HTTPSRequest sample. That works ok (though does always seem to get a wdt reset after closing the connection), so i then changed it to do a GET to https://www.btopenzone.com:8443, and that worked ok too, so then updated it to try to do a POST to http://www.btopenzone.com:8443/tbbLogon but that doesn't work and all i get back is a stream of ÿ characters.

This is the code i have, can anyone see a problem?

/*
 *  HTTP over TLS (HTTPS) example sketch
 *
 *  This example demonstrates how to use
 *  WiFiClientSecure class to access HTTPS API.
 *  We fetch and display the status of
 *  esp8266/Arduino project continous integration
 *  build.
 *
 *  Created by Ivan Grokhotkov, 2015.
 *  This example is in public domain.
 */

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

const char* ssid = "BTHub5-72W5";
const char* password = "xxxxxxxxxxx";

//const char* host = "api.github.com";
//const int httpsPort = 443;
const char* host = "www.btopenzone.com";
const int httpsPort = 8443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char* fingerprint = "CF 05 98 89 CA FF 8E D8 5E 5C E0 C2 E4 F7 E6 C3 C7 50 DD 5C";

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);
  while (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
//    return;
  }

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

//  String url = "/repos/esp8266/Arduino/commits/esp8266/status";
  String url = "/tbbLogon";
  String postData = "username=qaz123@btinternet.com&password=abc123&xhtmlLogon=https://www.btopenzone.com:8443/tbbLogon";
  Serial.print("requesting URL: ");
  Serial.println(url);
/*
  client.print(String("POST ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");
*/
  client.print("POST /tbbLogon HTTP/1.1\n");
  client.print("Host: www.btopenzone.com:8443\n");
  client.print("Connection: close\n");
//  client.print("Connection: keep-alive\n");
//  client.print("User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0\n");
  client.print("Content-Type: application/x-www-form-urlencoded\n");
  client.print("Content-Length: ");
  client.print(postData.length()+2);
  client.print("\n\n");
  client.print(postData);

  Serial.println("request sent");

  Serial.println("Receiving response");

  while (client.connected()) {
    /*
    String line = client.readStringUntil('\n');
    Serial.println(line);
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
    */
    Serial.write(client.read());
    delay(10);
  }
  Serial.println("2");
  String line = client.readStringUntil('\n');
  Serial.println(line);
  Serial.println("closing connection");
}

void loop() {
    delay(1000);
    Serial.println("done");
}

As an alternative test this does work and returns an html page:

curl --data "username=qaz123%40btinternet.com&password=abc123&xhtmlLogon=https://www.btopenzone.com:8443/tbbLogon" https://www.btopenzone.com:8443/tbbLogon

[igrr]

Serial.write(client.read());

When there is no character to be read, client.read() returns -1, which is then converted to a character by Serial.write. This results in a character with ASCII code of 255, which may look like ÿ depending on the terminal. You might want to check if client.available() > 0 before calling read.

There may be other issues, but this one is the most obvious.

Edit: perhaps you are not getting the reply because the server is expecting two more characters?

 client.print(postData.length()+2);

And then you send just postData.length() characters after the headers.

[torntrousers]

Ah ok, i should have known that. But i did have the client.available() check before and then it just hangs never getting back a response. Trying again now adding this code after the Serial.println("Receiving response") then it just sits forever printing out dots:

  while (client.available() < 1) {
    Serial.print(".");
    delay(100);
  } 

[igrr]

What happens if you change Content-Length header to match the real size of content?

[Links2004]

your header lineending need to be \r\n look in the http RFC for more info

[torntrousers]

Something like this?

  client.print("POST /tbbLogon HTTP/1.1\r\n");
  client.print("Host: www.btopenzone.com:8443\r\n");
  client.print("Connection: close\r\n");
  client.print("Content-Type: application/x-www-form-urlencoded\r\n");
  client.print("Content-Length: ");
  client.print(postData.length());
  client.print("\r\n");
  client.print(postData);
  Serial.println("request sent");
  Serial.println("Receiving response");

  while (client.available() < 1) {
    Serial.print(".");
    delay(100);
  }

Still just dots and no response comes back.

[igrr]

You still seem to be missing an empty line after the headers (see the \r\n\r\n sequence in the original example).

[torntrousers]

Ok, rearranging to be more closely like the original example:

  client.print(String("POST ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Content-Type: application/x-www-form-urlencoded\r\n" +
               "Content-Length: " + postData.length() + "\r\n" +
               "Connection: close\r\n\r\n");

  client.print(postData);

  Serial.println("request sent");
  Serial.println("Receiving response");

  while (client.available() < 1) {
    Serial.print(".");
    delay(100);
  }

but still just dots and no response.

[torntrousers]

Now i've tried running tcpmon on a local computer with a local non-ssl http port being forwarded to www.btopenzone.com port 8443 as SSL, and then changing the esp sketch to use WiFiClient instead of WiFiClientSecure and to use the tcpmon host address, but the rest of the sketch unchanged.
That works, tcpmon forwards the post request to btopenzone as ssl which returns an html response which the esp sketch receives and prints out. So the esp sketch must be formating the POST request and headers ok.
What else could the problem be, some bug with WiFiClientSecure?

[igrr]

Correct, if the same POST request works with WiFiClient then it's likely a bug in WiFiClientSecure.

[Links2004]

at every print you send a tcp package some servers dont like this.
you can try it like this:
#773 (comment)

you also shut have "\r\n\r\n" before postData or the server will interpreter it as invalidate request.

[torntrousers]

Getting a bit further...i think there might be a bug in WifiClientSecure - it looks like after sending the request client.available() is always returning 0 even when a response has come back, but doing client.readStringUntil('\n') does return the first line, and then client.available() starts working properly. You can see this by modifying the HTTPSRequest sample to have a while loop printing client.available() before and after the client.readStringUntil call.

So accounting for that i can get the https POST to work ok, yipee!

Unfortunately it still doesn't work for what i want, which is trying to have the ESP logon to a FON Wifi network. Not really for this git issue but i wonder if it needs to have two connections, one for the https connection which is kept open and another to do the other http requests on - can the esp do multiple client connections?

[Links2004]

yes up to 5 TCP and 4 UDP connections at one time.

Arduino/hardware/esp8266com/esp8266/libraries/ESP8266WiFi/src/lwip/opt.h

Line 243 in ea302aa

#define MEMP_NUM_UDP_PCB 4

[torntrousers]

Ok thanks for confirming that. Doing some debugging with netstat on a pc i don't think it uses multiple connections anyway. Not sure how it works, in case anyone has any suggestions here's what happens:

There are lots of FON access points named (in the UK) BTWifi-with-FON that are open with no security. So you connect to them and then if you try to use a browser then you can't connect to anything until you go to a FON web page with a form where you have to logon with a userid password, and then after that you're connected and all internet access works normally (for a short while after which you need to reenter the userid/password again).

I traced the web page form and it just does a POST with the userid/password so now i can replicate the logon with a curl command instead of using the web page form: curl --data "username=qaz123%40btinternet.com&password=abc123&xhtmlLogon=https://www.btopenzone.com:8443/tbbLogon" https://www.btopenzone.com:8443/tbbLogon

So i was hoping to replicate that on an ESP and do the https post and then be able to make other http connections to publish data. The https post seems to work fine and the headers and response the esp gets back look the same as what the curl command gets, but then further WifiClient requests on the ESP don't work and just fail to connect.

I thought it might just be using the mac address against the userid/password or somehting like that but i guess there must be more to it.

Any ideas?

[Links2004]

for me this looks like the FON access points runs a transparent proxy.
so the allow your ip or mac for some time to use the proxy.
where do you try to connect to? may they only allow some ports / protocols.

[torntrousers]

It started working! Don't know what i did, just tidied the code up a bit i thought, but now it works ok. Very pleased. Thanks for all the help.

Would be good to get the WifiClientSecure.available bug fixed though so i'll leave this open for that.