HTTPClient doesn't send the URL in HTTPS #1941

J6B · 2016-04-19T10:16:06Z

Basic Infos

Trying to make a GET to /something and receive a request for / only on web server side.

Hardware

Hardware: ESP-01
Core Version: 2.2.0

Description

My sketch do a GET using HTTPClient.
I was in release 2.0.0 and I just updated to 2.2.0 release (did not try in 2.1.0)
My sketch was able to send GET for HTTP and HTTPS URI using 2.0.0 release.
Since I updated to 2.2.0, HTTP works almost fine (getting frequent -11 return code while my server answering bellow 5sec).

But the issue is while i'm trying to send same GET request using HTTPS, My web server receive a wrong header requesting for / only.

Exemple:
I request for https://192.168.1.123/plugins/teleinfo/core/php/jeeTeleinfo.php?api=YQFMNcxGJ52XSrHYcF3U&ADCO=050522038502&HCHP=072686874&PAPP=00730
My apache2 log contains this:
192.168.1.104 - - [19/Apr/2016:11:57:21 +0200] "GET / HTTP/1.0" 400 0 "-" "-"

Settings in IDE

Module: Generic ESP8266 Module
Flash Size: 1MB
CPU Frequency: 80Mhz
Flash Mode: dio
Flash Frequency: 40Mhz
Upload Using: OTA
Reset Method: ck

Sketch

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

void setup() {
 //init Wifi
}

void loop() {

String completeURI = "https://192.168.1.123/plugins/teleinfo/core/php/jeeTeleinfo.php?api=YQFMNcxGJ52XSrHYcF3U&ADCO=050522038502&HCHP=072686874&PAPP=00730";

  //send HTTP request
  HTTPClient http;
  http.begin(completeURI);
  requestResult = http.GET();
  http.end();

}




<bountysource-plugin>

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/33106058-httpclient-doesn-t-send-the-url-in-https?utm_campaign=plugin&utm_content=tracker%2F14245935&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F14245935&utm_medium=issues&utm_source=github).
</bountysource-plugin>

J6B · 2016-04-19T10:43:48Z

I just tried with 2.1.0 and the same code works fine to reach server using HTTPS

…certificate fingerprint (#1941)

wouterds · 2017-10-12T11:58:21Z

I can confirm that I can't do any calls to https endpoints using the ESP8266HTTPClient.h lib.

Eg this code:

  Serial.print("API Call to: https://www.reddit.com/iphone.json");
  Serial.println();
    
  HTTPClient http;
  
  int beginResult = http.begin("https://www.reddit.com/iphone.json");
  http.addHeader("Content-Type", "text/plain");
  
  int httpCode = http.POST("Message from ESP8266");
  String response = http.getString();
  
  Serial.print("beginResult: ");
  Serial.print(beginResult);
  Serial.println();
  
  Serial.print("http: ");
  Serial.print(httpCode);
  Serial.println();
  
  Serial.print("response: ");
  Serial.println(response);
  Serial.println();
  
  http.end();

Should return 404 for the status code and {"message": "Not Found", "error": 404} for the response body. Instead it returns -1 and no contents (connection refused). This happens for all https endpoints.

J6B · 2017-10-12T12:07:21Z

Hi, this is because you need now to provide website certificat thumbprint in http.begin for https website.

http.begin("https://www.reddit.com/iphone.json","f8d1965323111e86e6874aa93cc7c52969fb22bf");
(I put the thumbprint of www.reddit.com :-) )

wouterds · 2017-10-12T12:11:50Z

What if it changes (thinking about tools like Letsencrypt and Cloudflare that generate free certificates on the fly that are only valid for short periods of time)?

igrr · 2017-10-12T12:15:14Z

That is a valid concern! I am currently working on that as part of #3700. That will add ability to verify the certificate in one of the several ways (fingerprint, subjectPublicKeyInfo, root cert), when using HTTPClient.

…

On Thu, Oct 12, 2017, 08:12 Wouter De Schuyter ***@***.***> wrote: What if it changes (thinking about tools like Letsencrypt and Cloudflare that generate free certificates on the fly)? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1941 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEJcevAIAJiR4XB76RgVaHldluQ162dwks5srgIPgaJpZM4IKjjO> .

J6B · 2017-10-12T12:16:53Z

Thanks igrr :-)

wouterds · 2017-10-12T12:17:46Z

Alright, good to know @igrr, thanks :-)!

Another question: is there a way to ignore it if you don't care about it? Eg like the unix command wget that has an option --no-check-certificate?

igrr · 2017-10-12T12:22:25Z

Currently there isn't a way to do that with HTTPClient. If you need this, you can copy the .cpp/.h files of HTTPClient into your sketch (renaming the .h file), and delete the line in .cpp file responsible for the check.

…

On Thu, Oct 12, 2017, 08:18 Wouter De Schuyter ***@***.***> wrote: Alright, good to know @igrr <https://github.com/igrr> :-)! Another question: is there a way to ignore it? If you don't care about? Eg like the unix command wget that has an option --no-check-certificate? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1941 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEJceszokb9CDpTnDOW-t8y-NdaM86vUks5srgN4gaJpZM4IKjjO> .

wouterds · 2017-10-12T12:31:10Z

I guess I'll do that for now, thanks!

hutch120 · 2017-10-17T11:31:24Z

Just wondering if anyone got this going? If so, any chance you can publish the relevant bits of the code to change in HTTPClient.cpp? This is what I tried...

I can see the entry point for https is:

bool HTTPClient::begin(String url, String httpsFingerprint)

So I detect the url using url.startsWith("https:"), then call client.begin with a fake string param like so:

  if (url.startsWith("https:")) {
    client.begin(url, "INSECURE_HTTPS"); //Specify request destination for HTTPS
  } else{    
    client.begin(url);  //Specify request destination for HTTP
  }

And tried changing this:

_transportTraits = TransportTraitsPtr(new TLSTraits(httpsFingerprint));

to this:

_transportTraits = TransportTraitsPtr(new TransportTraits());

Few other things I tried, but that didn't work.

return true on the verify function
delete the whole verify code block.

Any tips?

hutch120 · 2017-10-19T00:19:38Z

@wouterds Did you get this going? Any chance you can post the relevant changes? (See my attempts above)

wouterds · 2017-10-19T05:57:52Z

Hi! No, I did not get it working yet. I managed to copy the libs and include my custom libs, tried commenting all the verify methods out / make them early return true but without success. Still looking into it! Any help would be appreciated!

…

On 19 Oct 2017, at 02:19, Simon ***@***.***> wrote: @wouterds <https://github.com/wouterds> Did you get this going? Any chance you can post the relevant changes? (See my attempts above) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1941 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJ5BLDsb3uDb1muYiWrlTTwY9vQ13YGks5stpWpgaJpZM4IKjjO>.

TTnsp · 2017-10-19T11:50:14Z

@hutch120 : I have the same problem. So, to connect in https without certificat (fingerprint), i change the fonction "verify" in TLSTraits class in ESP8266HTTPClient.cpp. I made this fonction return always true, and that work fine.

hutch120 · 2017-10-19T21:26:44Z

Hi @igrr, Any chance you could help @wouterds and I out on this one? We've both had a crack at modifying the source in ESP8266HTTPClient.cpp but are coming up short.

Any tips?

zanechua · 2017-11-15T02:57:32Z

@hutch120

Actually @TTnsp already provided the answer.

Here's the modified code portion in the ESP8266HTTPClient.cpp file

class TLSTraits : public TransportTraits
{
public:
    TLSTraits(const String& fingerprint) :
        _fingerprint(fingerprint)
    {
    }

    std::unique_ptr<WiFiClient> create() override
    {
        return std::unique_ptr<WiFiClient>(new WiFiClientSecure());
    }

    bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        //Always return true to never validate certificate.
        return true;
    }

hutch120 · 2017-11-15T03:23:19Z

@zanechua Thanks... I'm pretty sure I tried returning true from the verify function, I even made a note that I tried it in my post.

I'll have another crack.... did you try returning true from the verify function @wouterds ?

zanechua · 2017-11-15T07:58:29Z

@hutch120

It works for me though. I'm using LetsEncrypt on my https server. It was a POST Request.

Did you modify the library directly? I actually copied them into my sketch folder and change the headers to use the local sketch libraries instead.

TTnsp · 2017-11-15T11:12:25Z

@zanechua

I'm in the same case of you and i copied the librairie into my sketch.

iamalonso · 2017-11-26T19:29:08Z

@zanechua Actually I'm trying to send a POST request from my Wemos d1 mini board to Mlab API via https and I have your same problem. I made a local copy of ESP8266HTTPClient.cpp into my sketch and I modified the verify function to return true but it didn't work.

zanechua · 2017-11-27T06:01:30Z

@alonsocarvajal

Not sure if I can help but what error are you getting?

iamalonso · 2017-11-27T17:14:05Z

@zanechua

Below I attach a portion of my code.

void loop() {

  // Get temperature and humidity
  float temperature = dht.readTemperature();
  float humidity = dht.readHumidity();

  // Creating JSON object
  JsonObject& root = jsonBuffer.createObject();
  root["temperature"] = temperature;
  root["humidity"] = humidity;
  size_t requestBodySize = root.measureLength() + 1;
  char requestBody[requestBodySize];
  root.printTo(requestBody, requestBodySize);

  // Starting  POST request 
  int httpBeginCode = http.begin("https://api.mlab.com/api/1/databases/MyDB/collections/MyColl?apiKey=XXXXXXXXXXXXXXXXXXXXXXXXX");
  http.addHeader("Content-Type", "application/json");
  int httpPOSTCode = http.POST(requestBody);

  // Printing responses
  Serial.println(httpBeginCode);
  Serial.println(httpPOSTCode);

  // Ending http request
  http.end();
  delay(3000);
}

The return codes that I'm receiving in httpBeginCode and httpPOSTCode variables are 0 and -1 respectively.

Remember that actually I modified the verify function located on ESP8266HTTPClient.cpp file to return true (see the next code).

bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        //return wcs.verify(_fingerprint.c_str(), host);
        return true;
    }

zanechua · 2017-11-27T17:34:34Z

Could you try this line with your post request instead?

int httpBeginCode = http.begin("https://api.mlab.com/api/1/databases/MyDB/collections/MyColl?apiKey=XXXXXXXXXXXXXXXXXXXXXXXXX", "imaginarykey");

iamalonso · 2017-11-27T19:14:40Z

@zanechua

I tried that alternative and now the return codes that I'm receiving in httpBeginCode and httpPOSTCode variables are 1 and -1 respectively. I suppose that now the http.begin() its ok but the http.POST() request stills fail.

Additionally I tried a GET request to get data from MLAB's and NASA's API but I received -1 as a return code. The code for this example is:

int httpBeginCode = http.begin("https://api.nasa.gov/planetary/apod?api_key=NNKOjkoul8n1CH18TWA9gwngW1s1SmjESPjNoUFo", "2E BD F3 9F C1 1D BC 49 96 25 6A DC 32 EB FE 2B F0 32 70 50");
int httpGETCode = http.GET();
  
// Print responses
Serial.println(httpBeginCode);
Serial.println(httpGETCode);
  
// Printing data
Serial.println(http.getString());
  
// Ending http request
http.end();
delay(3000);

In this occasion the return code httpBeginCode and httpGETCode are 1 and -1 respectively.

Therefore, in both cases (POST and GET request) the http.begin() is successful but the http.POST() and http.GET() methods are fail.

Note: I've also tried removing the return true in verify method into ESP8266HTTPClient.cpp file.

bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        //return wcs.verify(_fingerprint.c_str(), host);
        return true;
    }

bool verify(WiFiClient& client, const char* host) override
    {
        auto wcs = static_cast<WiFiClientSecure&>(client);
        return wcs.verify(_fingerprint.c_str(), host);
        //return true;
    }

Note 2: The fingerprint of Nasa API was obtained from https://www.grc.com/fingerprints.htm.

iamalonso · 2017-11-30T16:30:17Z

Hello @igrr, I've been trying to get data from NASA's API using the example HTTPSRequest.ino but I can not succeed. The same happens when I send a POST request to MLAB's API (see the post above).

My code based on HTTPSRequest.ino is:

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

const char* ssid = "10";
const char* password = "XXXXXXX";
const char* host = "api.nasa.gov";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char* fingerprint = "2E BD F3 9F C1 1D BC 49 96 25 6A DC 32 EB FE 2B F0 32 70 50";


void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

  String url = "/planetary/apod?api_key=NNKOjkoul8n1CH18TWA9gwngW1s1SmjESPjNoUFo";
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');
  if (line.startsWith("{\"state\":\"success\"")) {
    Serial.println("esp8266/Arduino CI successfull!");
  } else {
    Serial.println("esp8266/Arduino CI has failed");
  }
  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println("closing connection");
}

void loop() {
}

The serial monitor output is:

WiFi connected
IP address: 
192.168.160.10
connecting to api.nasa.gov
connection failed

It is possible that you can guide me regarding this problem?

shannonzchanz · 2017-12-07T04:17:28Z

Hello, I am experiencing the same problem as well. I attempted to change the return statement to true in my appdata, hardware folder.

`
class TLSTraits : public TransportTraits
{
public:
TLSTraits(const String& fingerprint) :
_fingerprint(fingerprint)
{
}

std::unique_ptr<WiFiClient> create() override
{
    return std::unique_ptr<WiFiClient>(new WiFiClientSecure());

}

bool verify(WiFiClient& client, const char* host) override
{
    auto wcs = reinterpret_cast<WiFiClientSecure&>(client);
    //return wcs.verify(_fingerprint.c_str(), host);          //original code
    return true;
}`

iamalonso · 2017-12-13T21:32:15Z

Hello @shannonzchanz

What kind of request are you doing: GET or POST?
What website are you sending these request to?
Could you attach your Arduino code?

liebman · 2017-12-13T23:14:47Z

@shannonzchanz try this PR #3933

shannonzchanz · 2017-12-14T02:23:29Z

I have resolved it! It's just that my esp8266 doesn't reach the localhost. So, I would have to publish it online. Thanks for the help.

jLynx · 2019-04-18T09:53:51Z

I am still having the same issue with not being able to load an https website.

This is what I have currently tried

http.begin("https://myFAKEsite.org/api/test", "6AEE739DA4EE7A518C87E2E96691F46136C9F060");      
    http.addHeader("Content-Type", "application/x-www-form-urlencoded");
    
    int httpCode = http.POST(postData);   //Send the request
    String payload = http.getString();                  //Get the response payload

But I still get -1, even with passing the current SH1 key for that specific site.

What am I doing wrong here?

hutch120 · 2019-09-07T22:11:25Z

Hi @theyogeshrathod Looks like maybe you are fairly new to Github, and just starting out with ESP boards? Please be aware that github issue lists are generally reserved for bugs and I'd suggest that a better place to ask random questions like this is on the StackOverflow website.

Also, if you do post a response to a thread please ensure you read and understand the issue first, or you just annoy people. This thread clearly has nothing to do with your problem, it is related to an issue with HTTPS as it says in the issue title.

Also, I suggest you check if a thread has been updated recently, this hasn't had an actual answer since 2017 and probably should be locked and closed.

@igrr please consider locking this issue.

theyogeshrathod · 2019-09-08T09:31:50Z

Oh, sorry @hutch120. I will just delete the above comment. Thanks!

igrr added type: bug component: libraries labels Apr 19, 2016

igrr added this to the 2.3.0 milestone Apr 19, 2016

igrr added a commit that referenced this issue Apr 22, 2016

Return error when HTTPClient::begin is called with HTTPS URL without …

81d3bb3

…certificate fingerprint (#1941)

igrr added the staged-for-release label Apr 22, 2016

igrr closed this as completed Jun 23, 2016

igrr removed the staged-for-release label Jun 23, 2016

zolv mentioned this issue Feb 11, 2017

HTTPClient https request fails if no fingerprint is provided #2902

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPClient doesn't send the URL in HTTPS #1941

HTTPClient doesn't send the URL in HTTPS #1941

J6B commented Apr 19, 2016 •

edited by igrr

Loading

J6B commented Apr 19, 2016

wouterds commented Oct 12, 2017

J6B commented Oct 12, 2017

wouterds commented Oct 12, 2017 •

edited

Loading

igrr commented Oct 12, 2017 via email

J6B commented Oct 12, 2017

wouterds commented Oct 12, 2017 •

edited

Loading

igrr commented Oct 12, 2017 via email

wouterds commented Oct 12, 2017

hutch120 commented Oct 17, 2017

hutch120 commented Oct 19, 2017

wouterds commented Oct 19, 2017 via email

TTnsp commented Oct 19, 2017

hutch120 commented Oct 19, 2017

zanechua commented Nov 15, 2017

hutch120 commented Nov 15, 2017

zanechua commented Nov 15, 2017 •

edited

Loading

TTnsp commented Nov 15, 2017

iamalonso commented Nov 26, 2017 •

edited

Loading

zanechua commented Nov 27, 2017

iamalonso commented Nov 27, 2017

zanechua commented Nov 27, 2017

iamalonso commented Nov 27, 2017 •

edited

Loading

iamalonso commented Nov 30, 2017

shannonzchanz commented Dec 7, 2017

iamalonso commented Dec 13, 2017

liebman commented Dec 13, 2017

shannonzchanz commented Dec 14, 2017

jLynx commented Apr 18, 2019

hutch120 commented Sep 7, 2019 •

edited

Loading

theyogeshrathod commented Sep 8, 2019

HTTPClient doesn't send the URL in HTTPS #1941

HTTPClient doesn't send the URL in HTTPS #1941

Comments

J6B commented Apr 19, 2016 • edited by igrr Loading

Basic Infos

Hardware

Description

Settings in IDE

Sketch

J6B commented Apr 19, 2016

wouterds commented Oct 12, 2017

J6B commented Oct 12, 2017

wouterds commented Oct 12, 2017 • edited Loading

igrr commented Oct 12, 2017 via email

J6B commented Oct 12, 2017

wouterds commented Oct 12, 2017 • edited Loading

igrr commented Oct 12, 2017 via email

wouterds commented Oct 12, 2017

hutch120 commented Oct 17, 2017

hutch120 commented Oct 19, 2017

wouterds commented Oct 19, 2017 via email

TTnsp commented Oct 19, 2017

hutch120 commented Oct 19, 2017

zanechua commented Nov 15, 2017

hutch120 commented Nov 15, 2017

zanechua commented Nov 15, 2017 • edited Loading

TTnsp commented Nov 15, 2017

iamalonso commented Nov 26, 2017 • edited Loading

zanechua commented Nov 27, 2017

iamalonso commented Nov 27, 2017

zanechua commented Nov 27, 2017

iamalonso commented Nov 27, 2017 • edited Loading

iamalonso commented Nov 30, 2017

shannonzchanz commented Dec 7, 2017

iamalonso commented Dec 13, 2017

liebman commented Dec 13, 2017

shannonzchanz commented Dec 14, 2017

jLynx commented Apr 18, 2019

hutch120 commented Sep 7, 2019 • edited Loading

theyogeshrathod commented Sep 8, 2019

J6B commented Apr 19, 2016 •

edited by igrr

Loading

wouterds commented Oct 12, 2017 •

edited

Loading

wouterds commented Oct 12, 2017 •

edited

Loading

zanechua commented Nov 15, 2017 •

edited

Loading

iamalonso commented Nov 26, 2017 •

edited

Loading

iamalonso commented Nov 27, 2017 •

edited

Loading

hutch120 commented Sep 7, 2019 •

edited

Loading