How to diagnose the infamous read error 4?

[mcordova1967]

I have an httplib client that invokes a single web service, but with certain data in the response, I receive error 4, the problem is... SoapUI does not have a problem invoking the same service with the same inputs and receiving the same response. I already discarded timeout, the headers look the same for all the cases including the bad one, and the body too, except it is bigger in the case that fails, but not that much.

I know that error 4 can be triggered by a problem parsing the headers or the body, but I wonder if the people who wrote that code could give me a hint about where to look, I see no DEBUG traces to enable, so where can I place some stdout in order to track or trap the thing that causes de Read Error 4, it is just that it looks so normal on Soap UI console! and it is only this case that fails with httplib.

Running with debugger is not an option, it's a production setup in a remote area, I have no remote access, I just can send a new EXE and they will send me the resulting output.

I suspect a CR or LF missing here or there, but where? the fact that SoapUI has no trouble makes it look like a bug on my side.

Any advice will be much appreciated. Keep in mind that the same service is being invoked without problem, receiving the same headers for all the cases. Content is chunked, no content length in headers.

Regards

[yhirose]

Thanks for the feedback. Unfortunately I can't give you any help unless it can be reproducible on my machine. Also since this is my personal after work project, I don't give any support. Thanks for your understanding. :)

[mcordova1967]

I was able to reproduce the error in my local environment, simulating the web service by sending the same payload for the error case, as was captured by SoapUI in its raw HTTP log.

Despite being closed, I think that this may be of help to other users of the Client API.

I managed to discover that using Chunked for the transfer-encoding header caused the problem Read Error 4, and this happened because the body of the request did not have the correct data-size markers for each chunk, it seems that SoapUI and Curl are permissive regarding this, I was able to receive the response with Curl despite de Chunk-encoding not being correct, but httplib can't.

As soon as the format was fixed with the chunk sizes and all the CRLF and the end chunk 0-size, etc, etc, it worked.

I think that the "Read Error" message could be enriched with very few more details, like "bad chunk format", it would save hours of diagnostics.

Regards.

[yhirose]

the request did not have the correct data-size markers for each chunk

Thanks for the fine report! Could you show me more detail of the incorrect format that the server produces?
I am not sure if it's a right way to accept such a format. But if it's a common format, I would like to take a look at the format at least.

[mcordova1967]

This was the string I did sent with a plan epoll-server to emulate the SOAP server, it's based on the tracelog captured by SoapUI for the case in question:

`

inline std::string get_resp_body2() noexcept {

std::string msg = R"(<?xml version="1.0" encoding="UTF-8"?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns0:ConsultarListasCentralizadasResponse xmlns:ns0="http://xxxxx.xxxxx.com/"><Response><dataHeader><caracterAceptacion>B</caracterAceptacion><idTransaccion>0000000</idTransaccion><nombreOperacion>ConsultarListasCentralizadas</nombreOperacion></dataHeader><data><codigoRespuesta>I</codigoRespuesta><descripcionRespuesta>Si puede realizar la operacion, Se informa que la persona est&#225; marcada con lista(s) informativa(s)</descripcionRespuesta><informacionAdicional>CORH</informacionAdicional><mensajeRespuesta>Persona en Listas Informativas:CORH</mensajeRespuesta></data></Response></ns0:ConsultarListasCentralizadasResponse></S:Body></S:Envelope>)";

std::ostringstream res;
res	<< "HTTP/1.1 200 OK" << "\r\n"
	<< "Date: Wed, 31 Aug 2022 01:00:23 GMT" << "\r\n" 
	<< "Content-Type: text/xml" << "\r\n" 
	<< "X-ORACLE-DMS-ECID: xxxxxxxx-2ac1-4a3a-a2b6-9a2e32be9339-000000ec" << "\r\n"		
	<< "X-Frame-Options: DENY" << "\r\n"		
	<< "X-ORACLE-DMS-RID: 0" << "\r\n"		
	<< "Via: 1.1 util.busRegional.federado" << "\r\n"		
	<< "__request-out-time: 1661907623651" << "\r\n"
	<< "__response-in-time: 1661907630906" << "\r\n"
	<< "__request-size: 1745" << "\r\n"
	<< "__response-size: 768" << "\r\n"
	<< "Transfer-Encoding: chunked" << "\r\n"
	<< "\r\n" 
	<< std::hex << msg.size() << "\r\n" << msg << "\r\n"
	<< "0\r\n\r\n";
return res.str();

}

`

If I remove any of the chunk-length markers in the body, then I can reproduce the problem on the httplib client, the function that processes the chunked content will fail. If I remove the "Transfer-Encoding: chunked" then it will not fail, I guess it goes via the Content-Without-Length code path.

Curl did receive the response but it also did print some warnings.
This was the only way I could reproduce the Read Error 4 with this response, when I removed the "Transfer-Encoding: chunked" it worked right away, so I started reading about the spec for chunked and fixed the output accordingly. Then it worked fine with chunked encoding with httplib and with curl, without warnings.

I am invoking this service without problem, this case has the peculiarity that the response is a bit longer than most of the cases, maybe the server they use has a bug and for certain cases it won't format the output in the right way for chunked encoding, that's my guess, because I did not see any other source of Read Errror 4 in httplib client API.

[mcordova1967]

I can confirm the following with the latest master version of httplib when using chunked encoding:

if the POST payload does not end with

`

0\r\n\r\n

`
Then Read error 4 occurs. For instance, this end of payload triggers the error (which I suspect is my case):

`

0\r\n

`
The lack of the last CR LF makes it fail.

I changed line 3441 of method read_content_chunked() and it worked, for well-formed chunked content as well as my case above with a single CR LF at the end.

`

if (strcmp(line_reader.ptr(), "\r\n") || strcmp(line_reader.ptr(), "")) { break; }

`

I just added the comparison with "" empty string.

Regards.

[yhirose]

Thanks for the information. When I have time, I'll see the code if cpp-httplib can return more meaningful error core rather than Read Error 4.

[DarthAhsoka]

This problem is 50% likely to occur

[oysteinmyrmo]

I'll add my findings to this issue. I have a global httplib::Client that is reused for all HTTP calls. I experienced a lot of Read errors and investigated a bit. The behavior is as follows:

1. If I continuously make HTTP requests (i.e. around every second) I never get Read error.
2. If I wait more than approximately 5 seconds for the next HTTP request I get a Read error.
3. The HTTP request after a Read error never gets a Read error, regardless of how long I wait.

I have called httplib::Client::set_keep_alive(true) because something did not work initially when writing the code. But removing this now also removes the Read errors. I cannot reproduce when this call is removed.

So summarized this seems to be timing related on my end. Maybe something to do with timeouts?

[yhirose]

@oysteinmyrmo, thanks for the report. It has nothing to do with this issue, but it may be related to #1481. Could you try with the latest httplib. in the master branch to see if it fixes your problem? Thanks!

EDIT on 9/7/2024: This could be related to #1801. If so, you can fix it by setting cli.set_read_timeout to a longer value like 60 (1 min).

[yhirose]

@mcordova1967 I changed cpp-httplib to allow chunked transfer encoding data without the final CRLF. The pseudo-code in the RFC9112 code also doesn't handle the CRLF. So it seems OK.
https://www.rfc-editor.org/rfc/rfc9112.html#section-7.1.3