Deal with non-RFC compliant servers

[ycamper]

There are many servers on the internet intending to be an HTTP service, but in reality, they do not adequately conform to the HTTP RFC. For example, many servers on the internet do not send back the full terminating CRLF or Connection: close headers, but we might still want to treat as HTTP, for example:

$ echo -ne 'GET / HTTP/1.0\r\n\r\n' | nc X.X.X.X 8085 | hexdump -C
00000000  48 54 54 50 2f 31 2e 31  20 34 30 34 20 4e 6f 74  |HTTP/1.1 404 Not|
00000010  20 46 6f 75 6e 64 0d 0a                           | Found..|
00000018

Note that the connection is terminated at the server-end for services like this (and lacks the proper response headers)

This patch attempts to fix this single issue by catching the EOF, and if data was captured in the response, do not immediately error.

For example, running zgrab2 on the master branch will result in a response like the following:

$ echo X.X.X.X | ./zgrab2 http -p 8085
INFO[0000] started grab at 2023-03-22T13:17:31-04:00
{"ip":"X.X.X.X","data":{"http":{"status":"unknown-error","protocol":"http","result":{},"timestamp":"2023-03-22T13:17:31-04:00","error":"unexpected EOF"}}}
INFO[0000] finished grab at 2023-03-22T13:17:31-04:00
{"statuses":{"http":{"successes":0,"failures":1}},"start":"2023-03-22T13:17:31-04:00","end":"2023-03-22T13:17:31-04:00","duration":"184.162517ms"}

Whereas with this change, we see:

$ echo X.X.X.X | ./zgrab2 http -p 8085
INFO[0000] started grab at 2023-03-22T13:20:26-04:00
{"ip":"X.X.X.X","data":{"http":{"status":"success","protocol":"http","result":{"response":{"status_line":"404 Not Found","status_code":404,"protocol":{"name":"HTTP/1.1","major":1,"minor":1},"request":{"url":{"scheme":"http","host":"X.X.X.X:8085","path":"/"},"method":"GET","headers":{"accept":["*/*"],"user_agent":["Mozilla/5.0 zgrab/0.x"]},"host":"X.X.X.X:8085"}}},"timestamp":"2023-03-22T13:20:26-04:00"}}}
INFO[0000] finished grab at 2023-03-22T13:20:27-04:00
{"statuses":{"http":{"successes":1,"failures":0}},"start":"2023-03-22T13:20:26-04:00","end":"2023-03-22T13:20:27-04:00","duration":"184.098753ms"}

How to Test

Here is a simple python script that can be used to mimick what I am seeing on a large number of hosts the internet:

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_address = ('localhost', 8085)
sock.bind(server_address)
sock.listen(1)

while True:
    connection, client_address = sock.accept()
    try:
        http_response = "HTTP/1.1 404 Not Found\r\n"
        connection.sendall(http_response.encode())
    finally:
        connection.close()

Start this script up, and run echo 127.0.0.1 | zgrab2 http -p 8085