Raw Request Bug [304 Response = i/o timeout]

[Impact-I]

Nuclei version:

[INF] Current Version: 2.8.9

Current Behavior:

When sending an unprocessed request with a 'If-Modified-sinCe' header, no response is received and the request goes into a long timeout.

Expected Behavior:

An raw request must receive a valid response.

Steps To Reproduce:

Template:

id: C
info:
  name: C
  author: C
  description: C
 
requests:
  - raw:
      - |+
        GET /service-worker.js HTTP/1.1
        Host: rutube.ru
        If-Modified-Since: Tue, 21 Feb 2023 14:51:35 GMT

    unsafe: true

Command:

./nuclei -debug -t temple.yaml -u https://rutube.ru

2. Notice that instead of response

HTTP/1.1 304 Not Modified
Connection: close
Cache-Control: max-age=0
Date: Sat, 25 Feb 2023 17:42:57 GMT
Last-Modified: Tue, 21 Feb 2023 14:51:35 GMT
Server: QRATOR
Set-Cookie: uuid=94823265-14f0-d067-34b0-180e;

we get exception could not read http body: read tcp 192.168.1.1:8080->198.248.233.148:443: i/o timeout

You can remove unsafe: true from the yaml file to see the correct response.
OS: Windows 10, Ubuntu 18.04

[ehsandeep]

@Impact-I are you still able to reproduce this?

bat test.yaml 
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: test.yaml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ id: C
   2   │ info:
   3   │   name: C
   4   │   author: C
   5   │   description: C
   6   │  
   7   │ requests:
   8   │   - raw:
   9   │       - |+
  10   │         GET /service-worker.js HTTP/1.1
  11   │         Host: rutube.ru
  12   │         If-Modified-Since: Tue, 21 Feb 2023 14:51:35 GMT
  13   │ 
  14   │     unsafe: true
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
nuclei $nuclei -v -t test.yaml -u https://rutube.ru

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.1

		projectdiscovery.io

[INF] Using Nuclei Engine 2.9.1 (latest)
[INF] Using Nuclei Templates 9.4.1 (latest)
[INF] Templates added in last update: 69
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[VER] [C] Sent HTTP request to https://rutube.ru/service-worker.js
[INF] No results found. Better luck next time!

[Impact-I]

@ehsandeep, can't, need to find any 304 response endpoint that handles If-Modified-Since.

Sorry I won't be able to do that anytime soon, it would be cool if you could find it yourself just turn on the proxy and look at the traffic.
If you can't, then give me time to find one. Thanks for your hard work!

[jimen0]

I was able to reproduce this, @ehsandeep.

How to set the test target server up

Host this Caddyfile somewhere.

http://localhost http://127.0.0.1 {
   root * /tmp/aaa
   encode gzip
   file_server
}

Note: make sure you update the HTTP header If-Modified-Since value. Your /tmp/aaa directory just holds a bunch of dummy JS files.

With unsafe this results in a timeout while trying to read the response body (something that shouldn't happen as 304 responses are guaranteed to come with no body in them!):

$ nuclei -t ./templates/test.yaml -u http://REDACTED/ -debug
...

[INF] Using Nuclei Engine 2.9.1 (latest)
[INF] Using Nuclei Templates 9.4.1 (latest)
[INF] Templates added in last update: 69
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] [C] Dumped HTTP request for http://REDACTED/foo.js

GET /foo.js HTTP/1.1
Host: REDACTED
If-Modified-Since: Wed, 05 Apr 2023 17:17:04 GMT

[WRN] [C] Could not execute request for http://REDACTED/: could not read http body: read tcp REDACTED:60004->REDACTED:80: i/o timeout
[INF] No results found. Better luck next time!

This is caused by nuclei using httpx.

$ httpx -debug -u 'http://REDACTED/foo.js' -H 'If-Modified-Since: Wed, 05 Apr 2023 17:17:04 GMT' -unsafe
...

[INF] Current httpx version v1.2.9 (latest)
[INF] Dumped HTTP request for http://REDACTED

GET /foo.js HTTP/1.1
Connection: close
Host:  REDACTED
If-Modified-Since: Wed, 05 Apr 2023 17:17:04 GMT
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko) Maxthon/4.5.2

[DBG] Failed 'http://REDACTED/foo.js': [urlutil:RUNTIME] failed to parse url got empty input

[jimen0]

@ehsandeep - per the HTTP specification: https://www.rfc-editor.org/rfc/rfc9110.html#name-304-not-modified

A 304 response is terminated by the end of the header section; it cannot contain content or trailers.

Here is a patch to fix this issue.

First, for rawhttp, the following changes should be introduced:

diff --git a/client/status.go b/client/status.go
index ecc1525..a371452 100644
--- a/client/status.go
+++ b/client/status.go
 func (s Status) IsRedirect() bool {
+       // Per RFC 9110 section 15.4.5 a 304 response is terminated by the end of the header section and it refers to a local resource.
+       // No further requests are supposed to be issued after a 304 response is received.
+       if s.Code == REDIRECTION_NOT_MODIFIED {
+               return false
+       }
        return s.Code >= REDIRECTION_MULTIPLE_CHOICES && s.Code < CLIENT_ERROR_BAD_REQUEST
 }
 func (s Status) IsError() bool { return s.Code >= CLIENT_ERROR_BAD_REQUEST }

Then, the new rawhttp version should be used (for testing I just used a go mod replace directive) and the following change should be made to httpx:

diff --git a/common/httpx/httpx.go b/common/httpx/httpx.go
index 56224d9..3948f83 100644
--- a/common/httpx/httpx.go
+++ b/common/httpx/httpx.go
@@ -204,8 +204,8 @@ get_response:
        resp.Raw = string(rawResp)
        resp.RawHeaders = string(headers)
        var respbody []byte
-       // websockets don't have a readable body
-       if httpresp.StatusCode != http.StatusSwitchingProtocols {
+       // websockets don't have a readable body and 304 responses are termnated by the end of the header section
+       if httpresp.StatusCode != http.StatusSwitchingProtocols || httpresp.StatusCode == http.StatusNotModified {
                var err error
                respbody, err = io.ReadAll(io.LimitReader(httpresp.Body, h.Options.MaxResponseBodySizeToRead))
                if err != nil && !shouldIgnoreBodyErrors {

The above prevents httpx from attempting to read body on HTTP 304 responses.

$ httpx -debug -u 'https://REDACTED/foo.js' -H 'If-Modified-Since: Thu, 06 Apr 2023 09:10:06 GMT' -unsafe
...

[INF] Current httpx version v1.2.9 (latest)
[INF] Dumped HTTP request for https://REDACTED

GET /foo.js HTTP/1.1
If-Modified-Since: Thu, 06 Apr 2023 09:10:06 GMT
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15
Connection: close
Host: REDACTED

[INF] Dumped HTTP response for https://REDACTED

HTTP/1.1 304 Not Modified
Connection: close
Date: Thu, 06 Apr 2023 09:46:26 GMT
Etag: "rsos4um"
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff

https://REDACTED

---

It's sad that this requires introducing changes in both httpx and rawhttp, but given how the codebase is structured today, it's the cleanest way I came up with.

FYI @Impact-I in case you want to apply the patches before these are introduced into a tagged version of the tools 😉 I've confirmed introducing this changes to nuclei makes it work as expected.

Regards,

[Mzack9999]

@Impact-I Thanks a lot for reporting this issue. I've opened two PRs with the solution proposed by @jimen0 (Thanks!)

While this is a valid bug, I'd recommend avoiding using unsafe: true for requests that are RFC compliant as the rawhttp library is minimal and doesn't handle all the edge cases covered instead by the standard library (there is a wip on extending the standard library to avoid all these edge cases at projectdiscovery/rawhttp#112)