add response read timeout flag #4944

dogancanbakir · 2024-03-25T22:06:58Z

Proposed changes

Closes #4942.

test_template.yaml:

id: fuzz-dirs

info:
  name: fuzz-dirs
  author: savik
  severity: info

requests:
  - method: GET
    path:
      - "{{BaseURL}}"
    skip-variables-check: true
    matchers:
      - type: status
        condition: or
        status:
          - 200
          - 201
          - 204
          - 206
          - 304
          - 500

$ go run . -t test_template.yaml -u https://download.binance.com/electron-desktop/mac/production/binance.dmg -rsr 324009984 -rrt 10

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.2

                projectdiscovery.io

[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v3.2.2 (latest)
[INF] Current nuclei-templates version: v9.8.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 85
[INF] Templates loaded for current scan: 1
[WRN] Loaded 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[fuzz-dirs] [http] [info] https://download.binance.com/electron-desktop/mac/production/binance.dmg

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

tarunKoyalwar

i think max-http-body-size is meant to be handled gracefully[i.e non-fatal/error] ( without error and only use first 4MB) . but currently it errors which is not expected behaviour

$ ./nuclei -t fuzz.yaml  -target https://download.binance.com/electron-desktop/mac/production/binance.dmg --debug

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.2

		projectdiscovery.io

[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v3.2.2 (latest)
[INF] Current nuclei-templates version: v9.8.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 85
[INF] Templates loaded for current scan: 1
[WRN] Loaded 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [fuzz-dirs] Dumped HTTP request for https://download.binance.com/electron-desktop/mac/production/binance.dmg

GET /electron-desktop/mac/production/binance.dmg HTTP/1.1
Host: download.binance.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:109.0) Gecko/20100101 Firefox/122.0
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[WRN] [fuzz-dirs] Could not execute request for https://download.binance.com/electron-desktop/mac/production/binance.dmg: [:RUNTIME] got err while executing https://download.binance.com/electron-desktop/mac/production/binance.dmg <- could not generate response chain: error reading response body: could not read response body: http: request body too large
[INF] No results found. Better luck next time!

keeping the 4MB limit , i think we should handle errors returned by responseChain in protocols/http/http.go more appropriately i.e in this case , we can warn user and use the first 4MB instead of erroring out

dogancanbakir · 2024-03-27T12:48:16Z

depends on projectdiscovery/utils#378

dogancanbakir · 2024-03-27T14:58:54Z

go run . -t test_template.yaml -u https://download.binance.com/electron-desktop/mac/production/binance.dmg 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.2

                projectdiscovery.io

[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v3.2.2 (latest)
[INF] Current nuclei-templates version: v9.8.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 85
[INF] Templates loaded for current scan: 1
[WRN] Loaded 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[fuzz-dirs] [http] [info] https://download.binance.com/electron-desktop/mac/production/binance.dmg

tarunKoyalwar

lgtm ! implementation wise

$  ./nuclei -t a.yaml  -target https://download.binance.com/electron-desktop/mac/production/binance.dmg --debug-req -v -svd             

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.2

		projectdiscovery.io

[VER] Started metrics server at localhost:9092
[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v3.2.2 (outdated)
[INF] Current nuclei-templates version: v9.8.1 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 77
[INF] Templates loaded for current scan: 1
[WRN] Loaded 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[DBG] HTTP Protocol request variables: 
	1. BaseURL => https://download.binance. .... ac/production/binance.dmg
	2. DN => binance
	3. FQDN => download.binance.com
	4. File => binance.dmg
	5. Host => download.binance.com
	6. Hostname => download.binance.com
	7. Input => https://download.binance. .... ac/production/binance.dmg
	8. Path => /electron-desktop/mac/production
	9. Port => 443
	10. RDN => binance.com
	11. RootURL => https://download.binance.com
	12. SD => download
	13. Scheme => https
	14. TLD => com
	15. ip => 

[INF] [fuzz-dirs] Dumped HTTP request for https://download.binance.com/electron-desktop/mac/production/binance.dmg

GET /electron-desktop/mac/production/binance.dmg HTTP/1.1
Host: download.binance.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Mobile/15E148 Safari/604.1
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[VER] [fuzz-dirs] Sent HTTP request to https://download.binance.com/electron-desktop/mac/production/binance.dmg
[DBG] Http Protocol response variables: 
	1. BaseURL => https://download.binance. .... ac/production/binance.dmg
	...
	44. x_nuclei_ignore_error => http: request body too large

[fuzz-dirs:body] [http] [info] https://download.binance.com/electron-desktop/mac/production/binance.dmg [1.048576e+07]

x_nuclei_ignore_error => http: request body too large

More

requesting changes in context of
- merge conflict
- integration test failure ( most likely flaky test but we should confirm just in case )

tarunKoyalwar · 2024-04-16T11:52:52Z

minor update

due to incorrect time unit response-read-timeout value was set to 5 ns instead of 5 sec which caused network tests to fail

 ./nuclei -h |& grep read-timeout
   -rrt, -response-read-timeout value    response read timeout in seconds (default 5ns)
   -irt, -input-read-timeout value  timeout on input read (default 3m0s)

also we have asnmap <-> nuclei unit test , looks like we need to add env secret with pdcp key cc: @ehsandeep @dogancanbakir

dogancanbakir · 2024-04-16T12:09:16Z

oh yes!

tarunKoyalwar

lgtm!

the failing test seems to be specific to windows environment for scan-all-ips feature , i think we can track and fix it in follow-up pr since it doesn't seem to be related to this and is specific to windows

cc: @dogancanbakir @ehsandeep

savushkin-yauheni · 2024-04-24T09:33:21Z

Hi @dogancanbakir @ehsandeep
I'm the author of #4942

Right now we have a bug in final calculation of maxBodylimit:

	maxBodylimit := MaxBodyRead // 10MB

	gologger.Info().Msgf("1: %s", maxBodylimit)
	if request.MaxSize > 0 {
		maxBodylimit = int64(request.MaxSize)
		gologger.Info().Msgf("2: %s", maxBodylimit)

	}
	if request.options.Options.ResponseReadSize != 0 {
		maxBodylimit = int64(request.options.Options.ResponseReadSize)
		gologger.Info().Msgf("3: %s", maxBodylimit)

	}

[INF] 1: %!s(int64=4194304)
[INF] 2: %!s(int64=20485760)
[INF] 3: %!s(int64=10485760)

So even if I set upmax-size in template it would be overwritten by such code ( I set up 20mb but instead we got 10mb):

if request.options.Options.ResponseReadSize != 0 {
		maxBodylimit = int64(request.options.Options.ResponseReadSize)		
}

My command:

nuclei -t fuzz-dirs.yaml -target https://download.binance.com/electron-desktop/mac/production/binance.dmg

I don't use ResponseReadSize option. so it shouldn't affect the calculation

Regards

dogancanbakir · 2024-04-24T09:45:28Z

Hi @savushkin-yauheni,
The request.options.Options.ResponseReadSize option has a default value, which is why it overrides the max limit. Ref https://github.com/projectdiscovery/nuclei/blob/dev/cmd/nuclei/main.go#L299

savushkin-yauheni · 2024-04-24T09:50:10Z

@dogancanbakir
But it doesn't make sense. does it mean that we can't control max-size through templates?
If so, why do we need it at all.

dogancanbakir · 2024-04-24T09:52:40Z

Yes; creating another issue to work on this.

add response read timeout flag

e8b5e15

dogancanbakir requested a review from ehsandeep March 25, 2024 22:06

dogancanbakir self-assigned this Mar 25, 2024

dogancanbakir requested a review from Mzack9999 March 26, 2024 14:58

fix test

6af5faa

tarunKoyalwar requested changes Mar 26, 2024

View reviewed changes

bump utils

464abd8

dogancanbakir requested a review from tarunKoyalwar March 27, 2024 14:58

tarunKoyalwar requested changes Apr 9, 2024

View reviewed changes

dogancanbakir added 2 commits April 15, 2024 17:06

Merge branch 'dev' into add_response_read_timeout_flag

f0a0550

fix network tests

4f04de8

dogancanbakir requested a review from tarunKoyalwar April 15, 2024 15:14

fix incorrect unit of response-read-timeout unit

033b8e3

tarunKoyalwar approved these changes Apr 23, 2024

View reviewed changes

ehsandeep approved these changes Apr 24, 2024

View reviewed changes

ehsandeep merged commit 8676cb6 into dev Apr 24, 2024
9 of 12 checks passed

ehsandeep deleted the add_response_read_timeout_flag branch April 24, 2024 07:35

dogancanbakir mentioned this pull request Apr 24, 2024

Can't control max-size through templates #5094

Closed

BrewTestBot mentioned this pull request Apr 25, 2024

nuclei 3.2.5 Homebrew/homebrew-core#170078

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add response read timeout flag #4944

add response read timeout flag #4944

dogancanbakir commented Mar 25, 2024

tarunKoyalwar left a comment •

edited

Loading

dogancanbakir commented Mar 27, 2024

dogancanbakir commented Mar 27, 2024

tarunKoyalwar left a comment

tarunKoyalwar commented Apr 16, 2024

dogancanbakir commented Apr 16, 2024

tarunKoyalwar left a comment

savushkin-yauheni commented Apr 24, 2024 •

edited

Loading

dogancanbakir commented Apr 24, 2024 •

edited

Loading

savushkin-yauheni commented Apr 24, 2024

dogancanbakir commented Apr 24, 2024

add response read timeout flag #4944

add response read timeout flag #4944

Conversation

dogancanbakir commented Mar 25, 2024

Proposed changes

Checklist

tarunKoyalwar left a comment • edited Loading

Choose a reason for hiding this comment

dogancanbakir commented Mar 27, 2024

dogancanbakir commented Mar 27, 2024

tarunKoyalwar left a comment

Choose a reason for hiding this comment

More

tarunKoyalwar commented Apr 16, 2024

minor update

dogancanbakir commented Apr 16, 2024

tarunKoyalwar left a comment

Choose a reason for hiding this comment

savushkin-yauheni commented Apr 24, 2024 • edited Loading

dogancanbakir commented Apr 24, 2024 • edited Loading

savushkin-yauheni commented Apr 24, 2024

dogancanbakir commented Apr 24, 2024

tarunKoyalwar left a comment •

edited

Loading

savushkin-yauheni commented Apr 24, 2024 •

edited

Loading

dogancanbakir commented Apr 24, 2024 •

edited

Loading