[FEATURE REQUEST] http/2 support #947

RavySena · 2023-07-19T13:52:50Z

Describe the bug
Sometimes on some sites the program freezes right at the beginning of the scan, maybe it's the request for robots.txt that freezes?

To Reproduce
Steps to reproduce the behavior:

feroxbuster -r -A -n -k -t 300 --rate-limit 300 -L 300 -u https://domain.com -w {wordlists}/disclosure.txt -H "Cookie:{cookie}" --no-state --json -o {caminho}/{dominio}/ferox/{nome_arq}.json -vv

Traceback / Error Output
If applicable, add error output to help explain your problem.

INF      1.949 feroxbuster::event_handlers::scans scan handler received https://domain.com - beginning scan
INF      1.949 feroxbuster::scanner::ferox_scanner Starting scan against: https://domain.com
WRN      8.962 feroxbuster::utils Error while making request: error sending request for url (https://domain.com/robots.txt): operation timed out
WRN      8.964 feroxbuster::event_handlers::scans error sending request for url (https://domain.com/robots.txt): operation timed out

Environment:

feroxbuster version: 2.10.0
Linux kali 6.1.0-kali9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1kali1 (2023-05-12) x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

epi052 · 2023-07-29T00:54:11Z

howdy! can you elaborate on freezing? it never proceeds/finishes? It proceeds after the robots.txt timeout? something else?

stale · 2023-09-16T19:59:15Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

devx00 · 2023-10-03T10:26:57Z

This is happening to me as well on v2.10.0. It says operation timed out for the target url but I can access that url just fine from any other application.

Weirdly, if I proxy through burp it works though. I thought it might be something to do with http/2 but I couldn't confirm that.

epi052 · 2023-10-03T10:54:42Z

hey @devx00 could you provide the command you used? is it sporadic, or consistent? if you have a consistently bugged target that you can share, that'd be super helpful

devx00 · 2023-10-03T11:00:37Z

hey @devx00 could you provide the command you used? is it sporadic, or consistent? if you have a consistently bugged target that you can share, that'd be super helpful

Sure let me gather some more data for you. It was happening consistently once it started but prior to that I was able to scan the target.

It could have been some option I started using. I will try and find the specific argument if I can.

Just updating my computer right now in case it was something at the OS level but I doubt it is since everything else could access it fine. I'll try again in a few minutes.

epi052 · 2023-10-03T11:03:00Z

no hurry, thanks for helping run this down!

devx00 · 2023-10-03T11:15:17Z

Ok so I just tried again and I actually can't get it to work at all for this target.

The minimal command that fails for me is

feroxbuster -u https://www.rei.com/ -a 'Firefox/118.0'

The -a 'Firefox/118.0' is necessary because it seems to block certain User-Agents. If you try it with curl you will see.

curl https://www.rei.com/ # fails
curl https://www.rei.com/ -H 'User-Agent: Firefox/118.0' # succeeds

epi052 · 2023-10-03T11:39:19Z

Does rei have safe harbor?

devx00 · 2023-10-03T11:39:51Z

Does rei have safe harbor?

Ya Im scanning for a BBP.
https://hackerone.com/rei_bbp/policy_scopes?type=team

epi052 · 2023-10-03T11:41:02Z

perfect, ill look into it

devx00 · 2023-10-03T11:43:20Z

I may see the problem but I'm not positive.

It looks like it doesnt use the supplied headers when performing the connectivity check here

feroxbuster/src/heuristics.rs

Lines 106 to 111 in e3ec3ae

    
           for target_url in target_urls { 
        
               let url = FeroxUrl::from_string(target_url, self.handles.clone()); 
        
               let request = skip_fail!(url.format("", None)); 
        
               let result = logged_request(&request, DEFAULT_METHOD, None, self.handles.clone()).await;

which would cause it to fail in this case. Am I understanding that correctly? I dont see any context being passed in that would cause the headers to be set.

devx00 · 2023-10-03T11:48:47Z

A little more context for why I was looking into the connectivity check

  TRC      1.806 feroxbuster::banner::container exit: check_for_updates -> UpToDate
  TRC      1.807 feroxbuster::heuristics enter: connectivity_test(["https://www.rei.com/"])
  TRC      1.807 feroxbuster::url enter: format(, None)
  TRC      1.807 feroxbuster::utils enter: parse_url_with_raw_path(https://www.rei.com/)
  TRC      1.807 feroxbuster::url exit: format_url -> https://www.rei.com/
  TRC      1.807 feroxbuster::utils enter: make_request(Configuration::Client, https://www.rei.com/, Default, UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x14500a200, ta
  il_position: 1 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 3, rx_fields: "..." } } })
  TRC      8.809 feroxbuster::utils exit: make_request -> error sending request for url (https://www.rei.com/): operation timed out
  WRN      8.810 feroxbuster::utils Error while making request: error sending request for url (https://www.rei.com/): operation timed out
  WRN      8.810 feroxbuster::utils err: error sending request for url (https://www.rei.com/): operation timed out
  TRC      8.810 feroxbuster::scan_manager::scan_container enter: get_base_scan_by_url(https://www.rei.com/)
  TRC      8.810 feroxbuster::scan_manager::scan_container enter: get_base_scan_by_url -> None
  WRN      8.811 feroxbuster::heuristics error sending request for url (https://www.rei.com/): operation timed out

epi052 · 2023-10-03T11:50:22Z

good thought, but i dont think so. if you check burp, it should go something like

check for updates (github)
connectivity
robots

all of them (minus maybe update check) should have the supplied headers

epi052 · 2023-10-03T11:56:45Z

it looks like a HTTP/2 problem. if i use the rustls backend instead of native-tls, i don't get any timeouts. performance goes down significantly tho. and i know i looked into different tls backends at some point with another guy that was submitting a PR. There were reasons we landed on native, but id have to dig up the convo to remember exatly why

devx00 · 2023-10-03T11:57:13Z

Whats weird is if I proxy it through burp. Literally just add --burp to the request it works.

TRC      1.740 feroxbuster::banner::container exit: check_for_updates -> UpToDate
TRC      1.741 feroxbuster::heuristics enter: connectivity_test(["https://www.rei.com/"])
TRC      1.741 feroxbuster::url enter: format(, None)
TRC      1.741 feroxbuster::utils enter: parse_url_with_raw_path(https://www.rei.com/)
TRC      1.741 feroxbuster::url exit: format_url -> https://www.rei.com/
TRC      1.741 feroxbuster::utils enter: make_request(Configuration::Client, https://www.rei.com/, Default, UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x14d808200, ta
il_position: 1 }, semaphore: Semaphore(0), rx_waker: AtomicWaker, tx_count: 3, rx_fields: "..." } } })
TRC      2.762 feroxbuster::utils exit: make_request -> Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("www.rei.com")), po
rt: None, path: "/", query: None, fragment: None }, status: 200, headers: {"content-type": "text/html;charset=UTF-8", "content-language": "zh-CN", "request-uuid": "ade40062d304", "x-frame
-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "x-permitted-cross-domain-policies": "none", "x-akamai-transformed": "9 - 0 pmb=mTOE,3", "cache-control": "max-age=300", "exp
ires": "Tue, 03 Oct 2023 11:56:24 GMT", "date": "Tue, 03 Oct 2023 11:51:24 GMT", "set-cookie": "akamai_session=23.45.12.4.30032821696333884126; path=/; domain=.rei.com", "set-cookie": "co
okie_policy=CA; expires=Mon, 01-Jan-2024 11:51:24 GMT; path=/", "set-cookie": "EdgeLocation=34.1977,-118.6018; expires=Wed, 04-Oct-2023 11:51:24 GMT; path=/", "set-cookie": "akaas_LaborDa
ySaleOfferCardTest=1696335684~rv=25~id=6156f07d5c67bbb8059e7dc356ee3207~rn=; path=/; Expires=Tue, 03 Oct 2023 12:21:24 GMT; Secure; SameSite=None", "set-cookie": "_abck=9D9D8481AC9DAA69F7
92E5D50AE434D7~-1~YAAQBAwtF+KO19mKAQAA3upg9QpfaZTVbBrRKzgFjktJZDm6qjol2FIxw0Hso5YqmBLiUkuZAU91YbWQKaNQIwN/PwqrE8i/gMyj8H2ITkpOSTjspxDhp+eNi4e0ficDrIyPGo6A9CVfDsd+mfrApn6ojmeqnfJgUcyK7j15K
3qiUVCI8smfNC8FfzwuUCIZmqRa4UPoloibAYbVO3Kbw/e0pw3VZepAFkYeVEdnRi/CeDFDLRImSV5N3aWLrhHkWXUXrBcaWAHfKhTUN7p5TS1MNNoFecM0IyVd6Q+4+8iSHXNtzFdjk5BXiAHbPABm7iUCkmuVfwjaUO+pbe+pwboislrPyfdlU2da
lUSALr995ILTNV4=~-1~-1~1696337458; Domain=.rei.com; Path=/; Expires=Wed, 02 Oct 2024 11:51:24 GMT; Max-Age=31536000; SameSite=None; Secure", "set-cookie": "bm_sz=C18ECACACC5EB9F88C048246E
7968571~YAAQBAwtF+OO19mKAQAA3upg9RWEDhMTtyAF8M8H98f41L6TKca6Oh4eRXxY24ATYqVfXmNWw/KyQuZC4w/JDL96U3i4Lcoi2Tge7b2H8hfBo2TIISE1PM3Ek2vPc7rO8Nx64VrA3DrtoQ6TmYl7Zw3jy6x/w+AaRHk+TSoEUohDIwRbkJC
PVTAmxp5ZOiMHL2Ia0cxDMv60NH/TyJXLPzb9/HLdzghA9DkrDlymkwAGUOWAXjDfi9crDDvjrkPuAYRocdhwhMVwDJ1CaSJIf5H72ragie6C4lehg34lXjE=~4473137~3159864; Domain=.rei.com; Path=/; Expires=Tue, 03 Oct 202
3 15:51:24 GMT; Max-Age=14400; SameSite=None; Secure", "strict-transport-security": "max-age=31536000 ; includeSubDomains", "content-security-policy": "default-src 'self' *.rei.com; frame
-ancestors 'self' https://www.chasepaymentechhostedpay.com *.rei.com *.rei-cloud.com; upgrade-insecure-requests; script-src 'self' 'report-sample' 'unsafe-inline' 'unsafe-eval' blob: http
s://tr.snapchat.com https://*.tealiumiq.com https://*.impactradius-event.com *.klarnacdn.net *.klarnaservices.com *.klarna.com *.klarna.net https://edge.curalate.com https://sc-static.net
https://*.snaplinker.com https://*.xg4ken.com googleads.g.doubleclick.net www.googletagmanager.com www.paypalobjects.com s2.go-mpulse.net js-agent.newrelic.com cdnjs.cloudflare.com cdn.a
vmws.com ssl.avmws.com connect.facebook.net odeaiqfw.micpn.com t.channeladvisor.com triggeredmail.appspot.com www.googleadservices.com www.paypal.com seal.websecurity.norton.com https://*
.googlesyndication.com https://*.channeladvisor.com https://api.smooch.io https://www.google.com https://cdn.speedcurve.com https://www.gstatic.com https://*.demdex.net https://*.googlead
services.com https://*.omtrdc.net https://www.cdn-net.com https://six.cdn-net.com https://prod.accdab.net static.ads-twitter.com https://js-agent.newrelic.com https://view.seekxr.com http
s://www.googletagmanager.com https://www.googletagservices.com https://googletagmanager.com https://www.google-analytics.com https://ssl.google-analytics.com https://google-analytics.com
https://*.ngpvan.com https://cdnjs.cloudflare.com https://secure.gravatar.com https://www.paypalobjects.com https://*.nr-data.net https://www.paypal.com https://translate.googleapis.com h
ttps://translate.google.com https://*.verygoodvault.com https://cdn.gladly.com https://www.youtube.com https://tags.tiqcdn.com https://d3rse9xjbp8270.cloudfront.net https://static.ads-twi
tter.com https://*.ep-mimecast.ads-twitter.com https://analytics.twitter.com https://bat.bing.com https://r.bing.com https://code.jquery.com tags.tiqcdn.com https://*.go-mpulse.net https:
//*.avantlink.com https://cdn.forms.office.net https://activitymap.adobe.com *.yottaa.com *.rei.com *.bazaarvoice.com https://www.myregistry.com *.qualtrics.com https://*.googleapis.com h
ttps://*.twitter.com https://cdn.jsdelivr.net https://cdn.syndication.twimg.com https://player.ooyala.com https://*.theknot.com https://*.richrelevance.com https://mpsnare.iesnare.com pna
pi.invoca.net solutions.invocacdn.com *.avmws.com https://*.microsofttranslator.com https://*.akamaihd.net https://*.speedcurve.com https://cdn.cookielaw.org https://geolocation.onetrust.
com https://js.narvar.com https://player.vimeo.com/ https://api.bluecore.com https://www.chasepaymentechhostedpay-var.com https://www.chasepaymentechhostedpay.com https://analytics.tiktok
.com https://bat.bing.com https://s.pinimg.com; style-src 'self' 'report-sample' 'unsafe-inline' *.bing.com fonts.googleapis.com https://cdn.gladly.com https://*.omtrdc.net translate.goog
leapis.com https://d3rse9xjbp8270.cloudfront.net cdnjs.cloudflare.com secure.gravatar.com www.googletagmanager.com https://*.bazaarvoice.com https://*.snaplinker.com code.jquery.com https
://vjs.zencdn.net https://cdn.jsdelivr.net https://platform.twitter.com https://ton.twimg.com https://player.ooyala.com https://cdn.richrelevance.com https://*.rei.com http://*.rei.com *.
klarnacdn.net https://www.gstatic.com; object-src 'self' https://*.rei.com https://*.online-metrix.net https://www.cdn-net.com https://mpsnare.iesnare.com; frame-src https://secure.everya
ction.com *.airstream.com *.klarna.com *.klarna.net *.klarnacdn.net https://player.simplecast.com https://tr.snapchat.com https://tr6.snapchat.com www.paypalobjects.com tpc.googlesyndicat
ion.com www.paypal.com https://*.doubleclick.net https://www.google.com https://connect.facebook.net *.rei.com *.youtube.com https://www.cdn-net.com https://prod.accdab.net https://*.akam
aihd.net https://view.seekxr.com https://*.facebook.com https://*.demdex.net http://*.demdex.net www.googletagmanager.com www.youtube-nocookie.com *.paypal.com reiadventures.force.com htt
ps://*.verygoodvault.com https://*.avantlink.com https://customervoice.microsoft.com https://*.snaplinker.com https://activitymap.adobe.com *.qualtrics.com https://www.youtube.com https:/
/*.bazaarvoice.com https://*.theknot.com https://*.twitter.com https://r.curalate.com sdx.microsoft.com https://www.chasepaymentechhostedpay.com https://player.ooyala.com https://www.myre
gistry.com https://forms.office.com *.narvar.com https://player.vimeo.com/ https://ct.pinterest.com; child-src blob: www.googletagmanager.com www.youtube.com www.paypalobjects.com *.paypa
l.com; img-src data: blob: *; font-src data: *; connect-src blob: about: https://rei.pxf.io *.klarna.net *.klarnaevt.com *.klarnaservices.com *.klarna.com https://edge.curalate.com https:
//tr.snapchat.com p11.techlab-cdn.com c.go-mpulse.net www.paypal.com *.qualtrics.com www.facebook.com https://api.bluecore.com https://gladly-production.sinter-collect.com https://*.akama
ihd.net https://*.akstat.io pnapi.invoca.net https://*.bing.com wss://*.bing.com https://*.online-metrix.net https://prod.accdab.net https://*.matterport.com https://view.seekxr.com t.co
www.googletagmanager.com www.googletagservices.com www.google-analytics.com https://*.googlesyndication.com fonts.gstatic.com https://translate.yandex.net https://*.ngpvan.com https://*.e
veryaction.com cdnjs.cloudflare.com https://labs.observepoint.com *.gravatar.com https://*.seekxr.com https://*.smooch.io wss://*.smooch.io https://*.gladly.com translate.googleapis.com t
ranslate.google.com https://*.verygoodvault.com https://*.nr-data.net https://*.doubleclick.net https://*.demdex.net https://*.omtrdc.net *.bing.com code.jquery.com *.go-mpulse.net *.akst
at.io https://*.snaplinker.com https://i.embed.ly https://bluecore.app *.paypal.com *.bazaarvoice.com *.google.com *.rei.com https://*.googleapis.com https://player.ooyala.com https://hm.
baidu.com https://*.richrelevance.com https://cdn.cookielaw.org https://geolocation.onetrust.com https://rum.browser-intake-datadoghq.com https://session-replay.browser-intake-datadoghq.c
om https://api.bluecore.app https://siteassets.bluecore.com https://onsitestats.bluecore.com https://www.chasepaymentechhostedpay-var.com https://bat.bing.com https://analytics.tiktok.com
https://ct.pinterest.com; base-uri 'self' https://*.nr-data.net https://recs.richrelevance.com about: *.rei.com; form-action https://tr.snapchat.com https://www.facebook.com https://*.fa
cebook.net https://twitter.com www.cdn-net.com https://home.creditcardila.com https://rei.co1.qualtrics.com https://*.qualtrics.com https://vote.escvote.com http://rei.jobs https://rei.jo
bs https://*.rei.com https://rei.com https://checkout.stripe.com https://*.bazaarvoice.com https://*.twitter.com https://www.reimastercard.com https://www.myregistry.com https://*.google.
com https://forms.office.com https://www.reifund.org; media-src 'self' data: https://cdn.gladly.com https://v2.zopim.com https://static.zdassets.com https://ssl.gstatic.com *.rei.com; wor
ker-src blob: https://*.rei.com; report-uri https://csp-report.browser-intake-datadoghq.com/api/v2/logs?dd-api-key=pubbcf1148cbb6c399749f66f5eed93b535&dd-evp-origin=content-security-polic
y&ddsource=csp-report&ddtags=env%3Aprod", "link": "<https://satchel.rei.com/>; rel=preconnect"} }
TRC      2.770 feroxbuster::heuristics exit: connectivity_test -> ["https://www.rei.com/"]

epi052 · 2023-10-03T11:59:13Z

yea, i suspect burp is transparently handling the http/2 part of the conversation on our behalf

devx00 · 2023-10-03T12:00:19Z

yea, i suspect burp is transparently handling the http/2 part of the conversation on our behalf

Ya that was my initial thought. Any idea why performance decreases when you try to use http/2 natively? I have a hard time believing that Burp's implementation of http in Java could outperform rust.

epi052 · 2023-10-03T12:07:18Z

i don't off hand, i use the reqwest library to handle the actual web stuff. i changed otu the backend to (probably) confirm it's http/2 at fault. There may be a way to get http/2 support without a perf hit. I'll need to dig up that old convo (may be on the feroxfuzz repo as well) and look at reqwest's http/2 capabilities.

epi052 · 2023-10-03T12:10:37Z

actually, nevermind on performance. I was comparing debug build to debug build, which is typically ok as a sniff test. I believe since i was using native-tls in debug, those are already compiled C bindings and reasonably fast. The rustls would have been compiled without optimizations, making it a lot slower

epi052 · 2023-10-03T12:10:51Z

built with --release and it performs as expected

epi052 · 2023-10-06T10:26:43Z

ok, here's the thread i was thinking of.

#892

rustls probably isn't a good fit for ferox. I'll see what other options are availalbe

epi052 · 2023-10-06T10:37:43Z

i made the following change and everything seems to work. rei.com scans fine, even withotu the firefox header

diff --git a/Cargo.toml b/Cargo.toml
index f1e7a09..e543390 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -35,7 +35,7 @@ tokio = { version = "1.29", features = ["full"] }
 tokio-util = { version = "0.7", features = ["codec"] }
 log = "0.4"
 env_logger = "0.10"
-reqwest = { version = "0.11", features = ["socks", "native-tls"] }
+reqwest = { version = "0.11", features = ["socks", "native-tls-alpn"] }
 # uses feature unification to add 'serde' to reqwest::Url
 url = { version = "2.4", features = ["serde"] }
 serde_regex = "1.1"

epi052 · 2023-10-06T10:52:03Z

also confirmed it doesn't break mTLS

epi052 · 2023-11-09T02:24:19Z

@all-contributors add @RavySena for idea

allcontributors · 2023-11-09T02:24:28Z

@epi052

I've put up a pull request to add @RavySena! 🎉

22Goose · 2024-04-16T15:35:25Z

hi, i'm new here. sorry for bothering.

i just found the same issue when i try to bruteforce the hackthebox machine "nibbles",
i use the following command
feroxbuster -u http://10.10.10.75/nibbleblog -w directory-list-2.3-medium.txt
and it will give the 'operation timed out‘ issue.

i notice the HTTP/2 problem you guys just mentioned, but this page uses HTTP/1.1
btw, i just add the --burp option and it will go well

i dont know where goes wrong thus come here searching for help
if u wanna more details just tell me

RavySena added the bug Something isn't working label Jul 19, 2023

stale bot added the stale label Sep 16, 2023

stale bot removed the stale label Oct 3, 2023

epi052 changed the title ~~Freeze at start of scan.~~ [FEATURE REQUEST] http/2 support Oct 5, 2023

epi052 added enhancement New feature or request and removed bug Something isn't working labels Oct 5, 2023

epi052 mentioned this issue Nov 8, 2023

add http/2 support #1020

Merged

17 tasks

epi052 closed this as completed in #1020 Nov 8, 2023

allcontributors bot mentioned this issue Nov 9, 2023

docs: add RavySena as a contributor for ideas #1025

Merged

epi052 mentioned this issue Jun 8, 2024

any one can help me to slove this error ? #1161

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE REQUEST] http/2 support #947

[FEATURE REQUEST] http/2 support #947

RavySena commented Jul 19, 2023

epi052 commented Jul 29, 2023

stale bot commented Sep 16, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 6, 2023

epi052 commented Oct 6, 2023

epi052 commented Oct 6, 2023

epi052 commented Nov 9, 2023

allcontributors bot commented Nov 9, 2023

22Goose commented Apr 16, 2024

[FEATURE REQUEST] http/2 support #947

[FEATURE REQUEST] http/2 support #947

Comments

RavySena commented Jul 19, 2023

epi052 commented Jul 29, 2023

stale bot commented Sep 16, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

devx00 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 3, 2023

epi052 commented Oct 6, 2023

epi052 commented Oct 6, 2023

epi052 commented Oct 6, 2023

epi052 commented Nov 9, 2023

allcontributors bot commented Nov 9, 2023

22Goose commented Apr 16, 2024