[FEATURE REQUEST] improve json logs for post processing

[Spidle]

Is your feature request related to a problem? Please describe.
When looking at my feroxbuster json logs, I'm unable to relate the statistics to a particular host, and additionally, I dont know when the feroxbuster was run. I'm trying to use statistical analysis on a massive amount of ferox JSON files.

Describe the solution you'd like
I would like each JSON entry to have a timestamp. Additionally, statistics entries should have the base URL that was passed in when then scan was initiated included in the data.

PS: Can the json statistics be added on exit if ctrl-c is pressed?

Describe alternatives you've considered
Post processing. It works (sorta).

[epi052]

both of these requests seem pretty reasonable, and easy to implement (pretty sure, haven't looked yet). pinning for now

[epi052]

howdy! started poking at this. when you say json logs, i assume you mean those produced by -o ... --json and do NOT mean those inside --debug-log .... You may be interested in having the configuration entry as well for analysis.

I'm looking into what it'll take code-wise to make the requested changes. lmk if the config entry is useful

{"type":"configuration","wordlist":"common.txt","config":"ferox-config.toml","proxy":"","replay_proxy":"","server_certs":[],"client_cert":"","client_key":"","target_url":"http://localhost:8000","status_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"replay_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"filter_status":[],"threads":50,"timeout":7,"verbosity":3,"silent":false,"quiet":false,"auto_bail":false,"auto_tune":false,"json":true,"output":"test-log2.log","debug_log":"test-debug.log","user_agent":"feroxbuster/2.10.2","random_agent":false,"redirects":false,"insecure":false,"extensions":[],"methods":["GET"],"data":[],"headers":{},"queries":[],"no_recursion":false,"extract_links":true,"add_slash":false,"stdin":false,"depth":4,"scan_limit":0,"parallel":0,"rate_limit":0,"filter_size":[],"filter_line_count":[],"filter_word_count":[],"filter_regex":[],"dont_filter":false,"resumed":false,"resume_from":"","save_state":false,"time_limit":"","filter_similar":[],"url_denylist":[],"regex_denylist":[],"collect_extensions":false,"dont_collect":["tif","tiff","ico","cur","bmp","webp","svg","png","jpg","jpeg","jfif","gif","avif","apng","pjpeg","pjp","mov","wav","mpg","mpeg","mp3","mp4","m4a","m4p","m4v","ogg","webm","ogv","oga","flac","aac","3gp","css","zip","xls","xml","gz","tgz"],"collect_backups":false,"backup_extensions":["~",".bak",".bak2",".old",".1"],"collect_words":false,"force_recursion":false}

[epi052]

PS: Can the json statistics be added on exit if ctrl-c is pressed?

this actually happens already, if you have save-state enabled, the statistics field should be the last entry in the .state file (the filename itself denotes the original target url -> ferox-http_localhost:8000-1711710683.state as well as the config entry within the state file).

However, with large distributed scans (which is what it sounds like you're doing), I THINK you should get a state file for any single target that was killed, but I haven't checked that for sure.

[epi052]

here's what things are looking like at the moment

response w/ timestamp

{"type":"response","url":"http://localhost:8000/stuff/nthings","original_url":"http:localhost:8000","path":"/stuff/nthings","wildcard":false,"status":401,"method":"GET","content_length":120,"line_count":1,"word_count":5,"headers":{"content-type":"text/plain; charset=utf-8","content-length":"120","x-content-type-options":"nosniff","server":"Rocket","x-frame-options":"SAMEORIGIN","date":"Sat, 30 Mar 2024 19:32:05 GMT","permissions-policy":"interest-cohort=()"},"extension":"","timestamp":1711827125.7913597}

statistics w/ targets

single scan

{"type":"statistics","timeouts":0,"requests":28389,"expected_per_scan":4724,"total_expected":28344,"errors"
:0,"successes":5811,"redirects":5257,"client_errors":11962,"server_errors":4593,"total_scans":6,"initial_ta
rgets":0,"links_extracted":0,"extensions_collected":0,"status_200s":737,"status_301s":785,"status_302s":784
,"status_401s":775,"status_403s":4584,"status_429s":4426,"status_500s":2305,"status_503s":769,"status_504s"
:0,"status_508s":0,"wildcards_filtered":344,"responses_filtered":344,"resources_discovered":15990,"url_form
at_errors":0,"redirection_errors":0,"connection_errors":0,"request_errors":0,"directory_scan_times":[10.584
397651,10.981936694,12.019307243,11.967351518,11.079414173,9.05110747],"total_runtime":[15.644809645],"targ
ets":["http:localhost:8000"]}                                                                             

two targets, not parallel

{"type":"statistics","timeouts":0,"requests":36,"expected_per_scan":2,"total_expected":17,"errors":0,"successes":11,"redirects":4,"client_errors":19,"server_errors":2,"total_scans":2,"initial_targets":0,"links_extracted":13,"extensions_collected":0,"status_200s":8,"status_301s":0,"status_302s":1,"status_401s":0,"status_403s":1,"status_429s":2,"status_500s":2,"status_503s":0,"status_504s":0,"status_508s":0,"wildcards_filtered":9,"responses_filtered":9,"resources_discovered":8,"url_format_errors":0,"redirection_errors":0,"connection_errors":0,"request_errors":0,"directory_scan_times":[0.028152285,2.2592075019999998],"total_runtime":[3.424981309],"targets":["http:localhost:8000","https://bitdiscovery.com"]}

two targets, parallel, creates a new directory, with a log file per target. each log file has the same array with a single entry as the single target shown above.

❯ lt stuff2_json-1711827125.logs
total 88
drwxrwxr-x 17 epi epi 65536 Mar 30 15:32 ..
drwxrwxr-x  2 epi epi  4096 Mar 30 15:32 .
-rw-rw-r--  1 epi epi  1659 Mar 30 15:32 ferox-http:localhost:8000-1711827125.log
-rw-rw-r--  1 epi epi 10894 Mar 30 15:32 ferox-https_bitdiscovery_com-1711827125.log

[epi052]

added configuration entry to -o outputfile (with and without --json)

{"type":"configuration","wordlist":"common.txt","config":"ferox-config.toml","proxy":"","replay_proxy":"","server_certs":[],"client_cert":"","client_key":"","target_url":"http:localhost:8000","status_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"replay_codes":[100,101,102,200,201,202,203,204,205,206,207,208,226,300,301,302,303,304,305,307,308,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,421,422,423,424,426,428,429,431,451,500,501,502,503,504,505,506,507,508,510,511,103,425],"filter_status":[],"threads":50,"timeout":7,"verbosity":0,"silent":false,"quiet":false,"auto_bail":false,"auto_tune":false,"json":true,"output":"stuff.json","debug_log":"","user_agent":"feroxbuster/2.10.2","random_agent":false,"redirects":false,"insecure":false,"extensions":[],"methods":["GET"],"data":[],"headers":{},"queries":[],"no_recursion":false,"extract_links":true,"add_slash":false,"stdin":false,"depth":4,"scan_limit":0,"parallel":0,"rate_limit":0,"filter_size":[],"filter_line_count":[],"filter_word_count":[],"filter_regex":[],"dont_filter":false,"resumed":false,"resume_from":"","save_state":false,"time_limit":"","filter_similar":[],"url_denylist":[],"regex_denylist":[],"collect_extensions":false,"dont_collect":["tif","tiff","ico","cur","bmp","webp","svg","png","jpg","jpeg","jfif","gif","avif","apng","pjpeg","pjp","mov","wav","mpg","mpeg","mp3","mp4","m4a","m4p","m4v","ogg","webm","ogv","oga","flac","aac","3gp","css","zip","xls","xml","gz","tgz"],"collect_backups":false,"backup_extensions":["~",".bak",".bak2",".old",".1"],"collect_words":false,"force_recursion":false}

[Spidle]

Epi, This is amazing, sorry I was out this weekend and didn't get back to you to provide my gratitude and 2-cents.

• Response with timestamp is great! Seems straight forward enough.
• The statistics and configuration entry is exactly what I was looking for. I'll be able to go back and see in the Json logs what wordlists I've run before with statistics about the scan. When doing large scopes, this will definitely help with knowing what was done and lead to additional tactics.

So much gratitude.

[epi052]

@all-contributors add @Spidle for ideas

[allcontributors]

@epi052

I've put up a pull request to add @Spidle! 🎉