Cannot handle dirbuster wordlist

[luke-goddard]

https://raw.githubusercontent.com/digination/dirbuster-ng/master/wordlists/big.txt
Tried running with the above URL, but rustbuster failed.

╰─[:(] % RUST_BACKTRACE=full rustbuster dir -u http://10.10.10.160:80/ -w /usr/share/hacking/wordlists/url/big.txt
~ rustbuster v3.0.3 ~ by phra & ps1dr3x ~

         _     _                 _         _           _        _                 _         _           _           _
        /\ \  /\_\              / /\      /\ \        / /\     /\_\              / /\      /\ \        /\ \        /\ \
       /  \ \/ / /         _   / /  \     \_\ \      / /  \   / / /         _   / /  \     \_\ \      /  \ \      /  \ \
      / /\ \ \ \ \__      /\_\/ / /\ \__  /\__ \    / / /\ \  \ \ \__      /\_\/ / /\ \__  /\__ \    / /\ \ \    / /\ \ \
     / / /\ \_\ \___\    / / / / /\ \___\/ /_ \ \  / / /\ \ \  \ \___\    / / / / /\ \___\/ /_ \ \  / / /\ \_\  / / /\ \_\
    / / /_/ / /\__  /   / / /\ \ \ \/___/ / /\ \ \/ / /\ \_\ \  \__  /   / / /\ \ \ \/___/ / /\ \ \/ /_/_ \/_/ / / /_/ / /
   / / /__\/ / / / /   / / /  \ \ \    / / /  \/_/ / /\ \ \___\ / / /   / / /  \ \ \    / / /  \/_/ /____/\   / / /__\/ /
  / / /_____/ / / /   / / _    \ \ \  / / /     / / /  \ \ \__// / /   / / _    \ \ \  / / /     / /\____\/  / / /_____/
 / / /\ \ \  / / /___/ / /_/\__/ / / / / /     / / /____\_\ \ / / /___/ / /_/\__/ / / / / /     / / /______ / / /\ \ \
/ / /  \ \ \/ / /____\/ /\ \/___/ / /_/ /     / / /__________/ / /____\/ /\ \/___/ / /_/ /     / / /_______/ / /  \ \ \
\/_/    \_\/\/_________/  \_____\/  \_\/      \/_____________\/_________/  \_____\/  \_\/      \/__________\/_/    \_\/


[?] Started at  : 2019-12-27 15:22:41

thread 'main' panicked at 'Something went wrong reading the wordlist file: Custom { kind: InvalidData, error: "stream did not contain valid UTF-8" }', src/libcore/result.rs:999:5
stack backtrace:
   0:     0x55b2069ad27b - backtrace::backtrace::libunwind::trace::h088077d8b0559777
                               at /build/rust/src/rustc-1.37.0-src/vendor/backtrace/src/backtrace/libunwind.rs:88
   1:     0x55b2069ad27b - backtrace::backtrace::trace_unsynchronized::h7198c0bb84b6a982
                               at /build/rust/src/rustc-1.37.0-src/vendor/backtrace/src/backtrace/mod.rs:66
   2:     0x55b2069ad27b - std::sys_common::backtrace::_print::hbf9d412504e2cf5e
                               at src/libstd/sys_common/backtrace.rs:47
   3:     0x55b2069ad27b - std::sys_common::backtrace::print::h1a2af1aaff9961fb
                               at src/libstd/sys_common/backtrace.rs:36
   4:     0x55b2069ad27b - std::panicking::default_hook::{{closure}}::hb50c1be487d5e83b
                               at src/libstd/panicking.rs:200
   5:     0x55b2069aca85 - std::panicking::default_hook::h22162fa7b1026bf3
                               at src/libstd/panicking.rs:214
   6:     0x55b2069aca85 - std::panicking::rust_panic_with_hook::h4e663330759b90e1
                               at src/libstd/panicking.rs:477
   7:     0x55b2069ac5d2 - std::panicking::continue_panic_fmt::hea2c5ffb8af99faf
                               at src/libstd/panicking.rs:384
   8:     0x55b2069ba136 - rust_begin_unwind
                               at src/libstd/panicking.rs:311
   9:     0x55b2069c3d8c - core::panicking::panic_fmt::ha8e419005b06d9fa
                               at src/libcore/panicking.rs:85
  10:     0x55b2068e7b92 - core::result::unwrap_failed::h4e5790d905474e2e
  11:     0x55b206a23c1c - core::result::Result<T,E>::expect::h0f191e719aaeeb36
  12:     0x55b206a474f4 - rustbuster::main::h8bf590bdd0801656
  13:     0x55b206a1b6da - std::rt::lang_start::{{closure}}::h5af60c1d7d96386c
  14:     0x55b206a42849 - main
  15:     0x7fc3a24ec153 - __libc_start_main
  16:     0x55b2068a90ce - _start
  17:                0x0 - <unknown>

Not sure what line in the word list it could not handle from the error message.
Maybe adding some more debugging info when a bad line is encountered, and just ignore the line to carry on with the rest of the wordlist
:)

[kaibyao]

I actually looked into the source code and found that the error (thread 'main' panicked at 'Something went wrong reading the wordlist file: Custom { kind: InvalidData, error: "stream did not contain valid UTF-8" }', src/dirbuster/utils.rs:14:9) is emitted by std::fs::read_to_string(), and there's no way to extract which line/text was not valid.

In any case, saving the text file in VSCode removed any invalid UTF-8 characters and running the rustbuster command w/ that file worked afterward shrug.

[phra]

Duplicate of #26