-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
http: allow raw header capture #347
http: allow raw header capture #347
Conversation
82aacc9
to
7b24667
Compare
The golang textproto library does a few things when parsing the HTTP headers: * consume some whitespace characters (e.g. \r\n) * canonicalizes the header keys (e.g. "content-type" => "Content-Type") * moves the headers into a map This all makes sense when parsing HTTP, but for a scanner some may want to have the exact headers, to match on order, non-canonical keys, etc. This adds that option, if '--raw-headers' is specified during an HTTP scan. This is accomplished by implementing a tee reader on the pconn interface, that tees before the bufio reader is put in place. The tee copy can be disabled once the headers have been read, so as to not waste memory while consuming the HTTP body. While denoted as "raw headers", this will also capture the raw status line as well.
7b24667
to
4f5a6bd
Compare
lgtm |
Hi @codyprime, thanks for this, it will be super helpful. Is there a reason why this has been merged to a feature branch and not to master? Could it be ported to master? |
The golang textproto library does a few things when parsing the HTTP headers: * consume some whitespace characters (e.g. \r\n) * canonicalizes the header keys (e.g. "content-type" => "Content-Type") * moves the headers into a map This all makes sense when parsing HTTP, but for a scanner some may want to have the exact headers, to match on order, non-canonical keys, etc. This adds that option, if '--raw-headers' is specified during an HTTP scan. This is accomplished by implementing a tee reader on the pconn interface, that tees before the bufio reader is put in place. The tee copy can be disabled once the headers have been read, so as to not waste memory while consuming the HTTP body. While denoted as "raw headers", this will also capture the raw status line as well. (cherry picked from commit 83e55e0) Signed-off-by: Jeff Cody <jcody@censys.io>
Hi! The reason for the feature branch was because A) I was using the feature branch :) and B) That enabled some volume testing. I've cherry-picked the commit and created a PR on the |
The golang textproto library does a few things when parsing the HTTP headers: * consume some whitespace characters (e.g. \r\n) * canonicalizes the header keys (e.g. "content-type" => "Content-Type") * moves the headers into a map This all makes sense when parsing HTTP, but for a scanner some may want to have the exact headers, to match on order, non-canonical keys, etc. This adds that option, if '--raw-headers' is specified during an HTTP scan. This is accomplished by implementing a tee reader on the pconn interface, that tees before the bufio reader is put in place. The tee copy can be disabled once the headers have been read, so as to not waste memory while consuming the HTTP body. While denoted as "raw headers", this will also capture the raw status line as well. (cherry picked from commit 83e55e0) Signed-off-by: Jeff Cody <jcody@censys.io>
The golang textproto library does a few things when parsing the HTTP
headers:
This all makes sense when parsing HTTP, but for a scanner some may want
to have the exact headers, to match on order, non-canonical keys, etc.
This adds that option, if '--raw-headers' is specified during an HTTP
scan. This is accomplished by implementing a tee reader on the pconn
interface, that tees before the bufio reader is put in place. The
tee copy can be disabled once the headers have been read, so as to not
waste memory while consuming the HTTP body.
How to Test
stdout output should look like: