http: Report unmatched requests or responses #6794
Conversation
packetbeat/protos/registry.go
Outdated
| @@ -50,6 +50,13 @@ type UDPPlugin interface { | |||
| ParseUDP(pkt *Packet) | |||
| } | |||
|
|
|||
| type ExpirationAwareTCPPlugin interface { | |||
There was a problem hiding this comment.
exported type ExpirationAwareTCPPlugin should have comment or be unexported
adriansr
force-pushed
the
feature/http_timeout
branch
from
ff77555 to
add0f1f
Compare
|
Partially addressing #257, as it only works with HTTP. |
adriansr
force-pushed
the
feature/http_timeout
branch
from
add0f1f to
63cd215
Compare
packetbeat/protos/http/http.go
Outdated
| func (http *httpPlugin) correlate(conn *httpConnectionData) { | ||
| // drop responses with missing requests | ||
| if conn.requests.empty() { | ||
| func (http *httpPlugin) correlate(conn *httpConnectionData, flush bool) { |
There was a problem hiding this comment.
Will the correlate always report all active messages? Depending on where packetbeat is running (passive sniffing), a request might be received after the corresponding resource. This make correlation quite tricky in general.
Is there a chance of flushRequests (at the end of the function) is prematurely flushing events? In which situation will the flush flag be set? Always on shutdown?
How about extracting the local functions into methods on the httpPlugin. The correlate can re-use them and we can remove the flush flag from correlate. Whenever we would set the flush flag, we would call flushAll, which calls flushResponses and flushRequests.
packetbeat/protos/http/http.go
Outdated
| } | ||
| } | ||
|
|
||
| // drop responses with missing requests |
|
LGTM. Let's pull change the local closures to httpPlugin methods and get rid of the |
adriansr
force-pushed
the
feature/http_timeout
branch
3 times, most recently
from
a20195b to
f49c939
Compare
|
Thanks @urso, I've addressed your comment. Also, I've included a fix for a race condition when expired streams are notified. libbeat's common.Cache uses a separate goroutine for the janitor, and removal notifications from the cache are reported from this goroutine. I've refactored it so that plugins are notified about this event from the same goroutine that handles the packet processing. Give me your opinion on it, it's all on the first commit, |
packetbeat/protos/http/http.go
Outdated
| @@ -415,14 +415,31 @@ func (http *httpPlugin) handleHTTP( | |||
| } | |||
| } | |||
|
|
|||
| func (http *httpPlugin) flushResponses(conn *httpConnectionData) { | |||
| for !conn.responses.empty() { | |||
| debugf("Response from unknown transaction. Reporting error.") | |||
There was a problem hiding this comment.
If there's any identifying information available about the flow it would likely be beneficial to include a few details in the log. The same in flushRequests.
Added a new interface: ExpirationAwareTCPPlugin. Protocol parsers implementing this interface will be notified when a TCP stream expires and have the oportunity to run some cleanup tasks and deliver pending events. This is an opt-in feature. Plugins conforming only to TCPPlugin will keep the existing behavior.
A couple of pcap files used in packetbeat's system tests include extra HTTP streams with unmatched HTTP responses that are unnecessary for the actual tests. This will result in a test failure when packetbeat starts reporting unmatched HTTP transactions.
adriansr
force-pushed
the
feature/http_timeout
branch
from
f49c939 to
da30fc5
Compare
Until now, unmatched requests or responses are silently discarded by the http plugin. This patch makes two changes to this behavior: - When an HTTP response is received without a matching request, an event with Error status is generated. - When an HTTP stream times out, any pending request or response that has received complete headers is also reported with Error status.
Added tests for the HTTP unmatched requests and responses. - test_unmatched_response: Checks that a response received out of order inside a stream is reported as an Error. - test_unmatched_request: Tests that a single stream containing only one request is reported after the stream times out. This tests can't be run with packetbeat's -t flag as it needs to wait 10s for the stream to expire. This timeout is currently not configurable.
adriansr
force-pushed
the
feature/http_timeout
branch
from
da30fc5 to
53d7c49
Compare
Until now, unmatched requests or responses are silently discarded by the http plugin.
This patch makes two changes to this behavior:
When an HTTP response is received without a matching request, an event with Error status is generated.
When an HTTP stream times out, any pending request or response that has received complete headers is also reported with Error status.