Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CEF: Panic handling datagram. Recovering, but please report this. #30010

Closed
garethhumphriesgkc opened this issue Jan 26, 2022 · 5 comments · Fixed by #30038
Closed

CEF: Panic handling datagram. Recovering, but please report this. #30010

garethhumphriesgkc opened this issue Jan 26, 2022 · 5 comments · Fixed by #30038
Assignees
@andrewkroh
Labels
bug Filebeat Filebeat

Comments

@garethhumphriesgkc
Copy link
Contributor

garethhumphriesgkc commented Jan 26, 2022 •
edited
Loading

Reporting as requested. Happens reliably within a few minutes of the starting the container up. Input is CEF log data from Azure defender SIEM integrator - payload not logged, even with debug, so can't be more specific than that.

Filebeat 7.16.3, CEF module
Docker 20.10.7

Seems to only manifest when a valid destination is connected. It claims to have recovered, but it hasn't - no further data gets processed until I recreate the container.

Initial testing indicates this is new since 7.13.2 - haven't yet seen a panic when running with identical configuration on that version.

2022-01-26T01:59:29.249Z	ERROR	[UDP]	logp/logger.go:218	Panic handling datagram. Recovering, but please report this.{panic 25 0  runtime error: slice bounds out of range [:-116]} {stack 15 0 github.com/elastic/beats/v7/libbeat/logp.(*Logger).Recover
	/go/src/github.com/elastic/beats/libbeat/logp/logger.go:218
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:1038
runtime.goPanicSliceAlen
	/usr/local/go/src/runtime/panic.go:100
github.com/elastic/beats/v7/x-pack/filebeat/processors/decode_cef/cef.replaceEscapes
	/go/src/github.com/elastic/beats/x-pack/filebeat/processors/decode_cef/cef/cef.go:173
github.com/elastic/beats/v7/x-pack/filebeat/processors/decode_cef/cef.(*Event).unpack
	cef.rl:76
github.com/elastic/beats/v7/x-pack/filebeat/processors/decode_cef/cef.(*Event).Unpack
	/go/src/github.com/elastic/beats/x-pack/filebeat/processors/decode_cef/cef/cef.go:125
github.com/elastic/beats/v7/x-pack/filebeat/processors/decode_cef.(*processor).Run
	/go/src/github.com/elastic/beats/x-pack/filebeat/processors/decode_cef/decode_cef.go:89
github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
	/go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:121
github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
	/go/src/github.com/elastic/beats/libbeat/publisher/processing/processors.go:121
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:101
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client.go:80
github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
	/go/src/github.com/elastic/beats/filebeat/beater/channels.go:136
github.com/elastic/beats/v7/filebeat/channel.(*outlet).OnEvent
	/go/src/github.com/elastic/beats/filebeat/channel/outlet.go:58
github.com/elastic/beats/v7/filebeat/harvester.(*Forwarder).Send
	/go/src/github.com/elastic/beats/filebeat/harvester/forwarder.go:50
github.com/elastic/beats/v7/filebeat/input/syslog.GetCbByConfig.func3
	/go/src/github.com/elastic/beats/filebeat/input/syslog/input.go:202
github.com/elastic/beats/v7/filebeat/inputsource/common/dgram.DatagramReaderFactory.func1.1
	/go/src/github.com/elastic/beats/filebeat/inputsource/common/dgram/handler.go:88
github.com/elastic/beats/v7/filebeat/inputsource/common/dgram.(*Listener).connectAndRun
	/go/src/github.com/elastic/beats/filebeat/inputsource/common/dgram/server.go:122
github.com/elastic/beats/v7/filebeat/inputsource/common/dgram.(*Listener).Start.func1
	/go/src/github.com/elastic/beats/filebeat/inputsource/common/dgram/server.go:112
github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1
	/go/pkg/mod/github.com/elastic/go-concert@v0.2.0/unison/taskgroup.go:163 <nil>}
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 26, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 26, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@andrewkroh andrewkroh self-assigned this Jan 26, 2022
@garethhumphriesgkc
Copy link
Contributor Author

Confirmed not present in 7.13.2

@andrewkroh
Copy link
Member

That's my bad. I added the fix in #29268 that caused this. I will investigate. I forgot to fuzz the processor after my change.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Jan 26, 2022
@andrewkroh
Fix panic in decode_cef when recovering from invalid data
5803c76 
When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes elastic#30010
@andrewkroh andrewkroh mentioned this issue Jan 26, 2022
Merged
7 tasks
@andrewkroh
Copy link
Member

Fix: #30038

andrewkroh added a commit that referenced this issue Jan 27, 2022
@andrewkroh
[Filebeat] Fix panic in decode_cef when recovering from invalid data (#…
47b8d02 
…30038)

* Fix panic in decode_cef when recovering from invalid data

When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes #30010

* Encapsulate non-ragel state

Document and encapsulate the non-ragel state variables.

```
$ benchcmp before.txt after.txt
benchmark                   old ns/op     new ns/op     delta
BenchmarkEventUnpack-12     1991          1544          -22.45%

benchmark                   old allocs     new allocs     delta
BenchmarkEventUnpack-12     13             13             +0.00%

benchmark                   old bytes     new bytes     delta
BenchmarkEventUnpack-12     642           642           +0.00%
```
mergify bot pushed a commit that referenced this issue Jan 27, 2022
@andrewkroh
[Filebeat] Fix panic in decode_cef when recovering from invalid data (#…
329a4a7 
…30038)

* Fix panic in decode_cef when recovering from invalid data

When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes #30010

* Encapsulate non-ragel state

Document and encapsulate the non-ragel state variables.

```
$ benchcmp before.txt after.txt
benchmark                   old ns/op     new ns/op     delta
BenchmarkEventUnpack-12     1991          1544          -22.45%

benchmark                   old allocs     new allocs     delta
BenchmarkEventUnpack-12     13             13             +0.00%

benchmark                   old bytes     new bytes     delta
BenchmarkEventUnpack-12     642           642           +0.00%
```

(cherry picked from commit 47b8d02)
mergify bot pushed a commit that referenced this issue Jan 27, 2022
@andrewkroh
[Filebeat] Fix panic in decode_cef when recovering from invalid data (#…
4134cc0 
…30038)

* Fix panic in decode_cef when recovering from invalid data

When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes #30010

* Encapsulate non-ragel state

Document and encapsulate the non-ragel state variables.

```
$ benchcmp before.txt after.txt
benchmark                   old ns/op     new ns/op     delta
BenchmarkEventUnpack-12     1991          1544          -22.45%

benchmark                   old allocs     new allocs     delta
BenchmarkEventUnpack-12     13             13             +0.00%

benchmark                   old bytes     new bytes     delta
BenchmarkEventUnpack-12     642           642           +0.00%
```

(cherry picked from commit 47b8d02)
andrewkroh added a commit that referenced this issue Jan 27, 2022
@mergify @andrewkroh
[Filebeat] Fix panic in decode_cef when recovering from invalid data (#…
10782e4 
…30038) (#30042)

* Fix panic in decode_cef when recovering from invalid data

When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes #30010

* Encapsulate non-ragel state

Document and encapsulate the non-ragel state variables.

```
$ benchcmp before.txt after.txt
benchmark                   old ns/op     new ns/op     delta
BenchmarkEventUnpack-12     1991          1544          -22.45%

benchmark                   old allocs     new allocs     delta
BenchmarkEventUnpack-12     13             13             +0.00%

benchmark                   old bytes     new bytes     delta
BenchmarkEventUnpack-12     642           642           +0.00%
```

(cherry picked from commit 47b8d02)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
andrewkroh added a commit that referenced this issue Jan 27, 2022
@mergify @andrewkroh
[Filebeat] Fix panic in decode_cef when recovering from invalid data (#…
7b1dcba 
…30038) (#30043)

* Fix panic in decode_cef when recovering from invalid data

When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes #30010

* Encapsulate non-ragel state

Document and encapsulate the non-ragel state variables.

```
$ benchcmp before.txt after.txt
benchmark                   old ns/op     new ns/op     delta
BenchmarkEventUnpack-12     1991          1544          -22.45%

benchmark                   old allocs     new allocs     delta
BenchmarkEventUnpack-12     13             13             +0.00%

benchmark                   old bytes     new bytes     delta
BenchmarkEventUnpack-12     642           642           +0.00%
```

(cherry picked from commit 47b8d02)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
yashtewari pushed a commit to build-security/beats that referenced this issue Jan 30, 2022
@andrewkroh @yashtewari
[Filebeat] Fix panic in decode_cef when recovering from invalid data (e…
0f46b26 
…lastic#30038)

* Fix panic in decode_cef when recovering from invalid data

When recovering from an invalid extension value the escape sequence state was
not cleared. This caused the parser to attempt to unescape the next extension which
resulted in invalid data or a panic.

Fixes elastic#30010

* Encapsulate non-ragel state

Document and encapsulate the non-ragel state variables.

```
$ benchcmp before.txt after.txt
benchmark                   old ns/op     new ns/op     delta
BenchmarkEventUnpack-12     1991          1544          -22.45%

benchmark                   old allocs     new allocs     delta
BenchmarkEventUnpack-12     13             13             +0.00%

benchmark                   old bytes     new bytes     delta
BenchmarkEventUnpack-12     642           642           +0.00%
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewkroh andrewkroh

Labels
bug Filebeat Filebeat
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Filebeat] Fix panic in decode_cef when recovering from invalid data andrewkroh/beats
4 participants
@andrewkroh @mtojek @elasticmachine @garethhumphriesgkc