Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filebeat CEF module need the option to set a timezone or an offset #27232

Closed
LaZyDK opened this issue Aug 4, 2021 · 2 comments · Fixed by #27727
Closed

Filebeat CEF module need the option to set a timezone or an offset #27232

LaZyDK opened this issue Aug 4, 2021 · 2 comments · Fixed by #27727
Assignees
@andrewkroh

Comments

@LaZyDK
Copy link
Contributor

LaZyDK commented Aug 4, 2021 •
edited
Loading

Describe the enhancement:
The Filebeat CEF module is missing the option to set a timezone offset and the module is not taking into account the event.timezone in the ingest pipeline.

Describe a specific use case for the enhancement or feature:
Set the correct timezone via the event.timezone field, eg. like:

- add_locale: ~

or

- add_fields:
    target: "event"
    fields: 
      timezone: "Europe/Amsterdam"

The @timestamp should align with this.

Log:

<143>Aug 04 2021 11:31:32 192.168.200.1 CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|cat=Session Logs dvc=192.168.200.1 duser=ffe1a1c54fff src=192.168.100.1 spt=23 cs3=Ethernet cs3Label=Port Type dmac=3C-FF-FF-C5-FF-1B dst=192.168.100.1 cs1=0005000FF7DF cs1Label=Session Id start=12976 out=2399580 in=1660906 cn2=1671542075 cn2Label=Output Octets cn1=1132738271 cn1Label=Input Octets ArubaClearpassRADIUSAcctServiceName=Framed-User rt=Aug 04 2021 11:31:15 ArubaClearpassRADIUSAcctAuthentic=RADIUS

event.ingested: Aug 4, 2021 @ 11:31:45.324
@timestamp: Aug 4, 2021 @ 13:31:15.000 (wrong)

In this case event.ingested is taken from the CEF field rt, that is translated to the field cef.extensions.deviceReceiptTime
The source is sending in UTC+2 timezone.
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 20, 2021
@andrewkroh andrewkroh self-assigned this Sep 3, 2021
@andrewkroh andrewkroh mentioned this issue Sep 3, 2021
Merged
6 tasks
@andrewkroh
Copy link
Member

I opened a PR for decode_cef and the CEF module at #27727.

andrewkroh added a commit to andrewkroh/beats that referenced this issue Sep 3, 2021
@andrewkroh
Add timezone config option to decode_cef
c47f3c9 
CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

Fixes elastic#27232
andrewkroh added a commit that referenced this issue Sep 7, 2021
@andrewkroh
[Filebeat] Add timezone config option to decode_cef and syslog input (#…
b3497ca 
…27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232
mergify bot pushed a commit that referenced this issue Sep 7, 2021
@andrewkroh
[Filebeat] Add timezone config option to decode_cef and syslog input (#…
8eeb377 
…27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

(cherry picked from commit b3497ca)
jarpy pushed a commit that referenced this issue Sep 9, 2021
@andrewkroh
[Filebeat] Add timezone config option to decode_cef and syslog input (#…
1c99541 
…27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

(cherry picked from commit b3497ca)
andrewkroh added a commit that referenced this issue Sep 9, 2021
@mergify @andrewkroh
[Filebeat] Add timezone config option to decode_cef and syslog input (#…
9256741 
…27727) (#27780)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes #27232

(cherry picked from commit b3497ca)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Icedroid pushed a commit to Icedroid/beats that referenced this issue Nov 1, 2021
@andrewkroh
[Filebeat] Add timezone config option to decode_cef and syslog input (e…
12f05fc 
…lastic#27727)

CEF message that contain timestamps without a timezone were parsed as UTC. The time zone was not
configurable. This adds a `timezone` option to the decode_cef processor and cef module to allow the
time zone to be specified when a timestamp does not contain an offset or zone.

    CEF:0|Aruba Networks|ClearPass|6.8.7.120583|2002|RADIUS Accounting|1|rt=Aug 04 2021 11:31:15

Note that the CEF module receives messages using the syslog input. The syslog input does not have
a configurable time zone and always assumes timestamps without time zones are given in the machine's
local time zone. This change won't affect how the syslog envelop's time stamp is parsed by the module.

This also replaces the deprecated `import "4d63.com/tz"` with Go's relatively new built-in
`time/tzdata` package. The `timestamp` processor was updated.

While I was adding the a timezone config type I made the syslog input's timezone configurable too.

Fixes elastic#27232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewkroh andrewkroh

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Filebeat] Add timezone config option to decode_cef and syslog input andrewkroh/beats
4 participants
@jsoriano @andrewkroh @LaZyDK @elasticmachine