Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parse_syslog function ignores timezone when rfc3164 #853

Open
timansky opened this issue May 19, 2024 · 3 comments
Open

parse_syslog function ignores timezone when rfc3164 #853

timansky opened this issue May 19, 2024 · 3 comments
Labels
type: bug A code related bug vrl: stdlib Changes to the standard library

Comments

@timansky
Copy link

timansky commented May 19, 2024

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Most devices are sending logs in rfc3164 format, which does not contain timezone.
When syslog message in rfc3164 format, parse_syslog function always using UTC timezone when parsing timestamp

Configuration

transform:
  adapt_syslog:
    type: remap
    timezone: "Asia/Almaty" # Apply this timezone when parsing timestamp
    inputs:
    - syslog
    source:  |
      d = parse_timestamp!("May 19 2024 12:00:00", format: "%h %d %Y %T")
      log("debug1:" + format_timestamp!(d, format: "%+"))

      d = parse_syslog!(s'<180>May 19 2024 12:00:00 hostname message')
      log("debug2:" + format_timestamp!(d.timestamp, format: "%+"))

Version

0.38.0

Debug Output

INFO transform{component_kind="transform" component_id=adapt_syslog component_type=remap}: vrl::stdlib::log: Internal log [debug1:2024-05-19T07:00:00+00:00] is being suppressed to avoid flooding.

INFO transform{component_kind="transform" component_id=adapt_syslog component_type=remap}: vrl::stdlib::log: Internal log [debug2:2024-05-19T12:00:00+00:00] is being suppressed to avoid flooding.

Example Data

<180>May 19 2024 12:12:12 hostname message

Additional Context

No response

References

No response

@timansky timansky added the type: bug A code related bug label May 19, 2024
@jszwedko
Copy link
Member

Thanks for this @timansky .

Internally, Vector stores all timestamps as UTC. The timestamp option just configures how the timestamp is interpreted if it is missing a timezone. See that the output:

INFO transform{component_kind="transform" component_id=adapt_syslog component_type=remap}: vrl::stdlib::log: Internal log [debug1:2024-05-19T07:00:00+00:00] is being suppressed to avoid flooding.

Is showing the correct timestamp in UTC for the given timestamp in Asia/Almaty which is +05:00.

vectordotdev/vector#3333 is a ticket that is tracking actually storing the timezone of parsed timestamps rather than always normalizing to UTC.

I'll close this out since there doesn't seem to be a bug here, just a lack of vectordotdev/vector#3333 being implemented, but let me know if I'm missing something!

@jszwedko jszwedko closed this as not planned Won't fix, can't repro, duplicate, stale May 20, 2024
@timansky
Copy link
Author

I know that vector storing timestamp in UTC.
But the behavior is different from what is described in the documentation. For example, syslog source parses timestamps with appling timezone if timezone missing in event and then converts to UTC. Also parse_timestamp applies but parse_syslog does not.

All this leads to the fact that we receive different data for the same event.

To reproduce it is just need to have timezone set and also 2 sources:
1 syslog
2 socket, then transform with parse_syslog

Example message: <180>May 19 2024 12:12:12 hostname message

Timestamp will be different

@jszwedko
Copy link
Member

. Also parse_timestamp applies but parse_syslog does not.

Ah I see now. Let me reopen this and move it to the VRL repository.

@jszwedko jszwedko reopened this May 22, 2024
@jszwedko jszwedko transferred this issue from vectordotdev/vector May 22, 2024
@jszwedko jszwedko added the vrl: stdlib Changes to the standard library label May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A code related bug vrl: stdlib Changes to the standard library
Projects
None yet
Development

No branches or pull requests

2 participants