Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!(gcp_stackdriver_logs sink): Support more fields #22470

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

fbs
Copy link

@fbs fbs commented Feb 18, 2025

draft: I'm not sure this is the right approach, as its a breaking change and doesn't follow the pattern of other fields, like severity key. However, using a remap to get events in the right format seems flexible than having a bunch of 'xx key' config options, for which you still need a remap to provide a default/fallback.

If the approach is ok I'll add more fields, fix up the tests and the changelog.

Summary

The LogEntry1 type supports a lot more fields than are currently supported by the sink. This change adds support for labels, but more fields can be added the same way.

Additionally this removes the timestamp from the message, if set, instead of having it in both the 'timestamp' field, as well as part of the message payload.

It also changes the payload type from json to text if the message is only a string. This makes the GCP search queries simpler, e.g. textPayload: "some message" instead of jsonPayload.msg: "some message".

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

Sending logs to GCP

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

  • Please read our Vector contributor resources.
    • make check-all is a good command to run locally. This check is
      defined here. Some of these
      checks might not be relevant to your PR. For Rust changes, at the very least you should run:
      • cargo fmt --all
      • cargo clippy --workspace --all-targets -- -D warnings
      • cargo nextest run --workspace (alternatively, you can run cargo test --all)
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

The LogEntry[1] type supports a lot more fields than are currently
supported by the sink. This change adds support for labels, but more
fields can be added the same way.

Additionally this removes the timestamp from the message, if set,
instead of having it in both the 'timestamp' field, as well as part of
the message payload.

It also changes the payload type from json to text if the message is
only a string. This makes the GCP search queries simpler, e.g.
`textPayload: "some message" instead of `jsonPayload.msg: "some
message".

[1]: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
@bits-bot
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: sinks Anything related to the Vector's sinks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants