Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Determining if redact has redacted #1040

Open
byronwolfman opened this issue Sep 18, 2024 · 1 comment
Open

Determining if redact has redacted #1040

byronwolfman opened this issue Sep 18, 2024 · 1 comment
Labels
vrl: stdlib Changes to the standard library

Comments

@byronwolfman
Copy link

Our team has been playing around with redact and something we're trying to wrap our heads around is how to determine if redact has modified the input. This is straightforward when the input is a string, but we're not sure what to do with objects. We've tried abusing assert_eq but the docs recommend against using assertions outside of unit tests.

An "easy" way around this problem would be if redact could return two values:

  1. The <string | object | array> output as usual
  2. A <boolean> indicating whether or not the output has changed

The workaround I can see for now is something along the lines of

redacted = redact(., [r'\d+'])

if encode_json(.) != encode_json(redacted) {
    . = redacted
    .tags = ["redacted:number"]
}

My hesitation is that I'm not sure how expensive it is to marshal JSON twice on every log event, especially as we're operating at a decently large scale. On the other hand, Vector has made me into a Rust true believer, and maybe I'm over-thinking the performance implications.

@jszwedko jszwedko added the vrl: stdlib Changes to the standard library label Sep 23, 2024
@jszwedko
Copy link
Member

Thanks for opening this request @byronwolfman ! I agree that having an indicator of whether data was redacted or not would be useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
vrl: stdlib Changes to the standard library
Projects
None yet
Development

No branches or pull requests

2 participants