Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add: the ability to redact values from Stringer and Error interface. #592

Merged
merged 8 commits into from
Apr 1, 2024

Conversation

moisesvega
Copy link
Collaborator

@moisesvega moisesvega commented Feb 7, 2024

Currently, there's no feature to exclude sensitive information from outputs generated by the Stringer and Error interface implementations. However, we use the nolog annotation to exclude specific properties from being logged by the zap.objectMarshal method. This is particularly useful for omitting sensitive data in zap logger outputs. Despite this, there are scenarios where sensitive information might still be exposed. For instance, consider the following example:

exception DoesNotExistException {
   /** Key that was missing. */
   1: required string key
   2: optional string Error (go.name="Error2")
   3: optional string userName (go.nolog)
}

The code snippet above leads to the following implementation for the MarshalLogObject method:

// MarshalLogObject allows DoesNotExistException to be logged quickly by zap.
func (v *DoesNotExistException) MarshalLogObject(enc zapcore.ObjectEncoder) error {
   if v == nil {
      return nil
   }
   enc.AddString("key", v.Key)
   if v.Error2 != nil {
      enc.AddString("Error", *v.Error2)
   }
   return nil
}

However, our implementation of the Stringer and Error interfaces inadvertently exposes properties marked with nolog:

func (v *DoesNotExistException) Error() string {
   return v.String()
}

// String provides a human-readable representation of DoesNotExistException.
func (v *DoesNotExistException) String() string {
   if v == nil {
      return "<nil>"
   }

   var fields [3]string
   i := 0
   fields[i] = fmt.Sprintf("Key: %v", v.Key); i++
   if v.Error2 != nil {
      fields[i] = fmt.Sprintf("Error2: %v", *v.Error2); i++
   }
   if v.UserName != nil {
      fields[i] = fmt.Sprintf("UserName: %v", *v.UserName); i++
   }

   return fmt.Sprintf("DoesNotExistException{%s}", strings.Join(fields[:i], ", "))
}

This behavior can lead to unintended exposure of information annotated as nolog. For example, when this exception is utilized as an Error, or when logged using zap.Stringer("exception", e) or zap.Error(e) , the nolog-annotated property is inadvertently revealed:

playground link

func main() {
   name := "Jane Doe"
   e := &DoesNotExistException{
      Key:      "",
      Error2:   nil,
      UserName: &name,
   }
   logger, _ := zap.NewProduction()
   defer logger.Sync() // Ensures buffer is flushed
   logger.Info("hello",
      zap.Object("exception", e),
      zap.Error(e),
   )
     // output: {"level":"info","msg":"hello","exception":{"key":""},"error":"DoesNotExistException{Key: , UserName: Jane Doe"}
}

This PR introduces a new feature that enables the redaction of specified properties through a new Go annotation, go.redacted. When applied, this annotation ensures that the actual value of a property is replaced with in the outputs generated by our Stringer and Error interface implementations.

Output after this diff:

  // output: {"level":"info","msg":"hello","exception":{"key":""},"error":"DoesNotExistException{Key: , UserName: <redacted>}"}

Implemented privacy measures to strip all PII from zap log entries,
and from the outputs of String() and Error() methods.

This update minimizes privacy risks by sanitizing log entries and error
information.
Copy link

codecov bot commented Feb 7, 2024

Codecov Report

Attention: Patch coverage is 76.27119% with 28 lines in your changes are missing coverage. Please review.

Project coverage is 69.03%. Comparing base (12ffd74) to head (6296da8).

Files Patch % Lines
gen/internal/tests/exceptions/exceptions.go 64.10% 7 Missing and 7 partials ⚠️
gen/internal/tests/structs/structs.go 64.10% 7 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #592      +/-   ##
==========================================
+ Coverage   69.01%   69.03%   +0.01%     
==========================================
  Files         140      140              
  Lines       23523    23625     +102     
==========================================
+ Hits        16235    16309      +74     
- Misses       4228     4242      +14     
- Partials     3060     3074      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@moisesvega moisesvega changed the title feat: Remove PII from Logs and Error Outputs feat: Remove PII from Logs and Error outputs Feb 7, 2024
Copy link
Collaborator Author

@moisesvega moisesvega left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on behalf of @r-hang

gen/field.go Outdated Show resolved Hide resolved
gen/field.go Outdated Show resolved Hide resolved
gen/zap_test.go Outdated Show resolved Hide resolved
@moisesvega moisesvega changed the title feat: Remove PII from Logs and Error outputs Add: the ability to redact values from Stringer and Error interface. Mar 14, 2024
Copy link
Contributor

@r-hang r-hang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good on behalf of @tchung1118.

We can add the redacted feature designation to zap logs instead if skipping them outright.

CHANGELOG.md Outdated Show resolved Hide resolved
gen/field.go Outdated Show resolved Hide resolved
gen/field.go Outdated Show resolved Hide resolved
@@ -852,3 +853,26 @@ func TestLogNilStruct(t *testing.T) {
require.NoError(t, x.MarshalLogObject(enc))
assert.Empty(t, enc.Fields)
}

func TestZapOptOut(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a new test? Should we also be adding a the case here for a redacted field since we're adding this new feature here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is. I noticed that no tests have been conducted for ZapOut. Implementing a small test could help increase our code coverage. I updated the test and add a test case.

@r-hang r-hang merged commit cebd65d into thriftrw:dev Apr 1, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants