Ability to remove PII data from an event stream through an API

[jeremydmiller]

Provide a mechanism where users can define a pattern of replacing or masking data within event data for a single stream at a time

Note: this is well short of "Crypto Shedding", which will not appear until at least Marten 8.

[jeremydmiller]

This is just getting pushed back to 7.29 is all. Still coming soon

[jeremydmiller]

Proposed Syntax

For configuration at bootstrapping time:

Right now,

// Specify what members are protected information by event type
// Might chain this off of the registration to add event types!
// Make this additive, params Expression<Func<TEvent, object>>[] members

// Allow you to decide on replacing the data with a mask in the JSON, or...
opts.Events.MaskProtectedInformation<TEvent>("****", x => x.Property, x => x.AnotherProperty);

// Delete the member in JSON
opts.Events.DeleteProtectedInformation<TEvent>(x => x.Property, x => x.AnotherProperty);

// What about child collections? Do you wipe out the entire collection? Mask inside the collection?

Runtime

// One stream at a time
IDocumentSession.Advanced.RemoveProtectedInformationFromStream(streamId, streamKey);

// Range of streams
IDocumentSession.Advanced.RemoveProtectedInformationFromStreams(Expression<Func<IEvent, bool>>)

// Still necessary
session.SaveChangesAsync();

Implementation

I have been assuming that we'd use the JSONPatch functionality we built in for this, but the worst case scenario is to use server side JSON manipulation code

[iamsamcoder]

Hi Jeremy,

For our particular case, we need to modify only certain events of TEvent in a stream. For example, if there was a stream of events including types A, B, C and there are multiple B events. We need to be able to mask only specific B events.

It would also be nice to add metadata to that event. Something like the following:

/// <summary>
/// Removes protected information from a stream based on a specified filter and adds metadata to the event.
/// </summary>
/// <param name="streamId">The unique identifier of the stream.</param>
/// <param name="streamKey">The key associated with the stream.</param>
/// <param name="filter">A filter expression to select specific events to be masked.</param>
/// <param name="metadataToAdd">A dictionary containing metadata to be added to the event.</param>
void RemoveProtectedInformationFromStream(Guid streamId, string streamKey, Expression<Func<IEvent, bool>> filter, IDictionary<string, object> metadataToAdd);

In our case, we would like to add DateMasked and OperationId to the metadata.