Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added an option to also mask a single event stream with a selected fi… #3517

Merged
merged 1 commit into from
Oct 25, 2024
Merged

Added an option to also mask a single event stream with a selected fi… #3517

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/documents/multi-tenancy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,7 +446,7 @@ builder.Services.AddMarten(opts =>
opts.Policies.PartitionMultiTenantedDocumentsUsingMartenManagement("tenants");
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L151-L168' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configure_marten_managed_tenant_partitioning' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L189-L206' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configure_marten_managed_tenant_partitioning' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The tenant to partition name mapping will be stored in a table created by Marten called `mt_tenant_partitions` with
Expand All @@ -467,7 +467,7 @@ await theStore
// with the named tenant ids
.AddMartenManagedTenantsAsync(CancellationToken.None, "a1", "a2", "a3");
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L57-L65' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add_managed_tenants_at_runtime' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L58-L66' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add_managed_tenants_at_runtime' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The API above will try to add any missing table partitions to all known document types. There is also a separate overload
Expand All @@ -492,7 +492,7 @@ public class DocThatShouldBeExempted1
public Guid Id { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L222-L230' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_donotpartitionattribute' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L260-L268' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_donotpartitionattribute' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

or exempt a single document type through the fluent interface:
Expand All @@ -502,7 +502,7 @@ or exempt a single document type through the fluent interface:
```cs
opts.Schema.For<DocThatShouldBeExempted2>().DoNotPartition();
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L207-L211' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_exempt_from_partitioning_through_fluent_interface' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L245-L249' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_exempt_from_partitioning_through_fluent_interface' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Implementation Details
Expand Down
32 changes: 28 additions & 4 deletions docs/events/protection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ builder.Services.AddMarten(opts =>
});
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L367-L395' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_defining_masking_rules' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L460-L488' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_defining_masking_rules' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

That's strictly a configuration time effort. Next, you can apply the masking on demand to any subset of events with
Expand All @@ -61,7 +61,31 @@ public static Task apply_masking_to_streams(IDocumentStore store, Guid streamId,
}, token);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L398-L414' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_to_a_single_stream' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L491-L507' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_to_a_single_stream' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

As a finer grained operation, you can specify an event filter (`Func<IEvent, bool>`) within an event stream to be masked with
this overload:

<!-- snippet: sample_apply_masking_to_a_single_stream_and_filter -->
<a id='snippet-sample_apply_masking_to_a_single_stream_and_filter'></a>
```cs
public static Task apply_masking_to_streams_and_filter(IDocumentStore store, Guid streamId, CancellationToken token)
{
return store
.Advanced
.ApplyEventDataMasking(x =>
{
// Mask selected events within a single stream by a user defined criteria
x.IncludeStream(streamId, e => e.EventTypesAre(typeof(MembersJoined), typeof(MembersDeparted)));

// You can add or modify event metadata headers as well
// BUT, you'll of course need event header tracking to be enabled
x.AddHeader("masked", DateTimeOffset.UtcNow);
}, token);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L509-L526' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_to_a_single_stream_and_filter' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

::: tip
Expand All @@ -82,7 +106,7 @@ public static Task apply_masking_by_filter(IDocumentStore store, Guid[] streamId
});
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L416-L426' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_by_filter' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L529-L539' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_by_filter' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Finally, if you are using multi-tenancy, you can specify the tenant id as part of the same fluent interface:
Expand All @@ -104,7 +128,7 @@ public static Task apply_masking_by_tenant(IDocumentStore store, string tenantId
});
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L428-L444' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_with_multi_tenancy' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/EventSourcingTests/removing_protected_information.cs#L541-L557' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_apply_masking_with_multi_tenancy' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Here's a couple more facts you might need to know:
Expand Down
113 changes: 113 additions & 0 deletions src/EventSourcingTests/removing_protected_information.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,100 @@ await theStore.Advanced.ApplyEventDataMasking(x =>
.Single().Name.ShouldBe("*****");
}

[Fact]
public async Task end_to_end_masking_by_guid_identified_stream_and_filter_within_stream()
{
StoreOptions(opts =>
{
opts.Events.AddMaskingRuleForProtectedInformation<QuestStarted>(x => x.Name = "*****");
opts.Events.AddMaskingRuleForProtectedInformation<MembersJoined>(x =>
{
for (int i = 0; i < x.Members.Length; i++)
{
x.Members[i] = "*****";
}
});
opts.Events.MetadataConfig.HeadersEnabled = true;
});

theSession.SetHeader("color", "blue");

var streamId = Guid.NewGuid();
theSession.Events.StartStream<Quest>(streamId, new QuestStarted{Name = "Find Gandalf"}, new MembersJoined(1, "Hobbiton", "Frodo", "Sam"), new MembersJoined(3, "Brandybuck", "Merry", "Pippin"));
await theSession.SaveChangesAsync();

theSession.Events.Append(streamId, new MembersDeparted { Members = new string[] { "Frodo" } });
await theSession.SaveChangesAsync();

await theStore.Advanced.ApplyEventDataMasking(x =>
{
x
.IncludeStream(streamId, e => e.Data is MembersJoined { Location: "Hobbiton" })
.AddHeader("color", "green")
.AddHeader("opid", 1);
});

var events = await theSession.Events.FetchStreamAsync(streamId);

// Should have matched and been masked
var hobbiton = events.OfType<Event<MembersJoined>>().Single(x => x.Data.Location == "Hobbiton");
hobbiton.Headers["color"].ShouldBe("green");
hobbiton.Data.Members.All(x => x == "*****").ShouldBeTrue();


// Should NOT have been matched or masked
var brandybuck = events.OfType<Event<MembersJoined>>().Single(x => x.Data.Location == "Brandybuck");
brandybuck.Headers["color"].ShouldBe("blue");
brandybuck.Data.Members.All(x => x != "*****").ShouldBeTrue();
}

[Fact]
public async Task end_to_end_masking_by_string_identified_stream_and_filter_within_stream()
{
StoreOptions(opts =>
{
opts.Events.StreamIdentity = StreamIdentity.AsString;
opts.Events.AddMaskingRuleForProtectedInformation<QuestStarted>(x => x.Name = "*****");
opts.Events.AddMaskingRuleForProtectedInformation<MembersJoined>(x =>
{
for (int i = 0; i < x.Members.Length; i++)
{
x.Members[i] = "*****";
}
});
opts.Events.MetadataConfig.HeadersEnabled = true;
});

theSession.SetHeader("color", "blue");

var streamId = Guid.NewGuid().ToString();
theSession.Events.StartStream<Quest>(streamId, new QuestStarted{Name = "Find Gandalf"}, new MembersJoined(1, "Hobbiton", "Frodo", "Sam"), new MembersJoined(3, "Brandybuck", "Merry", "Pippin"));
await theSession.SaveChangesAsync();

theSession.Events.Append(streamId, new MembersDeparted { Members = new string[] { "Frodo" } });
await theSession.SaveChangesAsync();

await theStore.Advanced.ApplyEventDataMasking(x =>
{
x
.IncludeStream(streamId, e => e.Data is MembersJoined { Location: "Hobbiton" })
.AddHeader("color", "green")
.AddHeader("opid", 1);
});

var events = await theSession.Events.FetchStreamAsync(streamId);

// Should have matched and been masked
var hobbiton = events.OfType<Event<MembersJoined>>().Single(x => x.Data.Location == "Hobbiton");
hobbiton.Headers["color"].ShouldBe("green");
hobbiton.Data.Members.All(x => x == "*****").ShouldBeTrue();


// Should NOT have been matched or masked
var brandybuck = events.OfType<Event<MembersJoined>>().Single(x => x.Data.Location == "Brandybuck");
brandybuck.Headers["color"].ShouldBe("blue");
brandybuck.Data.Members.All(x => x != "*****").ShouldBeTrue();
}
}


Expand Down Expand Up @@ -413,6 +506,26 @@ public static Task apply_masking_to_streams(IDocumentStore store, Guid streamId,

#endregion

#region sample_apply_masking_to_a_single_stream_and_filter

public static Task apply_masking_to_streams_and_filter(IDocumentStore store, Guid streamId, CancellationToken token)
{
return store
.Advanced
.ApplyEventDataMasking(x =>
{
// Mask selected events within a single stream by a user defined criteria
x.IncludeStream(streamId, e => e.EventTypesAre(typeof(MembersJoined), typeof(MembersDeparted)));

// You can add or modify event metadata headers as well
// BUT, you'll of course need event header tracking to be enabled
x.AddHeader("masked", DateTimeOffset.UtcNow);
}, token);
}

#endregion


#region sample_apply_masking_by_filter

public static Task apply_masking_by_filter(IDocumentStore store, Guid[] streamIds)
Expand Down
38 changes: 38 additions & 0 deletions src/Marten/Events/Protected/EventDataMasking.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,28 @@ public IEventDataMasking IncludeStream(string streamKey)
return this;
}

public IEventDataMasking IncludeStream(Guid streamId, Func<IEvent, bool> filter)
{
_sources.Add(async (s, t) =>
{
var raw = await s.Events.FetchStreamAsync(streamId, token: t).ConfigureAwait(false);
return raw.Where(filter).ToList();
});

return this;
}

public IEventDataMasking IncludeStream(string streamKey, Func<IEvent, bool> filter)
{
_sources.Add(async (s, t) =>
{
var raw = await s.Events.FetchStreamAsync(streamKey, token: t).ConfigureAwait(false);
return raw.Where(filter).ToList();
});

return this;
}

public IEventDataMasking IncludeEvents(Expression<Func<IEvent, bool>> filter)
{
_sources.Add((s, t) => s.Events.QueryAllRawEvents().Where(filter).ToListAsync(t));
Expand Down Expand Up @@ -108,6 +130,22 @@ public interface IEventDataMasking
/// <returns></returns>
IEventDataMasking IncludeStream(string streamKey);

/// <summary>
/// Apply data protection masking to this event stream
/// </summary>
/// <param name="streamId"></param>
/// <param name="filter">Further filter events within the stream to more finely target events for masking</param>
/// <returns></returns>
IEventDataMasking IncludeStream(Guid streamId, Func<IEvent, bool> filter);

/// <summary>
/// Apply data protection masking to this event stream
/// </summary>
/// <param name="streamKey"></param>
/// <param name="filter">Further filter events within the stream to more finely target events for masking</param>
/// <returns></returns>
IEventDataMasking IncludeStream(string streamKey, Func<IEvent, bool> filter);

/// <summary>
/// Apply data protection masking to events matching
/// this criteria
Expand Down
Loading