Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tenant partitioning improvements and fixes #3451

Merged
merged 2 commits into from
Sep 26, 2024
Merged

Tenant partitioning improvements and fixes #3451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6aa5216
Adding the ability to exempt certain documents from the per tenant pa…
jeremydmiller Sep 26, 2024
84ac25e
Upgrading Weasel to fix an issue with managed list partitions when no…
jeremydmiller Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions docs/configuration/retries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,16 @@ Out of the box, Marten is using Polly for resiliency on most operations with thi
return builder
.AddRetry(new()
{
ShouldHandle = new PredicateBuilder().Handle<NpgsqlException>().Handle<MartenCommandException>().Handle<EventLoaderException>(),
ShouldHandle = new PredicateBuilder()
.Handle<NpgsqlException>()
.Handle<MartenCommandException>()
.Handle<EventLoaderException>(),
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(50),
BackoffType = DelayBackoffType.Exponential
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten/Util/ResilientPipelineBuilderExtensions.cs#L13-L25' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_default_polly_setup' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten/Util/ResilientPipelineBuilderExtensions.cs#L13-L28' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_default_polly_setup' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The general idea is to have _some_ level of retry with an exponential backoff on typical transient errors encountered
Expand Down
6 changes: 3 additions & 3 deletions docs/configuration/storeoptions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ public static DocumentStore For(Action<StoreOptions> configure)
return new DocumentStore(options);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten/DocumentStore.cs#L521-L531' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_documentstore.for' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/Marten/DocumentStore.cs#L520-L530' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_documentstore.for' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The major parts of `StoreOptions` are shown in the class diagram below:
Expand Down Expand Up @@ -211,7 +211,7 @@ public class ConfiguresItself
}
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L837-L849' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-generic' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L851-L863' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-generic' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The `DocumentMapping` type is the core configuration class representing how a document type is persisted or
Expand All @@ -235,7 +235,7 @@ public class ConfiguresItselfSpecifically
}
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L851-L864' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-specifically' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/Configuration/DocumentMappingTests.cs#L865-L878' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configuremarten-specifically' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Document Policies
Expand Down
89 changes: 86 additions & 3 deletions docs/documents/identity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -529,7 +529,32 @@ public class LimitedDoc
public LowerLimit Lower { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/linq_querying_with_value_types.cs#L73-L88' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_limited_doc' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/linq_querying_with_value_types.cs#L74-L89' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_limited_doc' title='Start of snippet'>anchor</a></sup>
<a id='snippet-sample_limited_doc-1'></a>
```cs
[ValueObject<long>]
public partial struct UpperLimit;

[ValueObject<int>]
public partial struct LowerLimit;

[ValueObject<string>]
public partial struct Description;

[ValueObject<Guid>]
public partial struct GuidId;

public class LimitedDoc
{
public Guid Id { get; set; }

public GuidId? ParentId { get; set; }
public UpperLimit? Upper { get; set; }
public LowerLimit Lower { get; set; }
public Description? Description { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/Vogen/linq_querying_with_value_types.cs#L217-L241' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_limited_doc-1' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

And the `UpperLimit` and `LowerLimit` value types can be registered with Marten like so:
Expand All @@ -542,7 +567,17 @@ And the `UpperLimit` and `LowerLimit` value types can be registered with Marten
opts.RegisterValueType(typeof(UpperLimit));
opts.RegisterValueType(typeof(LowerLimit));
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/linq_querying_with_value_types.cs#L16-L23' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_registering_value_types' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/linq_querying_with_value_types.cs#L17-L24' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_registering_value_types' title='Start of snippet'>anchor</a></sup>
<a id='snippet-sample_registering_value_types-1'></a>
```cs
// opts is a StoreOptions just like you'd have in
// AddMarten() calls
opts.RegisterValueType(typeof(GuidId));
opts.RegisterValueType(typeof(UpperLimit));
opts.RegisterValueType(typeof(LowerLimit));
opts.RegisterValueType(typeof(Description));
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/Vogen/linq_querying_with_value_types.cs#L16-L25' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_registering_value_types-1' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

And that will enable you to seamlessly use the value types in LINQ expressions like so:
Expand Down Expand Up @@ -570,5 +605,53 @@ public async Task store_several_and_order_by()
ordered.ShouldHaveTheSameElementsAs(doc1.Id, doc4.Id, doc3.Id, doc2.Id);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/linq_querying_with_value_types.cs#L27-L49' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_value_type_in_linq' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/linq_querying_with_value_types.cs#L28-L50' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_value_type_in_linq' title='Start of snippet'>anchor</a></sup>
<a id='snippet-sample_using_value_type_in_linq-1'></a>
```cs
[Fact]
public async Task store_several_and_use_in_LINQ_order_by()
{
var commonParentId = GuidId.From(Guid.NewGuid());
var doc1 = new LimitedDoc { ParentId = commonParentId, Lower = LowerLimit.From(1), Upper = UpperLimit.From(20), Description = Description.From("desc1") };
var doc2 = new LimitedDoc { Lower = LowerLimit.From(5), Upper = UpperLimit.From(25), Description = Description.From("desc3") };
var doc3 = new LimitedDoc { Lower = LowerLimit.From(4), Upper = UpperLimit.From(15), Description = Description.From("desc2") };
var doc4 = new LimitedDoc { ParentId = commonParentId, Lower = LowerLimit.From(3), Upper = UpperLimit.From(10), Description = Description.From("desc4") };

theSession.Store(doc1, doc2, doc3, doc4);
await theSession.SaveChangesAsync();

var orderedByIntBased = await theSession
.Query<LimitedDoc>()
.OrderBy(x => x.Lower)
.Select(x => x.Id)
.ToListAsync();

orderedByIntBased.ShouldHaveTheSameElementsAs(doc1.Id, doc4.Id, doc3.Id, doc2.Id);

var orderedByLongBased = await theSession
.Query<LimitedDoc>()
.OrderBy(x => x.Upper)
.Select(x => x.Id)
.ToListAsync();

orderedByLongBased.ShouldHaveTheSameElementsAs(doc4.Id, doc3.Id, doc1.Id, doc2.Id);

var orderedByStringBased = await theSession
.Query<LimitedDoc>()
.OrderBy(x => x.Description)
.Select(x => x.Id)
.ToListAsync();

orderedByStringBased.ShouldHaveTheSameElementsAs(doc1.Id, doc3.Id, doc2.Id, doc4.Id);

var orderedByGuidBased = await theSession
.Query<LimitedDoc>()
.OrderBy(x => x.ParentId)
.Select(x => x.Id)
.ToListAsync();

orderedByGuidBased.ShouldHaveTheSameElementsAs(doc1.Id, doc4.Id, doc2.Id, doc3.Id);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/ValueTypeTests/Vogen/linq_querying_with_value_types.cs#L29-L76' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_value_type_in_linq-1' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->
70 changes: 62 additions & 8 deletions docs/documents/multi-tenancy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ using (var session = store.QuerySession())
session.Query<Target>().Count(x => x.TenantIsOneOf("Red")).ShouldBe(11);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L258-L325' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenancy-mixed-tenancy-non-tenancy-sample' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy_with_partitioning.cs#L268-L335' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenancy-mixed-tenancy-non-tenancy-sample' title='Start of snippet'>anchor</a></sup>
<a id='snippet-sample_tenancy-mixed-tenancy-non-tenancy-sample-1'></a>
```cs
using var store = DocumentStore.For(opts =>
Expand Down Expand Up @@ -208,7 +208,7 @@ using (var session = store.QuerySession())
session.Query<Target>().Count(x => x.TenantIsOneOf("Red")).ShouldBe(11);
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L988-L1055' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenancy-mixed-tenancy-non-tenancy-sample-1' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L249-L316' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenancy-mixed-tenancy-non-tenancy-sample-1' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

In some cases, You may want to disable using the default tenant for storing documents, set `StoreOptions.DefaultTenantUsageEnabled` to `false`. With this option disabled, Tenant (non-default tenant) should be passed via method argument or `SessionOptions` when creating a session using document store. Marten will throw an exception `DefaultTenantUsageDisabledException` if a session is created using default tenant.
Expand Down Expand Up @@ -244,14 +244,14 @@ filter:
var actual = await query.Query<Target>().Where(x => x.TenantIsOneOf("Green", "Red") && x.Flag)
.OrderBy(x => x.Id).Select(x => x.Id).ToListAsync();
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L402-L408' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenant_is_one_of' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy_with_partitioning.cs#L412-L418' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenant_is_one_of' title='Start of snippet'>anchor</a></sup>
<a id='snippet-sample_tenant_is_one_of-1'></a>
```cs
// query data for a selected list of tenants
var actual = await query.Query<Target>().Where(x => x.TenantIsOneOf("Green", "Red") && x.Flag)
.OrderBy(x => x.Id).Select(x => x.Id).ToListAsync();
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L1132-L1138' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenant_is_one_of-1' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L393-L399' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_tenant_is_one_of-1' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

Or the `AnyTenant()` filter:
Expand All @@ -263,14 +263,14 @@ Or the `AnyTenant()` filter:
var actual = query.Query<Target>().Where(x => x.AnyTenant() && x.Flag)
.OrderBy(x => x.Id).Select(x => x.Id).ToArray();
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L347-L353' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_any_tenant' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy_with_partitioning.cs#L357-L363' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_any_tenant' title='Start of snippet'>anchor</a></sup>
<a id='snippet-sample_any_tenant-1'></a>
```cs
// query data across all tenants
var actual = query.Query<Target>().Where(x => x.AnyTenant() && x.Flag)
.OrderBy(x => x.Id).Select(x => x.Id).ToArray();
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L1077-L1083' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_any_tenant-1' title='Start of snippet'>anchor</a></sup>
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/DocumentDbTests/MultiTenancy/conjoined_multi_tenancy.cs#L338-L344' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_any_tenant-1' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Configuring Tenancy
Expand Down Expand Up @@ -428,7 +428,26 @@ document storage.

Here's a sample of using this feature. First, the configuration is:

snippet: sample_configure_marten_managed_tenant_partitioning
<!-- snippet: sample_configure_marten_managed_tenant_partitioning -->
<a id='snippet-sample_configure_marten_managed_tenant_partitioning'></a>
```cs
var builder = Host.CreateApplicationBuilder();
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));

// Make all document types use "conjoined" multi-tenancy -- unless explicitly marked with
// [SingleTenanted] or explicitly configured via the fluent interfce
// to be single-tenanted
opts.Policies.AllDocumentsAreMultiTenanted();

// It's required to explicitly tell Marten which database schema to put
// the mt_tenant_partitions table
opts.Policies.PartitionMultiTenantedDocumentsUsingMartenManagement("tenants");
});
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L113-L130' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_configure_marten_managed_tenant_partitioning' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The tenant to partition name mapping will be stored in a table created by Marten called `mt_tenant_partitions` with
two columns:
Expand All @@ -439,7 +458,17 @@ two columns:
Before the application is initialized, it's possible to load or delete data directly into these tables. At runtime,
you can add new tenant id partitions with this helper API on `IDocumentStore.Advanced`:

snippet: sample_add_managed_tenants_at_runtime
<!-- snippet: sample_add_managed_tenants_at_runtime -->
<a id='snippet-sample_add_managed_tenants_at_runtime'></a>
```cs
await theStore
.Advanced
// This is ensuring that there are tenant id partitions for all multi-tenanted documents
// with the named tenant ids
.AddMartenManagedTenantsAsync(CancellationToken.None, "a1", "a2", "a3");
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L56-L64' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_add_managed_tenants_at_runtime' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

The API above will try to add any missing table partitions to all known document types. There is also a separate overload
that will take a `Dictionary<string, string>` argument that maps tenant ids to a named partition suffix. This might
Expand All @@ -451,6 +480,31 @@ Just like with the codegen-ahead model, you may want to tell Marten about all po
upfront so that it is better able to add the partitions for each tenant id as needed.
:::

To exempt document types from having partitioned tables, such as for tables you expect to be so small that there's no value and maybe
even harm by partitioning, you can use either an attribute on the document type:

<!-- snippet: sample_using_DoNotPartitionAttribute -->
<a id='snippet-sample_using_donotpartitionattribute'></a>
```cs
[DoNotPartition]
public class DocThatShouldBeExempted1
{
public Guid Id { get; set; }
}
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L184-L192' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_using_donotpartitionattribute' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

or exempt a single document type through the fluent interface:

<!-- snippet: sample_exempt_from_partitioning_through_fluent_interface -->
<a id='snippet-sample_exempt_from_partitioning_through_fluent_interface'></a>
```cs
opts.Schema.For<DocThatShouldBeExempted2>().DoNotPartition();
```
<sup><a href='https://github.com/JasperFx/marten/blob/master/src/MultiTenancyTests/marten_managed_tenant_id_partitioning.cs#L169-L173' title='Snippet source file'>snippet source</a> | <a href='#snippet-sample_exempt_from_partitioning_through_fluent_interface' title='Start of snippet'>anchor</a></sup>
<!-- endSnippet -->

## Implementation Details

At the moment, Marten implements two modes of tenancy, namely single tenancy and conjoined multi-tenancy.
Expand Down
Loading
Loading