Skip to content

Commit

Permalink
V15: Hybrid Caching (#16938)
Browse files Browse the repository at this point in the history
* Update to dotnet 9 and update nuget packages

* Update umbraco code version

* Update Directory.Build.props

Co-authored-by: Elitsa Marinovska <21998037+elit0451@users.noreply.github.com>

* Include preview version in pipeline

* update template projects

* update global json with specific version

* Update version.json to v15

* Rename TrimStart and TrimEnd to string specific

* Rename to Exact

* Update global.json

Co-authored-by: Ronald Barendse <ronald@barend.se>

* Remove includePreviewVersion

* Rename to trim exact

* Add new Hybridcache project

* Add tests

* Start implementing PublishedContent.cs

* Implement repository for content

* Refactor to use async everywhere

* Add cache refresher

* make public as needed for serialization

* Use content type cache to get content type out

* Refactor to use ContentCacheNode model, that goes in the memory cache

* Remove content node kit as its not needed

* Implement tests for ensuring caching

* Implement better asserts

* Implement published property

* Refactor to use mapping

* Rename to document tests

* Update to test properties

* Create more tests

* Refactor mock tests into own file

* Update property test

* Fix published version of content

* Change default cache level to elements

* Refactor to always have draft

* Refactor to not use PublishedModelFactory

* Added tests

* Added and updated tests

* Fixed tests

* Don't return empty object with id

* More tests

* Added key

* Another key

* Refactor CacheService to be responsible for using the hybrid cache

* Use notification handler to remove deleted content from cache

* Add more tests for missing functions

* Implement missing methods

* Remove HasContent as it pertains to routing

* Fik up test

* formatting

* refactor variable names

* Implement variant tests

* Map all the published content properties

* Get item out of cache first, to assert updated

* Implement member cache

* Add member test

* Implement media cache

* Implement property tests for media tests

* Refactor tests to use extension method

* Add more media tests

* Refactor properties to no longer have element caching

* Don't use property cache level

* Start implementing seeding

* Only seed when main

* Add Immutable for performance

* Implement permanent seeding of content

* Implement cache settings

* Implement tests for seeding

* Update package version

* start refactoring nurepo

* Refactor so draft & published nodes are cached individually

* Refactor RefreshContent to take node instead of IContent

* Refactor media to also use cache nodes

* Remove member from repo as it isn't cached

* Refactor media to not include preview, as media has no draft

* create new benchmark project

* POC Integration benchmarks with custom api controllers

* Start implementing content picker tests

* Implement domain cache

* Rework content cache to implement interface

* Start implementing elements cache

* Implement published snapshot service

* Publish snapshot tests

* Use snapshot for elements cache

* Create test proving we don't clear cache when updating content picker

* Clear entire elements cache

* Remove properties from element cache, when content gets updated.

* Rename methods to async

* Refactor to use old cache interfaces instead of new ones

* Remove snapshot, as it is no longer needed

* Fix tests building

* Refactor domaincache to not have snapshots

* Delete benchmarks

* Delete benchmarks

* Add HybridCacheProject to Umbraco

* Add comment to route value transformer

* Implement is draft

* remove snapshot from property

* V15 updated the hybrid caching integration tests to use ContentEditingService (#16947)

* Added builder extension withParentKey

* Created builder with ContentEditingService

* Added usage of the ContentEditingService to SETUP

* Started using ContentEditingService builder in tests

* Updated builder extensions

* Fixed builder

* Clean up

* Clean up, not done

* Added Ids

* Remove entries from cache on delete

* Fix up seeding logic

* Don't register hybrid cache twice

* Change seeded entry options

* Update hybrid cache package

* Fix up published property to work with delivery api again

* Fix dependency injection to work with tests

* Fix naming

* Dont make caches nullable

* Make content node sealed

* Remove path and other unused from content node

* Remove hacky 2 phase ctor

* Refactor to actually set content templates

* Remove umbraco context

* Remove "HasBy" methods

* rename property data

* Delete obsolete legacy stuff

* Add todo for making expiration configurable

* Add todo in UmbracoContext

* Add clarifying comment in content factory

* Remove xml stuff from published property

* Fix according to review

* Make content type cache injectible

* Make content type cache injectible

* Rename to database cache repository

* Rename to document cache

* Add TODO

* Refactor to async

* Rename to async

* Make everything async

* Remove duplicate line from json schema

* Move Hybrid cache project

* Remove leftover file

* Refactor to use keys

* Refactor published content to no longer have content data, as it is on the node itself

* Refactor to member to use proper content node ctor

* Move tests to own folder

* Add immutable objects to property and content data for performance

* Make property data public

* Fix member caching to be singleton

* Obsolete GetContentType

* Remove todo

* Fix naming

* Fix lots of exposed errors due to scope test

* Add final scope tests

* Rename to document cache service

* Rename test files

* Create new doc type tests

* Add ignore to tests

* Start implementing refresh for content type save

* Clear contenttype cache when contenttype is updated

* Fix test

Teh contenttype is not upated unless the property is dirty

* Use init for ContentSourceDto

* Fix get by key in PublishedContentTypeCache

* Remove ContentType from PublishedContentTypeCache when contenttype is deleted

* Update to preview 7

* Fix versions

* Increase timeout for sqlite integration tests

* Undo timeout increase

* Try and undo init change to ContentSourceDto

* That wasn't it chief

* Try and make DomainAndUrlsTests non NonParallelizable

* Update versions

* Only run cache tests on linux for now

---------

Co-authored-by: Elitsa Marinovska <21998037+elit0451@users.noreply.github.com>
Co-authored-by: Ronald Barendse <ronald@barend.se>
Co-authored-by: Andreas Zerbst <andr317c@live.dk>
Co-authored-by: Sven Geusens <sge@umbraco.dk>
Co-authored-by: Andreas Zerbst <73799582+andr317c@users.noreply.github.com>
Co-authored-by: nikolajlauridsen <nikolajlauridsen@protonmail.ch>
  • Loading branch information
7 people authored Sep 9, 2024
1 parent dcd6f1f commit 2704d4a
Show file tree
Hide file tree
Showing 102 changed files with 5,881 additions and 132 deletions.
42 changes: 22 additions & 20 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -12,27 +12,29 @@
</ItemGroup>
<!-- Microsoft packages -->
<ItemGroup>
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="9.0.0-preview.5.24306.11" />
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="9.0.0-preview.7.24406.2" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="4.10.0" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="9.0.0-preview.5.24306.3" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Sqlite" Version="9.0.0-preview.5.24306.3" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="9.0.0-preview.5.24306.3" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="9.0.0-preview.7.24405.3" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Sqlite" Version="9.0.0-preview.7.24405.3" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="9.0.0-preview.7.24405.3" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="8.0.6" />
<PackageVersion Include="Microsoft.Extensions.Caching.Abstractions" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Caching.Memory" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.FileProviders.Embedded" Version="9.0.0-preview.5.24306.11" />
<PackageVersion Include="Microsoft.Extensions.FileProviders.Physical" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Http" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Identity.Core" Version="9.0.0-preview.5.24306.11" />
<PackageVersion Include="Microsoft.Extensions.Identity.Stores" Version="9.0.0-preview.5.24306.11" />
<PackageVersion Include="Microsoft.Extensions.Logging" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Options" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Options.DataAnnotations" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="Microsoft.Extensions.Caching.Abstractions" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Caching.Memory" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Caching.SqlServer" Version="9.0.0-preview.7.24406.2" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.FileProviders.Embedded" Version="9.0.0-preview.7.24406.2" />
<PackageVersion Include="Microsoft.Extensions.FileProviders.Physical" Version="9.0.0-preview.7.24405.7"/>
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Http" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Identity.Core" Version="9.0.0-preview.7.24406.2" />
<PackageVersion Include="Microsoft.Extensions.Identity.Stores" Version="9.0.0-preview.7.24406.2" />
<PackageVersion Include="Microsoft.Extensions.Logging" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Options" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Options.DataAnnotations" Version="9.0.0-preview.7.24405.7" />
<PackageVersion Include="Microsoft.Extensions.Caching.Hybrid" Version="9.0.0-preview.7.24406.2" />
</ItemGroup>
<!-- Umbraco packages -->
<ItemGroup>
Expand Down Expand Up @@ -83,7 +85,7 @@
<!-- Dazinator.Extensions.FileProviders brings in a vulnerable version of System.Net.Http -->
<PackageVersion Include="System.Net.Http" Version="4.3.4" />
<!-- Examine brings in a vulnerable version of System.Security.Cryptography.Xml -->
<PackageVersion Include="System.Security.Cryptography.Xml" Version="9.0.0-preview.5.24306.7" />
<PackageVersion Include="System.Security.Cryptography.Xml" Version="9.0.0-preview.7.24405.7" />
<!-- Both Dazinator.Extensions.FileProviders and MiniProfiler.AspNetCore.Mvc bring in a vulnerable version of System.Text.RegularExpressions -->
<PackageVersion Include="System.Text.RegularExpressions" Version="4.3.1" />
<!-- Both OpenIddict.AspNetCore, Npoco.SqlServer and Microsoft.EntityFrameworkCore.SqlServer bring in a vulnerable version of Microsoft.IdentityModel.JsonWebTokens -->
Expand Down
2 changes: 1 addition & 1 deletion global.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"sdk": {
"version": "9.0.100-preview.5.24307.3",
"version": "9.0.100-preview.7.24407.12",
"rollForward": "latestFeature",
"allowPrerelease": true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ public static IUmbracoBuilder
.AddWebServer()
.AddRecurringBackgroundJobs()
.AddNuCache()
.AddUmbracoHybridCache()
.AddDistributedCache()
.AddCoreNotifications()
.AddExamine()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Umbraco.PublishedCache.HybridCache\Umbraco.PublishedCache.HybridCache.csproj" />
<ProjectReference Include="..\Umbraco.Cms.Api.Common\Umbraco.Cms.Api.Common.csproj" />
<ProjectReference Include="..\Umbraco.Infrastructure\Umbraco.Infrastructure.csproj" />
</ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@ public DomainCacheRefresher(
IPublishedSnapshotService publishedSnapshotService,
IEventAggregator eventAggregator,
ICacheRefresherNotificationFactory factory)
: base(appCaches, serializer, eventAggregator, factory) =>
: base(appCaches, serializer, eventAggregator, factory)
{
_publishedSnapshotService = publishedSnapshotService;
}

#region Json

Expand Down
1 change: 1 addition & 0 deletions src/Umbraco.Core/Constants-Configuration.cs
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ public static class Configuration
public const string ConfigDataTypes = ConfigPrefix + "DataTypes";
public const string ConfigPackageManifests = ConfigPrefix + "PackageManifests";
public const string ConfigWebhook = ConfigPrefix + "Webhook";
public const string ConfigCache = ConfigPrefix + "Cache";

public static class NamedOptions
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
using Umbraco.Cms.Core.Configuration;
using Umbraco.Cms.Core.Configuration.Models;
using Umbraco.Cms.Core.Configuration.Models.Validation;
using Umbraco.Extensions;
using Umbraco.Cms.Core.Models;

namespace Umbraco.Cms.Core.DependencyInjection;

Expand Down Expand Up @@ -85,7 +85,8 @@ public static IUmbracoBuilder AddConfiguration(this IUmbracoBuilder builder)
.AddUmbracoOptions<ContentDashboardSettings>()
.AddUmbracoOptions<HelpPageSettings>()
.AddUmbracoOptions<DataTypesSettings>()
.AddUmbracoOptions<WebhookSettings>();
.AddUmbracoOptions<WebhookSettings>()
.AddUmbracoOptions<CacheSettings>();

// Configure connection string and ensure it's updated when the configuration changes
builder.Services.AddSingleton<IConfigureOptions<ConnectionStrings>, ConfigureConnectionStrings>();
Expand Down
13 changes: 13 additions & 0 deletions src/Umbraco.Core/Models/CacheSettings.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
using Umbraco.Cms.Core.Configuration.Models;

namespace Umbraco.Cms.Core.Models;

[UmbracoOptions(Constants.Configuration.ConfigCache)]
public class CacheSettings
{
/// <summary>
/// Gets or sets a value for the collection of content type ids to always have in the cache.
/// </summary>
public List<Guid> ContentTypeKeys { get; set; } =
new();
}
22 changes: 22 additions & 0 deletions src/Umbraco.Core/Models/PublishedContent/IPublishedMember.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
namespace Umbraco.Cms.Core.Models.PublishedContent;

public interface IPublishedMember : IPublishedContent
{
public string Email { get; }

public string UserName { get; }

public string? Comments { get; }

public bool IsApproved { get; }

public bool IsLockedOut { get; }

public DateTime? LastLockoutDate { get; }

public DateTime CreationDate { get; }

public DateTime? LastLoginDate { get; }

public DateTime? LastPasswordChangedDate { get; }
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ public interface IPublishedMemberCache
/// </summary>
/// <param name="member"></param>
/// <returns></returns>
IPublishedContent? Get(IMember member);
IPublishedMember? Get(IMember member);

/// <summary>
/// Gets a content type identified by its unique identifier.
Expand All @@ -26,4 +26,12 @@ public interface IPublishedMemberCache
/// <returns>The content type, or null.</returns>
/// <remarks>The alias is case-insensitive.</remarks>
IPublishedContentType GetContentType(string alias);

/// <summary>
/// Get an <see cref="IPublishedContent" /> from an <see cref="IMember" />
/// </summary>
/// <param name="key">The key of the member to fetch</param>
/// <param name="preview">Will fetch draft if this is set to true</param>
/// <returns></returns>
Task<IPublishedMember?> GetAsync(IMember member);
}
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ private void InitializeLocked()

var deliveryApiPropertyValueConverter = _converter as IDeliveryApiPropertyValueConverter;

_cacheLevel = _converter?.GetPropertyCacheLevel(this) ?? PropertyCacheLevel.Snapshot;
_cacheLevel = _converter?.GetPropertyCacheLevel(this) ?? PropertyCacheLevel.Elements;
_deliveryApiCacheLevel = deliveryApiPropertyValueConverter?.GetDeliveryApiPropertyCacheLevel(this) ?? _cacheLevel;
_deliveryApiCacheLevelForExpansion = deliveryApiPropertyValueConverter?.GetDeliveryApiPropertyCacheLevelForExpansion(this) ?? _cacheLevel;
_modelClrType = _converter?.GetPropertyValueType(this) ?? typeof(object);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ public IEnumerable<UmbracoEntityReference> GetReferences(object? value)
// starting in v14 the passed in value is always a guid, we store it as a document Udi string. Else it's an invalid value
public override object? FromEditor(ContentPropertyData editorValue, object? currentValue) =>
editorValue.Value is not null
&& Guid.TryParse(editorValue.Value as string, out Guid guidValue)
&& Guid.TryParse(editorValue.Value.ToString(), out Guid guidValue)
? GuidUdi.Create(Constants.UdiEntityType.Document, guidValue).ToString()
: null;

Expand Down
1 change: 1 addition & 0 deletions src/Umbraco.Core/PropertyEditors/PropertyCacheLevel.cs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ public enum PropertyCacheLevel
/// In most cases, a snapshot is created per request, and therefore this is
/// equivalent to cache the value for the duration of the request.
/// </remarks>
[Obsolete("Caching no longer supports snapshotting")]
Snapshot = 3,

/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ public class ContentPickerValueConverter : PropertyValueConverterBase, IDelivery
Constants.Conventions.Content.Redirect.ToLower(CultureInfo.InvariantCulture),
};

private readonly IPublishedSnapshotAccessor _publishedSnapshotAccessor;
private readonly IPublishedContentCache _publishedContentCache;
private readonly IApiContentBuilder _apiContentBuilder;

public ContentPickerValueConverter(
IPublishedSnapshotAccessor publishedSnapshotAccessor,
IPublishedContentCache publishedContentCache,
IApiContentBuilder apiContentBuilder)
{
_publishedSnapshotAccessor = publishedSnapshotAccessor;
_publishedContentCache = publishedContentCache;
_apiContentBuilder = apiContentBuilder;
}

Expand Down Expand Up @@ -105,10 +105,9 @@ public override PropertyCacheLevel GetPropertyCacheLevel(IPublishedPropertyType
PropertiesToExclude.Contains(propertyType.Alias.ToLower(CultureInfo.InvariantCulture))) == false)
{
IPublishedContent? content;
IPublishedSnapshot publishedSnapshot = _publishedSnapshotAccessor.GetRequiredPublishedSnapshot();
if (inter is int id)
{
content = publishedSnapshot.Content?.GetById(id);
content = _publishedContentCache.GetById(id);
if (content != null)
{
return content;
Expand All @@ -121,7 +120,7 @@ public override PropertyCacheLevel GetPropertyCacheLevel(IPublishedPropertyType
return null;
}

content = publishedSnapshot.Content?.GetById(udi.Guid);
content = _publishedContentCache.GetById(udi.Guid);
if (content != null && content.ContentType.ItemType == PublishedItemType.Content)
{
return content;
Expand Down
37 changes: 37 additions & 0 deletions src/Umbraco.Core/PublishedCache/ICacheManager.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
using Umbraco.Cms.Core.Cache;

namespace Umbraco.Cms.Core.PublishedCache;

public interface ICacheManager
{
/// <summary>
/// Gets the <see cref="IPublishedContentCache" />.
/// </summary>
IPublishedContentCache Content { get; }

/// <summary>
/// Gets the <see cref="IPublishedMediaCache" />.
/// </summary>
IPublishedMediaCache Media { get; }

/// <summary>
/// Gets the <see cref="IPublishedMemberCache" />.
/// </summary>
IPublishedMemberCache Members { get; }

/// <summary>
/// Gets the <see cref="IDomainCache" />.
/// </summary>
IDomainCache Domains { get; }

/// <summary>
/// Gets the elements-level cache.
/// </summary>
/// <remarks>
/// <para>
/// The elements-level cache is shared by all snapshots relying on the same elements,
/// ie all snapshots built on top of unchanging content / media / etc.
/// </para>
/// </remarks>
IAppCache ElementsCache { get; }
}
31 changes: 31 additions & 0 deletions src/Umbraco.Core/PublishedCache/IDomainCacheService.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
using Umbraco.Cms.Core.Cache;
using Umbraco.Cms.Core.Routing;

namespace Umbraco.Cms.Core.PublishedCache;

public interface IDomainCacheService
{
/// <summary>
/// Gets all <see cref="Domain" /> in the current domain cache, including any domains that may be referenced by
/// documents that are no longer published.
/// </summary>
/// <param name="includeWildcards"></param>
/// <returns></returns>
IEnumerable<Domain> GetAll(bool includeWildcards);

/// <summary>
/// Gets all assigned <see cref="Domain" /> for specified document, even if it is not published.
/// </summary>
/// <param name="documentId">The document identifier.</param>
/// <param name="includeWildcards">A value indicating whether to consider wildcard domains.</param>
IEnumerable<Domain> GetAssigned(int documentId, bool includeWildcards = false);

/// <summary>
/// Determines whether a document has domains.
/// </summary>
/// <param name="documentId">The document identifier.</param>
/// <param name="includeWildcards">A value indicating whether to consider wildcard domains.</param>
bool HasAssigned(int documentId, bool includeWildcards = false);

void Refresh(DomainCacheRefresher.JsonPayload[] payloads);
}
Loading

0 comments on commit 2704d4a

Please sign in to comment.