Deploy Failure on GIT Push: "Recursive locks not allowed"

[hfloyd]

When pushing updates from local to Cloud environment, I see the message "The last umbraco data extraction failed on this environment " in the portal.

The details:

Error while extracting Umbraco data

This environment encountered an error while trying to extract Umbraco data, during a previous deployment

The environment contains a deploy-failure marker file, which indicates that extraction of Umbraco data, faulted on a previous deployment. To move forward, ensure that all templates, document types etc. are properly saved, and that you dont have duplicated .courier files in the data folder.

You can find more help by checking the offical troubleshooting documentation or by contacting support

Error Message
Recursive locks not allowed

{
  "Date": "2022-04-20T21:07:09.5368111+00:00",
  "Message": "Recursive locks not allowed",
  "Ex": {
    "ClassName": "System.InvalidOperationException",
    "Message": "Recursive locks not allowed",
    "Data": null,
    "InnerException": null,
    "HelpURL": null,
    "StackTraceString": "   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.Lock(WriteLockInfo lockInfo, Boolean forceGen)\r\n   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.ScopedWriteLock..ctor(ContentStore store, Boolean scoped)\r\n   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.<GetScopedWriteLock>b__27_0(Boolean scoped)\r\n   at Umbraco.Cms.Core.Scoping.ScopeContextualBase.Get[T](IScopeProvider scopeProvider, String key, Func`2 ctor)\r\n   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.GetScopedWriteLock(IScopeProvider scopeProvider)\r\n   at Umbraco.Cms.Infrastructure.PublishedCache.PublishedSnapshotService.Notify(JsonPayload[] payloads)\r\n   at Umbraco.Cms.Core.Cache.DataTypeCacheRefresher.<>c__DisplayClass9_0.<Refresh>b__0()\r\n   at Umbraco.Extensions.PublishedModelFactoryExtensions.WithSafeLiveFactoryReset(IPublishedModelFactory factory, Action action)\r\n   at Umbraco.Cms.Core.Cache.DataTypeCacheRefresher.Refresh(JsonPayload[] payloads)\r\n   at Umbraco.Cms.Infrastructure.Sync.ServerMessengerBase.DeliverLocal[TPayload](ICacheRefresher refresher, TPayload[] payload)\r\n   at Umbraco.Cms.Infrastructure.Sync.ServerMessengerBase.Deliver[TPayload](ICacheRefresher refresher, TPayload[] payload)\r\n   at Umbraco.Cms.Infrastructure.Sync.ServerMessengerBase.QueueRefresh[TPayload](ICacheRefresher refresher, TPayload[] payload)\r\n   at Umbraco.Cms.Core.Cache.DistributedCache.RefreshByPayload[TPayload](Guid refresherGuid, TPayload[] payload)\r\n   at Umbraco.Extensions.DistributedCacheExtensions.RefreshDataTypeCache(DistributedCache dc, IDataType dataType)\r\n   at Umbraco.Cms.Core.Cache.DistributedCacheBinder.Handle(DataTypeSavedNotification notification)\r\n   at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[T0,T1](CallSite site, T0 arg0, T1 arg1)\r\n   at Umbraco.Deploy.Infrastructure.DeployScopedNotificationPublisher.HandleNotifications()\r\n   at Umbraco.Deploy.Infrastructure.DeployScopedNotificationPublisher.ScopeExit(Boolean completed)\r\n   at Umbraco.Cms.Core.Scoping.Scope.<>c__DisplayClass106_0.<RobustExit>b__1()\r\n   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)\r\n   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)\r\n   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Action[] actions)\r\n   at Umbraco.Cms.Core.Scoping.Scope.RobustExit(Boolean completed, Boolean onException)\r\n   at Umbraco.Cms.Core.Scoping.Scope.DisposeLastScope()\r\n   at Umbraco.Cms.Core.Scoping.Scope.Dispose()\r\n   at Umbraco.Deploy.Infrastructure.Work.WorkItems.DiskReadWorkItem.ExecuteAsyncSub(IWorkContext context, List`1 resume, CancellationToken token)\r\n   at Umbraco.Deploy.Infrastructure.Work.WorkItems.DiskReadWorkItem.ExecuteAsync(IWorkContext context, CancellationToken token)",
    "RemoteStackTraceString": null,
    "RemoteStackIndex": 0,
    "ExceptionMethod": null,
    "HResult": -2146233079,
    "Source": "Umbraco.PublishedCache.NuCache",
    "WatsonBuckets": null
  },
  "Sender": "Umbraco.Deploy.Infrastructure.Work.WorkItems.DiskReadWorkItem"
}

Versions

Umbraco 9.4.3
Forms 9.3.0
Deploy 9.3.1
Umbraco Id

[AndyButland]

I've had a little walk through that stack trace this morning - can't see an obvious reason why this would get triggered, but then again it's happening deep in the the weeds of Umbraco's caching mechanism, which is complex to say the least. It's probably not strictly an issue with Deploy, but rather something not working correctly with this cache refreshing logic. As far as I can see from the stack trace, it's being triggered by data type changes.

It might help a bit to know if this error is consistent? I.e. if you go into the back-office and trigger an extraction, do you still get the error? I'm asking as wondering if the issue is that something else was going on on the environment - e.g. a back-office save - that meant the lock couldn't be acquired. And if it is consistently happening, sounds like something we might be able to replicate with a copy of your .uda files and database. I've tried triggering it myself making multiple data type changes, but can't see it occuring.

[hfloyd]

Thanks for looking into this. Usually I can get the extraction to work via the back office deploy dashboard after a few site restarts.

The only other thing I could say about this site is that since it is in active development and not live right now, there is only one cloud environment, the live one, so I am going from local to live. I'm not sure if that would make any difference to anything, but figured I'd mention it.

Right now there is really only myself and a single Content Editor accessing the back-office. I don't think the Content Editor was working on the site at the time of the issue...

[hfloyd]

Thanks for looking into this. Usually I can get the extraction to work via the back office deploy dashboard after a few site restarts. I haven't gotten this error when saving items in the back-office - however, I work primarily on local. The only other thing i could say about this site is that since it is in active development and not live right now, there is only one cloud environment, the live one, so i am going from local to live. I'm not sure if that would make any difference to anything, but figured I'd mention it. ~Heather

…

Sent from my phone

________________________________ From: Andy Butland ***@***.***> Sent: Thursday, April 21, 2022 3:38:09 AM To: umbraco/Umbraco.Deploy.Issues ***@***.***> Cc: Heather Floyd ***@***.***>; Author ***@***.***> Subject: Re: [umbraco/Umbraco.Deploy.Issues] Deploy Failure on GIT Push: "Recursive locks not allowed" (Issue #108) I've had a little walk through that stack trace this morning - can't see an obvious reason why this would get triggered, but then again it's happening deep in the the weeds of Umbraco's caching mechanism, which is complex to say the least. It's probably not strictly an issue with Deploy, but rather something not working correctly with this cache refreshing logic. As far as I can see from the stack trace, it's being triggered by data type changes. It might help a bit to know if this error is consistent? I.e. if you go into the back-office and trigger an extraction, do you still get the error? I'm asking as wondering if the issue is that something else was going on on the environment - e.g. a back-office save - that meant the lock couldn't be acquired. And if it is consistently happening, sounds like something we might be able to replicate with a copy of your .uda files and database. I've tried triggering it myself making multiple data type changes, but can't see it occuring. — Reply to this email directly, view it on GitHub<#108 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAJRZ4KKAWNMRUDKYB5GULLVGEAWDANCNFSM5T5JAYEQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ronaldbarendse]

Hi Heather! I was looking at our long standing open issues and noticed this is an issue that should be fixed with CMS PR umbraco/Umbraco-CMS#17246 (released in 13.5.2 and 14.3.1+). I'll close this, but please let us know if you're experiencing the same/a similar issue on newer versions...