Load and save+publish times increased between 12.0.0 and 12.1.2

[marklonquist]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

12.1.2

Bug summary

We have a huge ecom site which uses umbraco for content, and we make heavy use of Block List throughout the site plus have over 20 languages enabled.

On some of our nodes, we see an increase in load-time (GetById) from ~8s to ~60s and save+publish going from ~3m to >10m (times out).

While 3m is also incredible long, it does make sense when you have Block List, within Block List within Block List. Added a video showing the worst node we have.

Specifics

No response

Steps to reproduce

Have a site with 20+ languages, all with content, and deep nesting of Block Lists

recording.mp4

Expected result / actual result

Faster loading and faster saving

---

This item has been added to our backlog AB#33323

[github-actions]

Hi there @marklonquist!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[Zeegaan]

Hey I'm sorry you're experiencing that 😞
However I don't really have a giant site with 20+ languages on hand 😁

I would love if you could recreate this from a clean site, and then perhaps send me the backpack file of your database, that way it would be easy for me to test & reproduce 🤞

Even a database file of current database file would help 😁

[marklonquist]

@Zeegaan is there an email to which I can share the database?

[marklonquist]

@Zeegaan I created a repository with version that can load and save+publish the node we're really struggling with.
I've invited you as a collaborator.

The bacpac is in db_export folder

See the README.md

[marklonquist]

Hi @Zeegaan - you had time to play around with this? :p

[Zeegaan]

@marklonquist Yep just cloned it down, we need the username + password for the backoffice user though 😁

[marklonquist]

Hi @Zeegaan - what email can I send the username/password to?

[Zeegaan]

@marklonquist nge@umbraco.dk

[andr317c]

Hey! I tried looking at your issue. I was able to reproduce the high load times.
As you explained in the readme, I tried going to Configuration > Navigation Container > Burger Menu in the content section, to see how long it took to load, both for versions 12.0.0 and 12.1.2.

• Version 12.0.0 Load time
  • First try: 28 seconds
  • Second try: 27 seconds
• Version 12.1.2 Load time
  • First try: 4:13 minutes
  • Second try: 4:44 minutes

The load times seem to be around 9-10 times slower in version 12.1.2 than it was in 12.0.0.

I was not able to reproduce the slow Save and Publish time because I ran into this error

I will mark this issue as a sprint candidate.

Please let me know what I am missing so I'm able to Save and Publish! 😄

[marklonquist]

@Zeegaan I added the missing property editor - seems to be able to save and publish now

[andr317c]

Hey again, I tried running the project again with the missing property editor. I ran into another issue when trying to Save and Publish, this one:

Save and Publish was running for 23 minutes before I encountered this error. So I can confirm that the Save and Publish times are a lot longer in 12.2.0 than in 12.0.0 for your project

[marklonquist]

has there been any progress on this? :)

[dawoe]

Hi @marklonquist, @Zeegaan, @andr317c

Could be related to this issue (#14936 ). We experiencing a slow back office as well when making use of the block list with a lot of culture and segment variants.

[bielu]

So had debugging session and figure out a lot of slowiness will come from similar issue which was caused by block editor, which would be solved by pr, a lot of of slowiness will come also from PropertyValidationService which will load datatype for each property in each culture 😱, and based on that what i learned from fixing Block editor it has a lot of impact on performance, so going down umbraco loads almost 1.3k duplicated datypes in my specific case! so it would add up up to almost 2s!

[nielslyngsoe]

Hi @Migaroez

Does this PR #15731 fixes this or is there more that can be done — this issue is getting a bit stale so would be nice to give an update on the matter. Thanks in advance.

[marklonquist]

We are on 13.4.0-rc and we still have issues with long save+publish times

[marklonquist]

Does seem like it's way better in Umbraco 15. Just saved 86 languages, each having 2000 blocks (so 172000 total blocks) in 45 seconds.. That's pretty good.

[bielu]

@marklonquist v15 is not nucache based so same issue is not applying as memory cache is differently used. Also I wouldn't use v15 in production tbh.

[marklonquist]

@marklonquist v15 is not nucache based so same issue is not applying as memory cache is differently used. Also I wouldn't use v15 in production tbh.

we have a major assignment in migrating from v13, so there will be plenty of updates to v15 before we are done :P It should mature enough before we are done :D

[bielu]

@marklonquist I would personally wait till 17.3 or 5 but yeah that's not suitable case for most

[marklonquist]

After some more testing, Umbraco is still slow with a lot of localized content using block list.

Sadly, we still migrating to other CMS because of this. 😔

[bielu]

@marklonquist did you try to profile why it is slow? As I am curious as v15 shouldn't share as much this issue as V13 did 🤔

[marklonquist]

With ~90 languages and block lists inside block lists, it just seem to be having issues. No profiling. Just a comparison between our current v13.4 instance and a fresh v15.

Could try to profile it.

[bielu]

I am not that familar with v15 code but you can try #15573 similar steps to mine from v13 :) I had a lot fun learning that memory cache done by umbraco cause delays by over locking memory cache

[marklonquist]

Did some profiling. It seems it's just because of the amount of SQL calls (one per culture)

GET https___localhost_44357_api_testdata (4602.63 ms) - Profiling Results_compressed.pdf

(this was using test data, with 100 rather simple blocks in each culture)

[bielu]

@marklonquist that result seem to be worst than v13 about amount of sql queries 😱, As you already prepared test data can you share it? i might have a look into it as wanted get familiar with v15 code base anyway lol
Can you also confirm if v13 was faster/slower?

[marklonquist]

@bielu the performance between v13 and v15 seems to be the same in terms of this issue :)