Large amounts of "Updating search indexes" jobs when editing content

[mmikkel]

Since updating sites to Craft 3.7 (mostly going straight from 3.5), we're seeing a massive amount of "Updating search indexes" jobs being generated as authors edit content. On larger multi-sites with a lot of Matrix content and/or relations, these jobs can easily run into the thousands in a relatively short amount of time, just from authors working "normally" with creating and editing entries.

I don't believe anything really changed in 3.7 (or 3.6) in regards to how or when Craft actually updates the search indexes. The big difference is that in 3.7, all content editing happens in auto-saving drafts – where before, a lot of the time authors would just edit the current (without creating a draft or jumping into Preview). In other words the behind-the-scenes behaviour was probably the same, but it didn't really manifest as an issue because the total number of element (re)saves resulting from day-to-day content authoring was usually a lot lower.

So, this is sort of an open question, I guess. Is it possible to optimise this somehow? I get that the ideal thing is having search indexes always kept up to date, but with the current implementation Craft is basically creating a number of these (often resource intensive) jobs on every single input change event, which doesn't scale too well.

For sites where this is getting to be a performance and stability issue, I'd personally be completely satisfied if saving a draft didn't trigger a search index update at all (especially for provisional drafts). Craft is doing a ton of work to index these drafts, in most cases only to facilitate searching for them inside the CP – something we could definitely live without for most sites. Perhaps a config setting to disable search indexing for (provisional) drafts could be considered, if it's not currently feasible to optimise this in some other way?

[brandonkelly]

I just realized that this change in 3.6.5 introduced a bug, where Matrix/Neo/Super Table blocks are now getting search keywords indexed even if they belong to a revision. Just fixed that for the next release, which will help somewhat. (af8c200)

I wonder if this could be solved via debouncing, rather than completely eliminating search indexes for drafts. Maybe they only get updated once a minute or so per entry.

In the meantime, I’ve just added a new EVENT_BEFORE_UPDATE_SEARCH_INDEX event to the Elements service for the next release (315e811), which you can use to prevent the search index from getting updated for drafts like so:

use craft\events\ElementEvent;
use craft\helpers\ElementHelper;
use craft\services\Elements;
use yii\base\Event;

Event::on(
    Elements::class,
    Elements::EVENT_BEFORE_UPDATE_SEARCH_INDEX,
    function(ElementEvent $event) {
        if (ElementHelper::isDraft($event->element)) {
            $event->isValid = false;
        }
    }
);

[mmikkel]

Nice, thanks Brandon. In fact we could likely use that event to prevent only duplicate search index jobs from being created, by doing a quick query (we’re usually using the DB driver).

Beyond these changes, debouncing sounds like a good idea.

[brandonkelly]

Craft 3.7.12 is out now with that new event. You also may find #9802 of interest.

[lesaff]

Hi @brandonkelly I am currently on 3.7.13 and still experiencing the same issue. So many failed search indexes and it is slowing down the CP to the point where client couldn't login.

I installed Async Queue plugin to help out with those queues but it did not seem to help at this point.

Any insights would be appreciated.
Cheers
Rudy

[brandonkelly]

@lesaff That seems unrelated, since in your case they are piling up because they are failing. Please create a new issue or write into support@craftcms.com with whatever error message you are seeing on these failed jobs, and can help from there.

[FreekVR]

Just as a followup from my bug report at #10141 , why would Craft even add data from upublished drafts, or autosave drafts, to the search index? Is it used internally in the admin CP search?

It doesn't really make sense to have them searchable from the front-end, making the jobs somewhat redundant if their resulting search index changes aren't used anywhere in the CP.

It looks like even clicking the "create entry" button already dispatches a number of search index jobs, even without making changes.

[FreekVR]

Right - makes sense - but provisional drafts are the ones from autosave right? They're causing the most pain as they're being triggered on pretty much on every keypress now. Would it be possible to not dispatch any search index jobs for those drafts, and only do that once they're converted to a normal one? That would probably save 90% of the index jobs I'm seeing on our high-traffic and field-heavy sites.

[FreekVR]

I believe I was seeing some index jobs for Neo being triggered on create as well, though it's hard to tell as I had a queue with a few index jobs from different draft actions mixed up at that point.

[brandonkelly]

See my prior answer for a way to prevent these jobs from getting created.

[FreekVR]

Thanks, I'll look into that as a workaround. We already disabled autosave on one of our sites as a quickfix, but the issue is still popping up in preview mode, with about 28 jobs being dispatched just by typing a few characters in a single description field.

Just to check -- would your code snippet also work for the drafts created in preview mode?

[brandonkelly]

@FreekVR Live Preview relies on draft autosaving, so that setting is ignored for Live Preview. If you have the above code in place, it should prevent search index jobs even for provisional drafts, yes.