feat(plugin-search): added support for reindexing collections on demand

[akhrarovsaid]

What?

This PR aims to add reindexing capabilities to plugin-search to allow users to reindex entire searchable collections on demand.

Why?

As it stands, end users must either perform document reindexing manually one-by-one or via bulk operations. Both of these approaches are undesirable because they result in new versions being published on existing documents. Consider the case when plugin-search is only added after the project has started and documents have been added to existing collections. It would be nice if users could simply click a button, choose the searchable collections to reindex, and have the custom endpoint handle the rest.

How?

This PR adds on to the existing plugin configuration, creating a custom endpoint and a custom beforeListTable component in the form of a popup button. Upon clicking the button, a dropdown/popup is opened with options to select which collection to reindex, as well as a useful All Collections option to run reindexing on all configured search collections. It also adds a reindexBatchSize option in the config to allow users to specify in what quantity to batch documents to sync with search.

Big shoutout to @paulpopus & @r1tsuu for the triple-A level support on this one!

Fixes #8902

See it in action:

chrome_2024-11-12_12-18-13.mp4

Notes:

• Traditionally these kinds of long-running tasks would be better suited for a job. However, given how many users enjoy deploying to serverless environments, it would be problematic to offer this feature exclusive to jobs queues. I thought a significant amount about this and decided it would be best to ship the feature as-is with the intention of creating an opt-in method to use job queues in the future if/when this gets merged.
• In my testing, the collection description somehow started to appear in the document views after the on-demand RSC merge. I haven't reproduced this, but this PR has an example of that problem. Super strange.

[JarrodMFlesch]

@akhrarovsaid this looks great! We were wondering if you could modify this PR and place the reindex button in the actions slot for the list view instead of the beforeTable slot?

[akhrarovsaid]

Hey @JarrodMFlesch,

Thank you! So that was actually my first thought too - it fits more naturally there. However, in my testing there was an issue with the actual dropdown/popup that shows up. I think due to overflow hidden somewhere higher in the dom, it cuts the popup off. I'll try again and report my findings, maybe somethings changed since then.

[akhrarovsaid]

@JarrodMFlesch

There's something about the actions wrapper that does not agree here.

Search-Results---Payload.webm

Here's what I've tried:

• Moving component as is to actions slot
• Moving component as is but with relative wrapper and high z-index (10-999)
• Doing the same but with absolute wrapper and high z-index
• Same with fixed
• Tried all of the above but moved styles directly to PopupList.ButtonGroup via classNames
• Enforcing opacity 1 with and without !important
• All of the above with isolation: isolate

It's strange, I'm doing exactly what the localizer is doing except the localizer isn't wrapped in the actions wrapper, so I'm thinking it must be something conflicting from higher up the dom. I can see the correct active class being applied in devtools on interaction though.

[DanRibbens]

Is this ready for review again?

[akhrarovsaid]

@DanRibbens Yep, good to go

[github-actions]

🚀 This is included in version v3.2.0

[PP-Tom]

Thanks for the PR, it's amazing and extremely useful.

I'm having issues with large datasets, if one item fails the entire reindexing fails in this case with { txnNumber: 3 }, with larger datasets (1,200 documents), it seems like the transaction is timing out?

[18:22:55] ERROR: Error finding search document.
    err: {
      "type": "MongoExpiredSessionError",
      "message": "Cannot use a session that has ended",

The other error I seem to be getting is

[18:22:54] ERROR: Error finding search document.
    err: {
      "type": "MongoServerError",
...
"errmsg": "Transaction with { txnNumber: 1 } has been aborted.",
"code": 251,
"codeName": "NoSuchTransaction",

I personally feel like there is some underlying issues with transactions that might not be related to this PR. But it is preventing it from functioning correctly.