Skip to content

Commit

Permalink
feat(plugin-search): added support for reindexing collections on dema…
Browse files Browse the repository at this point in the history
…nd (#9391)

### What?
This PR aims to add reindexing capabilities to `plugin-search` to allow
users to reindex entire searchable collections on demand.

### Why?
As it stands, end users must either perform document reindexing manually
one-by-one or via bulk operations. Both of these approaches are
undesirable because they result in new versions being published on
existing documents. Consider the case when `plugin-search` is only added
_after_ the project has started and documents have been added to
existing collections. It would be nice if users could simply click a
button, choose the searchable collections to reindex, and have the
custom endpoint handle the rest.

### How?
This PR adds on to the existing plugin configuration, creating a custom
endpoint and a custom `beforeListTable` component in the form of a popup
button. Upon clicking the button, a dropdown/popup is opened with
options to select which collection to reindex, as well as a useful `All
Collections` option to run reindexing on all configured search
collections. It also adds a `reindexBatchSize` option in the config to
allow users to specify in what quantity to batch documents to sync with
search.

Big shoutout to @paulpopus & @r1tsuu for the triple-A level support on
this one!

Fixes #8902 

See it in action:


https://github.com/user-attachments/assets/ee8dd68c-ea89-49cd-adc3-151973eea28b

Notes:
- Traditionally these kinds of long-running tasks would be better suited
for a job. However, given how many users enjoy deploying to serverless
environments, it would be problematic to offer this feature exclusive to
jobs queues. I thought a significant amount about this and decided it
would be best to ship the feature as-is with the intention of creating
an opt-in method to use job queues in the future if/when this gets
merged.
- In my testing, the collection description somehow started to appear in
the document views after the on-demand RSC merge. I haven't reproduced
this, but this PR has an example of that problem. Super strange.

---------

Co-authored-by: Sasha <64744993+r1tsuu@users.noreply.github.com>
Co-authored-by: Paul Popus <paul@nouance.io>
  • Loading branch information
3 people authored Nov 26, 2024
1 parent bffd98f commit defa13e
Show file tree
Hide file tree
Showing 55 changed files with 1,424 additions and 196 deletions.
1 change: 1 addition & 0 deletions packages/plugin-search/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@
"test": "echo \"Error: no tests specified\""
},
"dependencies": {
"@payloadcms/next": "workspace:*",
"@payloadcms/ui": "workspace:*"
},
"devDependencies": {
Expand Down
183 changes: 4 additions & 179 deletions packages/plugin-search/src/Search/hooks/syncWithSearch.ts
Original file line number Diff line number Diff line change
@@ -1,182 +1,7 @@
import type { DocToSync, SyncWithSearch } from '../../types.js'
import type { SyncWithSearch } from '../../types.js'

export const syncWithSearch: SyncWithSearch = async (args) => {
const {
collection,
doc,
operation,
pluginConfig,
req: { payload },
req,
} = args
import { syncDocAsSearchIndex } from '../../utilities/syncDocAsSearchIndex.js'

const { id, _status: status, title } = doc || {}

const { beforeSync, defaultPriorities, deleteDrafts, searchOverrides, syncDrafts } = pluginConfig

const searchSlug = searchOverrides?.slug || 'search'

let dataToSave: DocToSync = {
doc: {
relationTo: collection,
value: id,
},
title,
}

if (typeof beforeSync === 'function') {
let docToSyncWith = doc
if (payload.config?.localization) {
docToSyncWith = await payload.findByID({
id,
collection,
locale: req.locale,
req,
})
}
dataToSave = await beforeSync({
originalDoc: docToSyncWith,
payload,
req,
searchDoc: dataToSave,
})
}

let defaultPriority = 0
if (defaultPriorities) {
const { [collection]: priority } = defaultPriorities

if (typeof priority === 'function') {
try {
defaultPriority = await priority(doc)
} catch (err: unknown) {
payload.logger.error(err)
payload.logger.error(
`Error gathering default priority for ${searchSlug} documents related to ${collection}`,
)
}
} else {
defaultPriority = priority
}
}

const doSync = syncDrafts || (!syncDrafts && status !== 'draft')

try {
if (operation === 'create') {
if (doSync) {
await payload.create({
collection: searchSlug,
data: {
...dataToSave,
priority: defaultPriority,
},
locale: req.locale,
req,
})
}
}

if (operation === 'update') {
try {
// find the correct doc to sync with
const searchDocQuery = await payload.find({
collection: searchSlug,
depth: 0,
locale: req.locale,
req,
where: {
'doc.relationTo': {
equals: collection,
},
'doc.value': {
equals: id,
},
},
})

const docs: Array<{
id: number | string
priority?: number
}> = searchDocQuery?.docs || []

const [foundDoc, ...duplicativeDocs] = docs

// delete all duplicative search docs (docs that reference the same page)
// to ensure the same, out-of-date result does not appear twice (where only syncing the first found doc)
if (duplicativeDocs.length > 0) {
try {
const duplicativeDocIDs = duplicativeDocs.map(({ id }) => id)
await payload.delete({
collection: searchSlug,
req,
where: { id: { in: duplicativeDocIDs } },
})
} catch (err: unknown) {
payload.logger.error({
err,
msg: `Error deleting duplicative ${searchSlug} documents.`,
})
}
}

if (foundDoc) {
const { id: searchDocID } = foundDoc

if (doSync) {
// update the doc normally
try {
await payload.update({
id: searchDocID,
collection: searchSlug,
data: {
...dataToSave,
priority: foundDoc.priority || defaultPriority,
},
locale: req.locale,
req,
})
} catch (err: unknown) {
payload.logger.error({ err, msg: `Error updating ${searchSlug} document.` })
}
}
if (deleteDrafts && status === 'draft') {
// do not include draft docs in search results, so delete the record
try {
await payload.delete({
id: searchDocID,
collection: searchSlug,
req,
})
} catch (err: unknown) {
payload.logger.error({ err, msg: `Error deleting ${searchSlug} document.` })
}
}
} else if (doSync) {
try {
await payload.create({
collection: searchSlug,
data: {
...dataToSave,
priority: defaultPriority,
},
locale: req.locale,
req,
})
} catch (err: unknown) {
payload.logger.error({ err, msg: `Error creating ${searchSlug} document.` })
}
}
} catch (err: unknown) {
payload.logger.error({ err, msg: `Error finding ${searchSlug} document.` })
}
}
} catch (err: unknown) {
payload.logger.error({
err,
msg: `Error syncing ${searchSlug} document related to ${collection} with id: '${id}'.`,
})
}

return doc
export const syncWithSearch: SyncWithSearch = (args) => {
return syncDocAsSearchIndex(args)
}
40 changes: 36 additions & 4 deletions packages/plugin-search/src/Search/index.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,17 @@
import type { CollectionConfig, Field } from 'payload'

import type { SearchPluginConfig } from '../types.js'
import type { SearchPluginConfigWithLocales } from '../types.js'

import { generateReindexHandler } from '../utilities/generateReindexHandler.js'

// all settings can be overridden by the config
export const generateSearchCollection = (pluginConfig: SearchPluginConfig): CollectionConfig => {
export const generateSearchCollection = (
pluginConfig: SearchPluginConfigWithLocales,
): CollectionConfig => {
const searchSlug = pluginConfig?.searchOverrides?.slug || 'search'
const searchCollections = pluginConfig?.collections || []
const collectionLabels = pluginConfig?.labels

const defaultFields: Field[] = [
{
name: 'title',
Expand All @@ -29,7 +37,7 @@ export const generateSearchCollection = (pluginConfig: SearchPluginConfig): Coll
},
index: true,
maxDepth: 0,
relationTo: pluginConfig?.collections || [],
relationTo: searchCollections,
required: true,
},
{
Expand All @@ -48,20 +56,44 @@ export const generateSearchCollection = (pluginConfig: SearchPluginConfig): Coll

const newConfig: CollectionConfig = {
...(pluginConfig?.searchOverrides || {}),
slug: pluginConfig?.searchOverrides?.slug || 'search',
slug: searchSlug,
access: {
create: (): boolean => false,
read: (): boolean => true,
...(pluginConfig?.searchOverrides?.access || {}),
},
admin: {
components: {
views: {
list: {
actions: [
{
path: '@payloadcms/plugin-search/client#ReindexButton',
serverProps: {
collectionLabels,
searchCollections,
searchSlug,
},
},
],
},
},
},
defaultColumns: ['title'],
description:
'This is a collection of automatically created search results. These results are used by the global site search and will be updated automatically as documents in the CMS are created or updated.',
enableRichTextRelationship: false,
useAsTitle: 'title',
...(pluginConfig?.searchOverrides?.admin || {}),
},
endpoints: [
...(pluginConfig?.searchOverrides?.endpoints || []),
{
handler: generateReindexHandler(pluginConfig),
method: 'post',
path: '/reindex',
},
],
fields:
pluginConfig?.searchOverrides?.fields &&
typeof pluginConfig?.searchOverrides?.fields === 'function'
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import { ChevronIcon, Pill, useTranslation } from '@payloadcms/ui'

export const ReindexButtonLabel = () => {
const {
i18n: { t },
} = useTranslation()
return (
<Pill className="pill--has-action" icon={<ChevronIcon />} pillStyle="light">
{t('general:reindex')}
</Pill>
)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
@import '../../../../../../ui/src/scss/styles.scss';

@layer payload-default {
.reindex-confirm-modal {
@include blur-bg;
display: flex;
align-items: center;
justify-content: center;
height: 100%;

&__wrapper {
z-index: 1;
position: relative;
display: flex;
flex-direction: column;
gap: base(2);
padding: base(2);
}

&__content {
display: flex;
flex-direction: column;
gap: base(1);

> * {
margin: 0;
}
}

&__controls {
display: flex;
gap: base(0.4);

.btn {
margin: 0;
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { Button, Modal, useTranslation } from '@payloadcms/ui'

import './index.scss'

type Props = {
description: string
onCancel: () => void
onConfirm: () => void
slug: string
title: string
}

const baseClass = 'reindex-confirm-modal'

export const ReindexConfirmModal = ({ slug, description, onCancel, onConfirm, title }: Props) => {
const {
i18n: { t },
} = useTranslation()
return (
<Modal className={baseClass} slug={slug}>
<div className={`${baseClass}__wrapper`}>
<div className={`${baseClass}__content`}>
<h1>{title}</h1>
<p>{description}</p>
</div>
<div className={`${baseClass}__controls`}>
<Button buttonStyle="secondary" onClick={onCancel} size="large">
{t('general:cancel')}
</Button>
<Button onClick={onConfirm} size="large">
{t('general:confirm')}
</Button>
</div>
</div>
</Modal>
)
}
Loading

0 comments on commit defa13e

Please sign in to comment.