Skip to content

Commit

Permalink
feat: add includeFilter option for segmented indexing (fixes #33)
Browse files Browse the repository at this point in the history
  • Loading branch information
@wilr
wilr committed Jul 13, 2023
1 parent 3dce6c8 commit 6644249
Show file tree
Hide file tree
Showing 9 changed files with 432 additions and 139 deletions.
233 changes: 189 additions & 44 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Status](http://img.shields.io/travis/wilr/silverstripe-algolia.svg?style=flat-sq

## Maintainer Contact

* Will Rossiter (@wilr) <will@fullscreen.io>
- Will Rossiter (@wilr) <will@fullscreen.io>

## Installation

Expand All @@ -23,7 +23,7 @@ multiple indexes.

:ballot_box_with_check: Integrates into existing versioned workflow.

:ballot_box_with_check: No dependancies on the CMS, supports any DataObject
:ballot_box_with_check: No dependencies on the CMS, supports any DataObject
subclass.

:ballot_box_with_check: Queued job support for offloading operations to Algolia.
Expand Down Expand Up @@ -52,25 +52,26 @@ DataObjects.
First, sign up for Algolia.com account and install this module. Once installed,
Configure the API keys via YAML (environment variables recommended).

*app/_config/algolia.yml*
_app/\_config/algolia.yml_

```yml
---
Name: algolia
After: silverstripe-algolia
---
SilverStripe\Core\Injector\Injector:
Wilr\SilverStripe\Algolia\Service\AlgoliaService:
properties:
adminApiKey: '`ALGOLIA_ADMIN_API_KEY`'
searchApiKey: '`ALGOLIA_SEARCH_API_KEY`'
applicationId: '`ALGOLIA_SEARCH_APP_ID`'
indexes:
IndexName:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
indexSettings:
attributesForFaceting:
- 'filterOnly(objectClassName)'
Wilr\SilverStripe\Algolia\Service\AlgoliaService:
properties:
adminApiKey: "`ALGOLIA_ADMIN_API_KEY`"
searchApiKey: "`ALGOLIA_SEARCH_API_KEY`"
applicationId: "`ALGOLIA_SEARCH_APP_ID`"
indexes:
IndexName:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
indexSettings:
attributesForFaceting:
- "filterOnly(objectClassName)"
```
Once the indexes and API keys are configured, run a `dev/build` to update the
Expand All @@ -92,7 +93,6 @@ ALGOLIA_PREFIX_INDEX_NAME='dev_will'
Or for testing with live data on dev use `ALGOLIA_PREFIX_INDEX_NAME='live'`
### Defining Replica Indexes
If your search form provides a sort option (e.g latest or relevance) then you
Expand All @@ -107,31 +107,31 @@ Name: algolia
After: silverstripe-algolia
---
SilverStripe\Core\Injector\Injector:
Wilr\SilverStripe\Algolia\Service\AlgoliaService:
properties:
adminApiKey: '`ALGOLIA_ADMIN_API_KEY`'
searchApiKey: '`ALGOLIA_SEARCH_API_KEY`'
applicationId: '`ALGOLIA_SEARCH_APP_ID`'
indexes:
IndexName:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
indexSettings:
attributesForFaceting:
- 'filterOnly(ObjectClassName)'
replicas:
- IndexName_Latest
IndexName_Latest:
indexSettings:
ranking:
- 'desc(objectCreated)'
- 'typo'
- 'words'
- 'filters'
- 'proximity'
- 'attribute'
- 'exact'
- 'custom'
Wilr\SilverStripe\Algolia\Service\AlgoliaService:
properties:
adminApiKey: "`ALGOLIA_ADMIN_API_KEY`"
searchApiKey: "`ALGOLIA_SEARCH_API_KEY`"
applicationId: "`ALGOLIA_SEARCH_APP_ID`"
indexes:
IndexName:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
indexSettings:
attributesForFaceting:
- "filterOnly(ObjectClassName)"
replicas:
- IndexName_Latest
IndexName_Latest:
indexSettings:
ranking:
- "desc(objectCreated)"
- "typo"
- "words"
- "filters"
- "proximity"
- "attribute"
- "exact"
- "custom"
```

## Indexing
Expand Down Expand Up @@ -219,7 +219,7 @@ class MyPage extends Page {
}
```

### Customising the indexed relationships
### Customizing the indexed relationships

Out of the box, the default is to push the ID and Title fields of any
relationships (`$has_one`, `$has_many`, `$many_many`) into a field
Expand Down Expand Up @@ -255,7 +255,7 @@ operations. The queuing feature can be disabled via the Config YAML.

```yaml
Wilr\SilverStripe\Algolia\Extensions\AlgoliaObjectExtension:
use_queued_indexing: false
use_queued_indexing: false
```
## Displaying and fetching results
Expand Down Expand Up @@ -316,7 +316,7 @@ it in a `objectForTemplate` field in Algolia. This content is parsed via the
```html
<main>
$ElementalArea
$ElementalArea
<!-- will be indexed via Algolia -->
</main>
```
Expand All @@ -336,3 +336,148 @@ Wilr\SilverStripe\Algolia\Service\AlgoliaPageCrawler:
content_xpath_selector: '//[data-index]'
```

## Subsite support

If you use the Silverstripe Subsite module to run multiple websites you can
handle indexing in a couple ways:

- Use separate indexes per site.
- Use a single index, but add a `SubsiteID` field in Algolia.

The decision to go either way depends on the nature of the websites and how
related they are but separate indexes are highly recommended to prevent leaking
information between websites and mucking up analytics and query suggestions.

### Subsite support with a single index

If subsites are frequently being created then you may choose to prefer a single
index since index names need to be controlled via YAML so any new subsite would
require a code change.

The key to this approach is added `SubsiteID` to the attributes for faceting
and at the query time.

Step 1. Add the field to Algolia

```
SilverStripe\Core\Injector\Injector:
Wilr\SilverStripe\Algolia\Service\AlgoliaService:
properties:
adminApiKey: "`ALGOLIA_ADMIN_API_KEY`"
searchApiKey: "`ALGOLIA_SEARCH_API_KEY`"
applicationId: "`ALGOLIA_SEARCH_APP_ID`"
indexes:
index_main_site:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
indexSettings:
distinct: true
attributeForDistinct: "objectLink"
searchableAttributes:
- objectTitle
- objectContent
- objectLink
- Summary
- objectForTemplate
attributesForFaceting:
- "filterOnly(objectClassName)"
***- "filterOnly(SubsiteID)"***
```

Step 2. Expose the field on `SiteTree` via a DataExtension (make sure to apply the extension)

```
<?php
class SiteTreeExtension extends DataExtension
{
private static $algolia_index_fields = [
'SubsiteID'
];
}
```

Step 3. Filter by the Subsite ID in your results

```php
<?php

use SilverStripe\Core\Injector\Injector;
use Wilr\SilverStripe\Algolia\Service\AlgoliaQuerier;

class PageController extends ContentController
{
public function results()
{
$hitsPerPage = 25;
$paginatedPageNum = floor($this->request->getVar('start') / $hitsPerPage);

$results = Injector::inst()->get(AlgoliaQuerier::class)->fetchResults(
'indexName',
$this->request->getVar('search'), [
'page' => $this->request->getVar('start') ? $paginatedPageNum : 0,
'hitsPerPage' => $hitsPerPage,
'facetFilters' => [
'SubsiteID' => SubsiteState::singleton()->getSubsiteId()
]
]
);

return [
'Title' => 'Search Results',
'Results' => $results
];
}
}
```

### Subsite support with separate indexes

Create multiple indexes in your config and use the `includeFilter` parameter to
filter the records per index.

The `includeFilter` should be in the format `{$Class}`: `{$WhereQuery}` where
the `$WhereQuery` is a basic SQL statement performed by the ORM on the given
class.

```
SilverStripe\Core\Injector\Injector:
Wilr\SilverStripe\Algolia\Service\AlgoliaService:
properties:
adminApiKey: "`ALGOLIA_ADMIN_API_KEY`"
searchApiKey: "`ALGOLIA_SEARCH_API_KEY`"
applicationId: "`ALGOLIA_SEARCH_APP_ID`"
indexes:
index_main_site:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
includeFilter:
"SilverStripe\\CMS\\Model\\SiteTree": "SubsiteID = 0"
indexSettings:
distinct: true
attributeForDistinct: "objectLink"
searchableAttributes:
- objectTitle
- objectContent
- objectLink
- Summary
- objectForTemplate
attributesForFaceting:
- "filterOnly(objectClassName)"
index_subsite_pages:
includeClasses:
- SilverStripe\CMS\Model\SiteTree
includeFilter:
"SilverStripe\\CMS\\Model\\SiteTree": "SubsiteID > 0"
indexSettings:
distinct: true
attributeForDistinct: "objectLink"
searchableAttributes:
- objectTitle
- objectContent
- objectLink
- Summary
- objectForTemplate
attributesForFaceting:
- "filterOnly(objectClassName)"
```
15 changes: 11 additions & 4 deletions src/Extensions/AlgoliaObjectExtension.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ public function updateSettingsFields(FieldList $fields)
$fields->addFieldsToTab(
'Root.Search',
[
ReadonlyField::create('AlgoliaIndexed', _t(__CLASS__.'.LastIndexed', 'Last indexed in Algolia'))
ReadonlyField::create('AlgoliaIndexed', _t(__CLASS__ . '.LastIndexed', 'Last indexed in Algolia'))
->setDescription($this->owner->AlgoliaError),
ReadonlyField::create('AlgoliaUUID', _t(__CLASS__.'.UUID', 'Algolia UUID'))
ReadonlyField::create('AlgoliaUUID', _t(__CLASS__ . '.UUID', 'Algolia UUID'))
]
);
}
Expand All @@ -99,7 +99,9 @@ public function requireDefaultRecords()
$this->ranSync = true;
$algolia = Injector::inst()->create(AlgoliaService::class);

if(!$this->indexEnabled()) return;
if (!$this->indexEnabled()) {
return;
}

try {
$algolia->syncSettings();
Expand Down Expand Up @@ -145,18 +147,23 @@ public function onAfterPublish()
public function markAsRemovedFromAlgoliaIndex()
{
$this->touchAlgoliaIndexedDate(true);

return $this->owner;
}

/**
* Update the AlgoliaIndexed date for this object.
*/
public function touchAlgoliaIndexedDate($isDeleted = false)
{
$newValue = $isDeleted ? 'null' : 'NOW()';
$newValue = $isDeleted ? 'null' : DB::get_conn()->now();

$this->updateAlgoliaFields([
'AlgoliaIndexed' => $newValue,
'AlgoliaUUID' => "'" . $this->owner->AlgoliaUUID . "'",
]);

return $this->owner;
}

/**
Expand Down
15 changes: 8 additions & 7 deletions src/Jobs/AlgoliaReindexAllJob.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,14 @@ public function setup()
// and process simply handles one batch at a time.
foreach ($algoliaService->indexes as $index) {
$classes = (isset($index['includeClasses'])) ? $index['includeClasses'] : null;
$indexFilters = (isset($index['includeFilters'])) ? $index['includeFilters'] : null;

if ($classes) {
foreach ($classes as $candidate) {
$filter = (isset($filters[$candidate])) ? $filters[$candidate] : '';
$count = 0;

foreach ($task->getItems($candidate, $filter)->column('ID') as $id) {
foreach ($task->getItems($candidate, $filter, $indexFilters)->column('ID') as $id) {
$count++;

if (!isset($this->indexData[$candidate])) {
Expand All @@ -83,7 +84,7 @@ public function setup()
$this->totalSteps++;
}

$this->addMessage('Indexing '. $count . ' '. $candidate . ' instances with filters '. $filter);
$this->addMessage('Indexing ' . $count . ' ' . $candidate . ' instances with filters ' . $filter);
}
}
}
Expand Down Expand Up @@ -118,19 +119,19 @@ public function process()

try {
if ($batching) {
if ($task->indexItems($class, '', DataObject::get($class)->filter('ID', $take), false)) {
$this->addMessage('Successfully indexing '. $class . ' ['. implode(', ', $take) . ']');
if ($task->indexItems($class, DataObject::get($class)->filter('ID', $take), false)) {
$this->addMessage('Successfully indexing ' . $class . ' [' . implode(', ', $take) . ']');
} else {
$this->addMessage('Error indexing '. $class . ' ['. implode(', ', $take) . ']');
$this->addMessage('Error indexing ' . $class . ' [' . implode(', ', $take) . ']');
}
} else {
$items = DataObject::get($class)->filter('ID', $take);

foreach ($items as $item) {
if ($task->indexItem($item)) {
$this->addMessage('Successfully indexed '. $class . ' ['. $item->ID . ']');
$this->addMessage('Successfully indexed ' . $class . ' [' . $item->ID . ']');
} else {
$this->addMessage('Error indexing '. $class . ' ['. $item->ID . ']');
$this->addMessage('Error indexing ' . $class . ' [' . $item->ID . ']');
}
}
}
Expand Down
Loading

0 comments on commit 6644249

Please sign in to comment.