Elasticsearch Pipeline Support

[cbadger-montecitobank]

Having the ability to specify the name of an ingest pipeline to use when indexing content items into Elasticsearch would be helpful. All it would take is an extra index setting that let's you specify a pipeline name, then a one-line code change in ElasticIndexManager.cs -> StoreDocumentsAsync():

  descriptor.Index<Dictionary<string, object>>(op => op
      .Id(document.GetValueOrDefault("ContentItemId").ToString())
      .Pipeline("my_index_pipeline") // <----new line ----
      .Document(document)
      .Index(GetFullIndexName(indexName))
  );

This would allow document manipulation before it gets stored. My understanding is that when documents are ingested via API, there's no other options for running pipelines unless they are specified by the client during an index request.

[Piedone]

That's a good idea, yes. However what exactly would you like to use the pipeline for? A workaround may be possible today.

Is this something you'd like to work on contributing? If you, please open an issue, and let's finalize the implementation details under it.

[cbadger-montecitobank]

Appreciate the response. In my scenario I'd like to add some additional values to each indexed document (data enrichment). This could probably be done by adding additional fields to the corresponding content types, but I'd hate to clutter them up with this type of thing if it can be avoided. Working on this as a code contribution is certainly a possibility. I'll open an issue as you suggested if I can set aside the time.

As I dig deeper into the Elasticsearch integration there are some other ideas I'd like to explore. Specifically, indexing documents into an App Search Engine, though I don't know if that's possible with the .NET API client currently being used by Orchard.

[Piedone]

I see. Then yes, you'd need a pipeline. I wanted to suggest to add a new index field (not content field), because if you implement e.g. ContentPartIndexHandler<TPart>, then you can do indexing following arbitrary logic (even for content parts/fields that are built into OC and indexed otherwise too). However, we don't support adding weights there.

So the most suitable solution here looks like an Elasticsearch-specific extension point.

[cbadger-montecitobank]

Thanks for the tip on using ContentPartIndexHandler as another way to get more control over the indexing. I'll dig into that further as well, since it may prove useful in some situations.