Add elasticsearch _score and highlights to ElasticsearchQueryResult

[denispetrische]

I have a task to get low level elasticsearch information such as _score of a hit and highlights. And for now it is impossible to get it.

So this fix:

1. Added support of highlights in the queries
2. Added return of elasticsearch _score and highlights in graphQL using object return scheme

[MikeAlhayek]

Suggested change

	KeyValuePair.Create(x.Key, (JsonNode)JsonValue.Create(x.Value.ToString()))).ToList();
	KeyValuePair.Create(x.Key, (JsonNode)JsonValue.Create(x.Value.ToString())));

[denispetrische]

I need ToList() here to further add two more KeyValuePairs to it with _score and highlight

[MikeAlhayek]

Why Hits here not TopDocs?

[Skrypt]

Not good. This is the old way of returning results. You should use TopDocs now. This will alter all the search results.

[denispetrische]

Hits contain all the data I need (including _score and highlights), topDocs is the information that contains in Hits.Source.

[MikeAlhayek]

you should be able to use return _score and highlights in the TopDocs

[denispetrische]

you should be able to use return _score and highlights in the TopDocs

For now, we get topDocs from elasticSearch searchResponse.Documents.

searchResponse.Documents == searchResponse.Hit.Source so it doesn't contain _score and highlights.

[MikeAlhayek]

@denispetrische I think you should modify the ElasticQueryService class, before the line "hits.Add(row);" you could add

row["_source"] = hit.Source;
row["highlight"] = hit.Highlight;

Then in the source you'll have that info available to be returned.

But, if you add highlight and score, you'll need to probably also use them when returning ContentItems. I am wondering if we can return a new object instead of returning content item directly something like

public class ElasticsearchQueryResult
{
    public string Key { get; set; }

    public double? Score { get; set; }

    public string Highlight { get; set; }

    public IEnumerable<ContentItem> ContentItems { get; set; }
}

But I am not sure about the extend of adding ElasticsearchQueryResult instead of returning ContentItems because it likely to require changes to IQueryResults

[denispetrische]

I think you should modify the ElasticQueryService class, before the line "hits.Add(row);" you could add

But this will work only if we specify Fields

[denispetrische]

But, if you add highlight and score, you'll need to probably also use them when returning ContentItems.

I am not sure about the extend of adding ElasticsearchQueryResult instead of returning ContentItems because it likely to require changes to IQueryResults

Yeah, It's easier to add _score and highlight only for object return type). Will it be okay if I leave contentItems as that, without score and highlight?

I think there is no problem to get all the info I need with object return type.

[MikeAlhayek]

@Skrypt would be better as answering these questions as he is more involved in the implementation.

[denispetrische]

@Skrypt What do you think?)

[Skrypt]

Ok, I'm trying to understand what you are trying to achieve or fix here in this PR. I will need to run this PR locally and it will take some time so that I can review everything and see what's happening when debugging. I don't remember everything related with ElasticSearch. These proposed changes may be appropriate or not depending on what we did with Lucene implementation and we need to make sure that both implementations are similar.

And the part involving GraphQL is probably a Query Schema that you expose for GraphQL simply right?

[denispetrische]

Ok, I'm trying to understand what you are trying to achieve or fix here in this PR. I will need to run this PR locally and it will take some time so that I can review everything and see what's happening when debugging. I don't remember everything related with ElasticSearch. These proposed changes may be appropriate or not depending on what we did with Lucene implementation and we need to make sure that both implementations are similar.

And the part involving GraphQL is probably a Query Schema that you expose for GraphQL simply right?

Right)

[MikeAlhayek]

@denispetrische I have a PR to upgrade Elasticsearch library. In the process, I implemented _score and Highlights. If possible, please test drive PR #17027 . Hopefully, it gives you the support you are looking for.

[denispetrische]

@denispetrische I have a PR to upgrade Elasticsearch library. In the process, I implemented _score and Highlights. If possible, please test drive PR #17027 . Hopefully, it gives you the support you are looking for.

Okay, I'll check it

[denispetrische]

@denispetrische I have a PR to upgrade Elasticsearch library. In the process, I implemented _score and Highlights. If possible, please test drive PR #17027 . Hopefully, it gives you the support you are looking for.

It doesn't work for me now, throws exception.

I used an object return type for the query.

[MikeAlhayek]

@denispetrische if you encountered that issue after testing the other PR, please comment directly on that PR so we can fix the issue there.

[github-actions]

This pull request has merge conflicts. Please resolve those before requesting a review.

[MikeAlhayek]

Using the latest in main branch will provide you access to Highligh and score info by default. If you are able to, try out the latest 3.0.0-preview from cloud smith tomorrow or after 7 hours from now so you can get the latest changes from Elasticsearch.