Backend: Migrate Lucene metrics to the backend

[idastambuk]

Note: Blocked by #224

Currently, Lucene metric queries are processed as time series (frontend). This flow needs to be compared and migrated to the backend. Some of the logic already exists on the backend here, but it might need to be updated.

---

To tackle this issue, we'll start by comparing the way the backend and frontend:

• build the query to OpenSearch
• return the response in Grafana

Differences between frontend/backend:

• Metric, Metric Max, Group By Terms
  • Query building to OpenSearch
    • In terms, order's key is _term in backend, _key in frontend Lucene metrics: Fix replacement of _term to _key in terms order #270
    • min_doc_count missing in backend, present in frontend Lucene metrics: Parse MinDocCount as int or string #268
    • size is 500 in backend, 0 in frontend Lucene metrics: Remove "size":500 from backend processTimeSeriesQuery #269
  • Grafana data frames response:
    • Backend has: additional typeInfo object for AvgTicketPrice and Max with "typeInfo": { "frame": "float64", "nullable": true
    • Frontend does not have typeInfo
    • Conclusion: Backend seems acceptable as this is how the equivalent Elasticsearch query also returns data frames.
• Metric Sum / Group By Date Histogram / Alias
  • Query building to OpenSearch
    • no difference between backend and frontend
  • Grafana data frames response:
    • Backend has additional "config" object with "displayNameFromDS": "distanceKms" : this is expected due to Backend: Set field.Config.DisplayNameFromDS instead of frame.name #180. Frontend has an additional "name": "distanceKms" object which is replaced by the displayNameFromDS in the backend. This is also the same behavior in Elasticsearch for the same type of query.
    • Similar to above, backend has additional typeInfo object
    • Backend has additional "meta": { "type": "timeseries-multi", "typeVersion": [ 0, 0 ] } object. This is due to code based on an Elasticsearch PR Elasticsearch: backend: do not set name for time-fields grafana#60169 and was included in Backend: Set field.Config.DisplayNameFromDS instead of frame.name #180 and Fix backend pipeline aggregation query parsing and data frame building #168. This is also the same behavior in Elasticsearch for the same type of query.
    • Backend has additional "labels": { "metricId": "1" } object. The presence and use of this object is consistent with Elasticsearch for the same type of query.
• Metric Average / Metric Derivative / Group By Date Histogram
  • Query building to OpenSearch
    • no difference between backend and frontend
  • Grafana data frames response:
    • same differences as above

---

Some general guidelines are:

• Create visualization to compare between frontend and backend flows, preferably in a shared Cloud dashboard e.g. clouddatasources.grafana.net
• Unit tests in the frontend should be copied or updated in the backend to assert the same behavior and add unit tests when there are none.
  • Check Elasticsearch repo for related tests – helpful when there are no OpenSearch tests in the frontend or backend
• We should match OpenSearch's backend behavior to the current frontend behavior, but be aware it may not always make sense. Elasticsearch's current behavior is also another resource for how OpenSearch should behave.
  • Let's be pragmatic about this – our foremost goal is the migration of OpenSearch's frontend behavior. We can aim for some Elasticsearch feature parity, but only within reason.
• Often there is a PR in the Elasticsearch backend data source migration Elasticsearch: Convert the datasource to be a backend datasource grafana#54011 which is 75% of what OpenSearch will need
• Migrate or implement: i.e. make any changes to code in backend
• Use the dashboard created above to compare visualization results (data frame comparison)
  • Can also inspect data frame on a more detailed basis