Update matrix-stats aggregation.

[dwelsch-esi]

Description

Updated descriptions. Added explanation of result statistics.

This PR will probably require some major revisions before it's published. I have several questions. See line comments.

Issues Resolved

Version

Frontend features

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

[dwelsch-esi]

mode did not seem to affect the result when I tested it with array data. Maybe it doesn't work when one field is a single value and another is an array? But that doesn't make much sense.

What is the default mode?

[kolchfa-aws]

@jainankitk Could you please answer this quiestion?

[jainankitk]

@dwelsch-esi - Can you share the query that you ran?

[dwelsch-esi]

@jainankitk Try the following:

DELETE /students

POST _bulk
{ "create": { "_index": "students", "_id": "1" } }
{ "name": "John Doe", "gpa": 3.89, "class_grades": [3.0, 3.9, 4.0], "grad_year": 2022}
{ "create": { "_index": "students", "_id": "2" } }
{ "name": "Jonathan Powers", "gpa": 3.85, "class_grades": [4.1, 3.0, 4.0], "grad_year": 2025 }
{ "create": { "_index": "students", "_id": "3" } }
{ "name": "Jane Doe", "gpa": 3.52, "class_grades": [3.2, 2.1, 3.8], "grad_year": 2024 }

GET students/_search
{
  "size": 0,
  "aggs": {
    "matrix_stats_taxful_total_price": {
      "matrix_stats": {
        "fields": ["gpa", "class_grades"],
        "mode": "avg"
      }
    }
  }
}

I tried different modes (min, max) instead of avg. In all cases, the correlation between gpa and class_grades is the same, 0.9820867239098596. Surely it should be different if the value used for class_grades is computed differently for different mode values?

[jainankitk]

Thanks @dwelsch-esi for close observation. I noticed that the first response is cached resulting in this discrepancy. You can try using the GET students/_search?request_cache=false parameter to avoid this issue. Or alternatively, POST /_cache/clear before each search request

[jainankitk]

we have PR for fixing this discrepancy - opensearch-project/OpenSearch#18254, in case anyone is curious

[dwelsch-esi]

1. Does matrix-stats always sample every available document? If not, how is sampling done?

2. What assumptions does matrix-stats make about independence of data samples? Is it sufficent to say that it assumes documents are independent of each other?

[jainankitk]

Again will probably keep it as multi-value metric aggregation and maybe link to the metric aggregation page

[dwelsch-esi]

@jainankitk Please see my technical questions throughout this file. Can you provide any insights?

[kolchfa-aws]

@jainankitk Could you please answer this quiestion?

[jainankitk]

Does matrix-stats always sample every available document? If not, how is sampling done?

I don't think there is any sampling, all the documents are processed

What assumptions does matrix-stats make about independence of data samples? Is it sufficent to say that it assumes documents are independent of each other?

Matrix-stats does not make any assumptions about the data. I am wondering why does it need to make any assumption?

[dwelsch-esi]

@jainankitk There are several assumptions that have to be met for a correlation to be valid: https://www.statology.org/pearson-correlation-assumptions/. However, that level of detail is probably beyond the scope of the documentation here. At some level, the user is assumed to know what they're doing if they use an aggregation.

[kolchfa-aws]

@jainankitk Could you please review this PR? Thanks!

[jainankitk]

Again will probably keep it as multi-value metric aggregation and maybe link to the metric aggregation page

[kolchfa-aws]

@jainankitk Thanks for your feedback. I addressed your comments and added a section about the missing parameter.

[natebower]

@kolchfa-aws One comment and a couple of changes. Thanks!