Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[APM] Latency correlations: Error Unknown NamedWriteable category #105047

Closed
walterra opened this issue Jul 9, 2021 · 9 comments
Closed

[APM] Latency correlations: Error Unknown NamedWriteable category #105047

walterra opened this issue Jul 9, 2021 · 9 comments
Assignees
@walterra @benwtrent
Labels
apm:correlations bug Fixes for quality problems that affect the customer experience :ml Team:APM All issues that need APM UI Team support v7.14.0

Comments

@walterra
Copy link
Contributor

walterra commented Jul 9, 2021

Versions
Kibana: 7.14
Elasticsearch: 7.14

Note: So far this is not reproducable on master, only on 7.14.

This can be reproduced both on showcase as well on local instances using the apm-correlations dataset. For certain queries to get correlations, we receive a Unknown NamedWriteable category error message.

Example query triggering the error on 7.14: 

GET apm-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "processor.event": "transaction"
          }
        },
        {
          "term": {
            "service.name": "frontend-node"
          }
        },
        {
          "term": {
            "url.path": "/product/L9ECAV7KIM"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-02-01T09:29:00.000Z",
              "lte": "2021-02-03T09:29:34.890Z"
            }
          }
        }
      ]
    }
  },
  "size": 0,
  "aggs": {
    "latency_ranges": {
      "range": {
        "field": "transaction.duration.us",
        "ranges": [
          {
            "to": 849.25
          },
          {
            "from": 849.25,
            "to": 989.25
          },
          {
            "from": 989.25,
            "to": 1157.75
          },
          {
            "from": 1157.75,
            "to": 2017.75
          },
          {
            "from": 2017.75,
            "to": 2307.75
          },
          {
            "from": 2307.75,
            "to": 2529.75
          },
          {
            "from": 2529.75,
            "to": 2719.75
          },
          {
            "from": 2719.75,
            "to": 2871.75
          },
          {
            "from": 2871.75,
            "to": 2987.75
          },
          {
            "from": 2987.75,
            "to": 3087.75
          },
          {
            "from": 3087.75,
            "to": 3185.75
          },
          {
            "from": 3185.75,
            "to": 3283.75
          },
          {
            "from": 3283.75,
            "to": 3385.75
          },
          {
            "from": 3385.75,
            "to": 3489.75
          },
          {
            "from": 3489.75,
            "to": 3599.75
          },
          {
            "from": 3599.75,
            "to": 3721.75
          },
          {
            "from": 3721.75,
            "to": 3857.75
          },
          {
            "from": 3857.75,
            "to": 4011.75
          },
          {
            "from": 4011.75,
            "to": 4179.75
          },
          {
            "from": 4179.75,
            "to": 4347.75
          },
          {
            "from": 4347.75,
            "to": 4515.75
          },
          {
            "from": 4515.75,
            "to": 4691.75
          },
          {
            "from": 4691.75,
            "to": 4875.75
          },
          {
            "from": 4875.75,
            "to": 5067.75
          },
          {
            "from": 5067.75,
            "to": 5271.75
          },
          {
            "from": 5271.75,
            "to": 5467.75
          },
          {
            "from": 5467.75,
            "to": 5667.75
          },
          {
            "from": 5667.75,
            "to": 5867.75
          },
          {
            "from": 5867.75,
            "to": 6079.75
          },
          {
            "from": 6079.75,
            "to": 6307.75
          },
          {
            "from": 6307.75,
            "to": 6567.75
          },
          {
            "from": 6567.75,
            "to": 6895.75
          },
          {
            "from": 6895.75,
            "to": 7331.75
          },
          {
            "from": 7331.75,
            "to": 7967.75
          },
          {
            "from": 7967.75,
            "to": 8791.75
          },
          {
            "from": 8791.75,
            "to": 9503.75
          },
          {
            "from": 9503.75,
            "to": 10167.75
          },
          {
            "from": 10167.75,
            "to": 10831.75
          },
          {
            "from": 10831.75,
            "to": 11615.75
          },
          {
            "from": 11615.75,
            "to": 12711.75
          },
          {
            "from": 12711.75,
            "to": 15119.75
          },
          {
            "from": 15119.75,
            "to": 30719.75
          },
          {
            "from": 30719.75,
            "to": 66943.75
          },
          {
            "from": 66943.75,
            "to": 123455.75
          },
          {
            "from": 123455.75,
            "to": 183551.75
          },
          {
            "from": 183551.75,
            "to": 408575.75
          },
          {
            "from": 408575.75,
            "to": 416255.75
          },
          {
            "from": 416255.75,
            "to": 597503.75
          },
          {
            "from": 597503.75,
            "to": 978431.75
          },
          {
            "from": 978431.75
          }
        ]
      }
    },
    "transaction_duration_correlation": {
      "bucket_correlation": {
        "buckets_path": "latency_ranges>_count",
        "function": {
          "count_correlation": {
            "indicator": {
              "fractions": [
                0.020082101464723407,
                0.019969010466766846,
                0.019975509949408027,
                0.02005610353415868,
                0.0200782017751387,
                0.019891016675072665,
                0.019966410673710373,
                0.0202068915314341,
                0.02010939929181637,
                0.01975062785002314,
                0.020108099395288136,
                0.019901415847298556,
                0.020182193497397607,
                0.019875417916733827,
                0.019987209018162155,
                0.02006780260291281,
                0.019907915329939737,
                0.0199638108806539,
                0.02030958335716477,
                0.019997608190388046,
                0.019779225573644338,
                0.02019129277309526,
                0.020036605086235136,
                0.019729829505571356,
                0.020126297946683445,
                0.019758427229192556,
                0.020149696084191697,
                0.019909215226467974,
                0.020104199705703426,
                0.02009120074042106,
                0.019997608190388046,
                0.019813022883378485,
                0.0200496040515175,
                0.01996771057023861,
                0.020050903948045734,
                0.019858519261866756,
                0.02018479329045408,
                0.019883217295903245,
                0.019971610259823316,
                0.019982009432049207,
                0.020009307259142173,
                0.01997421005287979,
                0.020008007362613937,
                0.020004107673029226,
                0.019997608190388046,
                0.02036547890787893,
                0.019737628884740774,
                0.019933913260504463,
                0.019950811915371538,
                0.019997608190388046
              ],
              "expectations": [
                849.25,
                919.2499999999999,
                1073.5,
                1587.75,
                2162.75,
                2418.75,
                2624.75,
                2795.7500000000005,
                2929.75,
                3037.75,
                3136.75,
                3234.7499999999995,
                3334.7499999999995,
                3437.7499999999995,
                3544.7500000000005,
                3660.75,
                3789.75,
                3934.7499999999995,
                4095.7499999999995,
                4263.75,
                4431.749999999999,
                4603.75,
                4783.750000000001,
                4971.75,
                5169.75,
                5369.75,
                5567.75,
                5767.75,
                5973.75,
                6193.75,
                6437.75,
                6731.749999999999,
                7113.749999999999,
                7649.75,
                8379.75,
                9147.75,
                9835.750000000002,
                10499.75,
                11223.75,
                12163.75,
                13915.75,
                22919.75,
                48831.75,
                95199.75,
                153503.75,
                296063.75,
                412415.75,
                506879.75000000006,
                787967.75,
                978431.75
              ],
              "doc_count": 76929
            }
          }
        }
      }
    },
    "ks_test": {
      "bucket_count_ks_test": {
        "fractions": [
          0.020082101464723407,
          0.019969010466766846,
          0.019975509949408027,
          0.02005610353415868,
          0.0200782017751387,
          0.019891016675072665,
          0.019966410673710373,
          0.0202068915314341,
          0.02010939929181637,
          0.01975062785002314,
          0.020108099395288136,
          0.019901415847298556,
          0.020182193497397607,
          0.019875417916733827,
          0.019987209018162155,
          0.02006780260291281,
          0.019907915329939737,
          0.0199638108806539,
          0.02030958335716477,
          0.019997608190388046,
          0.019779225573644338,
          0.02019129277309526,
          0.020036605086235136,
          0.019729829505571356,
          0.020126297946683445,
          0.019758427229192556,
          0.020149696084191697,
          0.019909215226467974,
          0.020104199705703426,
          0.02009120074042106,
          0.019997608190388046,
          0.019813022883378485,
          0.0200496040515175,
          0.01996771057023861,
          0.020050903948045734,
          0.019858519261866756,
          0.02018479329045408,
          0.019883217295903245,
          0.019971610259823316,
          0.019982009432049207,
          0.020009307259142173,
          0.01997421005287979,
          0.020008007362613937,
          0.020004107673029226,
          0.019997608190388046,
          0.02036547890787893,
          0.019737628884740774,
          0.019933913260504463,
          0.019950811915371538,
          0.019997608190388046
        ],
        "buckets_path": "latency_ranges>_count",
        "alternative": [
          "less",
          "greater",
          "two_sided"
        ]
      }
    }
  }
}

The error response looks like this:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
      },
      {
        "type" : "illegal_argument_exception",
        "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
      },
      {
        "type" : "illegal_argument_exception",
        "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
      },
      {
        "type" : "illegal_argument_exception",
        "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "",
    "phase" : "fetch",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "apm-7.11.0-transaction-000001",
        "node" : "Hj_mOoneRVOcI4qapo3Cmw",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
        }
      },
      {
        "shard" : 0,
        "index" : "apm-7.12.1-error-000001",
        "node" : "Hj_mOoneRVOcI4qapo3Cmw",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
        }
      },
      {
        "shard" : 0,
        "index" : "apm-7.12.1-metric-000001",
        "node" : "Hj_mOoneRVOcI4qapo3Cmw",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
        }
      },
      {
        "shard" : 0,
        "index" : "apm-7.12.1-profile-000001",
        "node" : "Hj_mOoneRVOcI4qapo3Cmw",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Unknown NamedWriteable category [org.elasticsearch.xpack.ml.aggs.correlation.CorrelationFunction]"
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "[doc_count] must be a positive value"
    }
  },
  "status" : 400
}

Note the "[doc_count] must be a positive value" message despite the doc_count in the request being 76929. Does it refer to the foreground distribution?
@walterra walterra added the Team:APM All issues that need APM UI Team support label Jul 9, 2021
@walterra walterra self-assigned this Jul 9, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/apm-ui (Team:apm)

@walterra walterra added :ml apm:correlations bug Fixes for quality problems that affect the customer experience v7.14.0 labels Jul 9, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@walterra
Copy link
Contributor Author

walterra commented Jul 9, 2021

Further testing with the query from the issue description with ES 7.14 only (using Postman to do the request, no Kibana involved):

  • startup ES with apm-correlations already available
  • first request via postman: WORKS!
  • second request via postman: FAILS! -> includes error reason part "[doc_count] must be a positive value"
  • third request via postman: FAILS! -> doesn’t include the part about doc_count anymore

@droberts195
Copy link
Contributor

droberts195 commented Jul 9, 2021 •
edited
Loading

@benwtrent I think this is being caused by an omission from the backport of elastic/elasticsearch#72133. In that PR namedWriteables.addAll(new CorrelationNamedContentProvider().getNamedWriteables()); was added to MachineLearning.getNamedWriteables(). In the backport PR, elastic/elasticsearch#72896, it wasn't, which is correct because it needs to go in XPackClientPlugin.getNamedWriteables() instead 🎉 . But it wasn't added there either 😿 . Please can you fix this in the 7.x and 7.14 branches and also check if there were any other subtle differences between master and 7.x that weren't accounted for.

@benwtrent
Copy link
Member

OK, big miss from me. The named writeable serialization in 7.x is different than in 8.x

Additionally, there are no multi-node tests for these aggregations. I think this is a larger gap in the testing frame work for the aggregations team, but I think for this fix, I will add some specific multi node agg tests for the ML aggs.

@benwtrent benwtrent mentioned this issue Jul 12, 2021
Merged
benwtrent added a commit to elastic/elasticsearch that referenced this issue Jul 12, 2021
@benwtrent
[ML] fixing 7.x serialization for correlation aggregation (#75224)
49d78c6 
NamedWriteable Serialization was not declared in the original implementation for 7.x.

This commit fixes this. Relates to: #72133

closes: elastic/kibana#105047
@benwtrent benwtrent mentioned this issue Jul 12, 2021
Merged
benwtrent added a commit to elastic/elasticsearch that referenced this issue Jul 12, 2021
@benwtrent
[ML] fixing 7.x serialization for correlation aggregation (#75224) (#…
b9b5380 
…75234)

NamedWriteable Serialization was not declared in the original implementation for 7.x.

This commit fixes this. Relates to: #72133

closes: elastic/kibana#105047
@droberts195
Copy link
Contributor

@walterra this is fixed in the 7.14 branch by elastic/elasticsearch#75234. That should be in the next 7.14.0 build candidate later this week, and also in a promoted ES snapshot for Kibana CI within 24 hours.

@zube zube bot added the [zube]: 7.14 label Jul 12, 2021
@walterra
Copy link
Contributor Author

Note after testing: Quite same error will trigger in CCS scenarios where a current 7.14 with the new Latency correlations queries a pre-7.14 cluster. We will track this and think of useful error messages in the UI. For example, if this exact error triggers and we know the index pattern used has a : in it like my_remote_cluster:apm-* then we could display a hint about possible CSS issues in addition to the raw ES error response.

@benwtrent
Copy link
Member

Quite same error will trigger in CCS scenarios where a current 7.14 with the new Latency correlations queries a pre-7.14 cluster.

This is to be expected. While serializing pipeline aggregations are not TECHNICALLY required to search against remote clusters, it still occurs.

Its a bit of techdebt: elastic/elasticsearch#73680

@walterra
Copy link
Contributor Author

Fixed by #105200.

@zube zube bot added [zube]: Done and removed [zube]: 7.14 labels Jul 19, 2021
2lambda123 pushed a commit to 2lambda123/elastic-elasticsearch that referenced this issue May 3, 2024
@benwtrent
[ML] fixing 7.x serialization for correlation aggregation (#75224)
d9778cc 
NamedWriteable Serialization was not declared in the original implementation for 7.x.

This commit fixes this. Relates to: #72133

closes: elastic/kibana#105047
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walterra walterra

@benwtrent benwtrent

Labels
apm:correlations bug Fixes for quality problems that affect the customer experience :ml Team:APM All issues that need APM UI Team support v7.14.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@smith @walterra @benwtrent @droberts195 @elasticmachine