_search calls to OpenSearch with terminate_after throw null_pointer_exception. Among others, this makes same visualizations that worked on OpenSearch 2.13 to fail on 2.14
#14198
Assignees
Labels
bug
Something isn't working
Search:Aggregations
Search
Search query, autocomplete ...etc
v2.15.0
Issues and PRs related to version 2.15.0
Comments
|
I am seeing the same error with 2.14, the visualization worked fine with 2.13 I see this notification: in the details is this information Request: {
"timeout": "1000ms",
"terminate_after": 100000,
"size": 0,
"aggs": {
"termsAgg": {
"terms": {
"order": {
"_count": "desc"
},
"field": "fields.umgebung.keyword"
}
}
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
{
"field": "@timestamp",
"format": "date_time"
}
],
"_source": {
"excludes": []
},
"query": {
"bool": {
"must": [],
"filter": [],
"should": [],
"must_not": []
}
}
}Response (removed all but two mentioned indices, so the numbers dont match): {
"took": 38,
"timed_out": false,
"terminated_early": false,
"_shards": {
"total": 110,
"successful": 53,
"skipped": 0,
"failed": 57,
"failures": [
{
"shard": 0,
"index": "filebeat-kmewo-prodo-7.12.1-2024.05.06",
"node": "7VUNjvDMSoSWpgKVDlGTiA",
"reason": {
"type": "null_pointer_exception",
"reason": "Cannot invoke \"org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null"
}
},
{
"shard": 0,
"index": "filebeat-kmewo-prodo-7.12.1-2024.05.07",
"node": "7VUNjvDMSoSWpgKVDlGTiA",
"reason": {
"type": "null_pointer_exception",
"reason": "Cannot invoke \"org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null"
}
}
]
},
"hits": {
"total": 1831452,
"max_score": null,
"hits": []
},
"aggregations": {
"termsAgg": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "qso",
"doc_count": 897627
},
{
"key": "prodo",
"doc_count": 504180
},
{
"key": "sem",
"doc_count": 238448
},
{
"key": "prodw",
"doc_count": 119040
},
{
"key": "qsw",
"doc_count": 72157
}
]
}
}
} |
|
@AMoo-Miki are you able to check on this. |
|
@franfreire could you please provide information on your indices and any configurations within OpenSearch and OpenSearch Dashboards. Also your local setup? Thank you. |
|
Hi, it's the same here. |
|
@disaster37, I am assuming you and @franfreire are together or the same person. If so and you have ascertained that the issue is caused by a call Controls viz is triggering, I would recommend looking at the Network tab of your browser's DevTools to see what specific query was run to cause this failure. @mjakob-gh, it would be great if you could provide some info about what the underlying failure is by checking Dev Tools too. On the surface, both of your errors are directly arriving from OpenSearch and I want to make sure we get more info while we transfer it there. |
|
@opensearch-project/admin please transfer to OpenSearch engine. |
|
I have the same issue than @franfreire but we are not the same person and we are not working on same organization. I will try to grab more useful trace on devtools like you suggest tomorrow. |
|
Hi, i have the same issue on 2.14 with Controls viz, with exactly the same stack traces than @franfreire on my opensearch client node. Works fine in 2.13. |
|
@AMoo-Miki I desn't understand what you need exactly ? I have look up on Chrom devtools, but I doesn't found more info than is already provided before by another people. I have doing some test on fresh cluster install with Opensearch 2.14.0. It is not so easy to reproduce issue. First, I create Index template (with data stream option) and some components templates linked for mapping. Now I go to visualize and I create default control visu on my Now, If create new index pattern To conclude, the problem seems appear when the index-pattern target multiple index and there are many document indexed on it and I try to create visu of type Controle. |
|
I too have many indices matching a index pattern. I do not see errors in the opensearch and opensearch-dashboard logs. |
|
Same here. Our Control-Vis has 1 Control reading from a tiny index (only 10 docs), which always works. The cluster metrics (heap used, indexing pressure, index time, threadpool write queue, system load) haven't changed from 2.13 to 2.14. |
|
Hum, after investigating a little more, maybee I found somethink. If I Take the query send to Opensearch by dashboard on devtools, I get the same error I get errors Now, If I removed the `"terminate_after": 1000000``from the query, no more error (Or If I put a very big value on it like "terminate_after": 1000000000) I get the expected result |
|
I looked into inputControlVis.plugin.js and found terminate_after:Number(settings.autocompleteTerminateAfter) Additionally I cleared the cached js files in the browser. For me it works now! => maybe just clearing the browser cache helps |
|
Spoke to @peternied about this. A short version of the problem is that We have at least 4 users of 2.14 reporting the same issue. @franfreire, @mjakob-gh, @disaster37, and @maaaax: thank you all for all of the analysis and details you have provided. This information will be of substantial help to the team. |
AMoo-Miki
changed the title
_search calls to OpenSearch with terminate_after throw null_pointer_exception. Among others, this makes same visualizations that worked on OpenSearch 2.13 to fail on 2.14
|
Added the search label to make sure it gets looked at by the right component team. Also, I noticed |
|
@andrross I think this is probably unrelated to CachedQueryResult, since that class was already present in 2.13 and hasn't been changed since then (besides a PR marking its inner class PolicyValues as ExperimentalAPI). From the stacktrace and looking at the code, it seems like InternalAggregations.getInternalAggregations() is returning a list that incorrectly contains some null values, which must not have been happening in 2.13. @disaster37 , would you be able to share data we can use with your query to repro this? |
|
Looked into this with @jed326 and it is related to #12697 . He is creating a PR to fix it. The issue happens with combination of |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Visualization type: Controls
I had this visualization working in opensearch until I updated to 2.14. I tested it back in opensearch 2.13 and in this version it works properly
When I check opensearch logs I see the following error:
OpenSearch Version
2.14
Dashboards Version
2.14
The text was updated successfully, but these errors were encountered: