Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Flaky BWC Test #3297

Open
dhrubo-os opened this issue Dec 26, 2024 · 1 comment
Open

[BUG] Flaky BWC Test #3297

dhrubo-os opened this issue Dec 26, 2024 · 1 comment
Labels
bug Something isn't working untriaged

Comments

@dhrubo-os
Copy link
Collaborator

What is the bug?
Our BWC tests fails often due to Flaky behavior.

Example REF: Test MLCommons BWC / Test MLCommons BWC (11) (pull_request) Failing after 5m:

REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:mlCommonsBwcCluster#twoThirdsUpgradedClusterTask' --tests "org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility" -Dtests.seed=11A13B1333FF513B -Dtests.security.manager=false -Dtests.locale=sr -Dtests.timezone=US/East-Indiana -Druntime.java=11
MLCommonsBackwardsCompatibilityIT > testBackwardsCompatibility FAILED

Suite: Test class org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT
  2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:mlCommonsBwcCluster#twoThirdsUpgradedClusterTask' --tests "org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility" -Dtests.seed=11A13B1333FF513B -Dtests.security.manager=false -Dtests.locale=sr -Dtests.timezone=US/East-Indiana -Druntime.java=11
  2> org.junit.ComparisonFailure: expected:<[m_l_limit_exceeded]_exception> but was:<[null_pointer]_exception>
        at __randomizedtesting.SeedInfo.seed([11A13B1333FF513B:FA9E06DD492484B1]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility(MLCommonsBackwardsCompatibilityIT.java:219)
  2> NOTE: leaving temporary files on disk at: /home/runner/work/ml-commons/ml-commons/plugin/build/testrun/mlCommonsBwcCluster#twoThirdsUpgradedClusterTask/temp/org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT_11A13B1333FF513B-001
  2> NOTE: test params are: codec=Asserting(Lucene912): {}, docValues:{}, maxPointsInLeafNode=1049, maxMBSortInHeap=6.9585268942530245, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=sr, timezone=US/East-Indiana
  2> NOTE: Linux 6.8.0-1017-azure amd64/Azul Systems, Inc. 11.0.25 (64-bit)/cpus=4,threads=1,free=413015552,total=5368[70](https://github.com/opensearch-project/ml-commons/actions/runs/12485923295/job/34845364486?pr=3295#step:5:71)912
  2> NOTE: All tests run in this JVM: [MLCommonsBackwardsCompatibilityIT]
    org.junit.ComparisonFailure: expected:<[m_l_limit_exceeded]_exception> but was:<[null_pointer]_exception>
        at __randomizedtesting.SeedInfo.seed([11A13B1333FF513B:FA9E06DD492484B1]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility(MLCommonsBackwardsCompatibilityIT.java:219)
  1> [2024-12-24T15:31:58,190][INFO ][o.o.m.b.MLCommonsBackwardsCompatibilityIT] [testBackwardsCompatibility] before test
  1> [2024-12-24T15:31:58,402][INFO ][o.o.m.b.MLCommonsBackwardsCompatibilityIT] [testBackwardsCompatibility] initializing REST clients against [http://[::1]:34625, http://127.0.0.1:41483, http://[::1]:43127, http://127.0.0.1:37889, http://[::1]:40915, http://127.0.0.1:41841]
  1> [2024-12-24T15:32:01,085][INFO ][o.o.m.b.MLCommonsBackwardsCompatibilityIT] [testBackwardsCompatibility] after test

Tests with failures:
 - org.opensearch.ml.bwc.MLCommonsBackwardsCompatibilityIT.testBackwardsCompatibility


1 test completed, 1 failed

=== Standard output of node `node{:opensearch-ml-plugin:mlCommonsBwcCluster0-0}` ===
»    ↓ errors and warnings from /home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/mlCommonsBwcCluster0-0/logs/opensearch.stdout.log ↓
» WARN ][o.o.g.DanglingIndicesState] [mlCommonsBwcCluster0-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
»   ↑ repeated 2 times ↑
» WARN ][o.o.d.FileBasedSeedHostsProvider] [mlCommonsBwcCluster0-0] expected, but did not find, a dynamic hosts list at [/home/runner/work/ml-commons/ml-commons/plugin/build/testclusters/mlCommonsBwcCluster0-0/config/unicast_hosts.txt]
»   ↑ repeated 12 times ↑
» WARN ][o.o.c.c.ClusterFormationFailureHelper] [mlCommonsBwcCluster0-0] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [mlCommonsBwcCluster0-0, mlCommonsBwcCluster0-1, mlCommonsBwcCluster0-2] to bootstrap a cluster: have discovered [{mlCommonsBwcCluster0-0}{MhRx5MfAROu5EtR1P1Ug_g}{8qKOvXoMT0aAx-gnim80Tg}{127.0.0.1}{127.0.0.1:46535}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}]; discovery will continue using [] from hosts providers and [{mlCommonsBwcCluster0-0}{MhRx5MfAROu5EtR1P1Ug_g}{8qKOvXoMT0aAx-gnim80Tg}{127.0.0.1}{127.0.0.1:46535}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
» WARN ][o.o.c.NodeConnectionsService] [mlCommonsBwcCluster0-0] failed to connect to {mlCommonsBwcCluster0-1}{XCkBVNJ7Sa2Nyi5K3PbKQg}{o3fHphW7SnWp1HZE1KrCHA}{127.0.0.1}{127.0.0.1:36181}{dimr}{testattr=test, shard_indexing_pressure_enabled=true} (tried [1] times)
»  org.opensearch.transport.ConnectTransportException: [mlCommonsBwcCluster0-1][127.0.0.1:36181] connect_exception
»  	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1106) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]

»  	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
»  	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
=== Standard error of node `node{:opensearch-ml-plugin:mlCommonsBwcCluster0-0}` ===
»  	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) ~[?:?]

»  	at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:[72](https://github.com/opensearch-project/ml-commons/actions/runs/12485923295/job/34845364486?pr=3295#step:5:73)) ~[opensearch-common-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
=== Standard output of node `node{:opensearch-ml-plugin:mlCommonsBwcCluster0-1}` ===
»  	at org.opensearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:83) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
»  	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
»  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:326) ~[?:?]
»  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:342) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:[74](https://github.com/opensearch-project/ml-commons/actions/runs/12485923295/job/34845364486?pr=3295#step:5:75)) ~[?:?]
»  	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
»  Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 127.0.0.1/127.0.0.1:36181
»  Caused by: java.net.ConnectException: Connection refused
»  	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
»  	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
»  	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336) ~[?:?]
»  	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339) ~[?:?]
»  	... 7 more
» ERROR][o.o.m.c.MLCommonsClassLoader] [mlCommonsBwcCluster0-0] Failed to init instance for type PREDICTION
»  java.lang.reflect.InvocationTargetException: null
»  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
»  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
»  	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
»  	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
»  	at org.opensearch.ml.common.MLCommonsClassLoader.init(MLCommonsClassLoader.java:242) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.MLCommonsClassLoader.initMLInstance(MLCommonsClassLoader.java:206) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.output.MLOutput.fromStream(MLOutput.java:36) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.ml.common.transport.MLTaskResponse.<init>(MLTaskResponse.java:39) [opensearch-ml-common-2.19.0.0-SNAPSHOT.jar:?]
»  	at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:85) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:52) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1508) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1495) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:405) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
»  Caused by: java.io.EOFException
»  	at org.opensearch.core.common.io.stream.InputStreamStreamInput.readByte(InputStreamStreamInput.java:74) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.common.io.stream.FilterStreamInput.readByte(FilterStreamInput.java:55) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.common.io.stream.StreamInput.readBoolean(StreamInput.java:586) ~[opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.ml.common.output.MLPredictionOutput.<init>(MLPredictionOutput.java:72) ~[?:?]
»  	... 42 more
» WARN ][r.suppressed             ] [mlCommonsBwcCluster0-0] path: /_plugins/_ml/_train_predict/kmeans, params: {algorithm=kmeans}
»  java.lang.NullPointerException: null
»  	at org.opensearch.ml.common.transport.MLTaskResponse.toXContent(MLTaskResponse.java:64) ~[?:?]
»  	at org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:62) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:57) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:49) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:52) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:64) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1517) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.doHandleResponse(NativeMessageHandler.java:427) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleResponse(NativeMessageHandler.java:419) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:174) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:120) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:112) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.forwardFragments(InboundBytesHandler.java:137) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:77) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.115.Final.jar:4.1.115.Final]
»  	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
» ERROR][o.o.m.a.MLModelAutoReDeployer] [mlCommonsBwcCluster0-0] Failed to query need undeploy models, no action will be performed
»  org.opensearch.index.IndexNotFoundException: no such index [.plugins-ml-model]
»  	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1070) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1007) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:963) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:255) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:231) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:993) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1033) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:877) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$5(TransportSearchAction.java:517) ~[opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$1(TransportSearchAction.java:4[76](https://github.com/opensearch-project/ml-commons/actions/runs/12485923295/job/34845364486?pr=3295#step:5:77)) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.search.pipeline.Pipeline.transformRequest(Pipeline.java:125) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:34) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:483) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:320) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:136) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]
»  	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) [opensearch-2.19.0-SNAPSHOT.jar:2.19.0-SNAPSHOT]

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

  • OS: [e.g. iOS]
  • Version [e.g. 22]
  • Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.
@dhrubo-os dhrubo-os added bug Something isn't working untriaged labels Dec 26, 2024
@mingshl
Copy link
Collaborator

mingshl commented Jan 6, 2025

same issue happened to #3333

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dhrubo-os @mingshl