Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

fix detector and feature serialization #322

Merged
merged 1 commit into from
Dec 3, 2020

Conversation

ylwu-amzn
Copy link
Contributor

Issue #, if available:

Description of changes:

Detector and feature serialization has issue, see such error:

[2020-12-03T03:10:00,566][ERROR][c.a.o.a.t.ADTaskManager  ] [node-1] Fail to execute batch task action BYSTJnYBeb-NN24R-TEn
org.elasticsearch.transport.RemoteTransportException: [node-3][127.0.0.1:9302][cluster:admin/opendistro/ad/detector/batch_run_remotely]
Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.elasticsearch.search.aggregations.PipelineAggregationBuilder][value]
	at org.elasticsearch.common.io.stream.NamedWriteableRegistry.getReader(NamedWriteableRegistry.java:112) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:45) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:39) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.<init>(AggregatorFactories.java:258) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.<init>(AbstractAggregationBuilder.java:60) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.<init>(ValuesSourceAggregationBuilder.java:166) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder$LeafOnly.<init>(ValuesSourceAggregationBuilder.java:122) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.search.aggregations.metrics.SumAggregationBuilder.<init>(SumAggregationBuilder.java:75) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:46) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:39) ~[elasticsearch-7.10.0.jar:7.10.0]
	at com.amazon.opendistroforelasticsearch.ad.model.Feature.<init>(Feature.java:75) ~[?:?]
	at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1220) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.common.io.stream.StreamInput.readList(StreamInput.java:1170) ~[elasticsearch-7.10.0.jar:7.10.0]
	at com.amazon.opendistroforelasticsearch.ad.model.AnomalyDetector.<init>(AnomalyDetector.java:275) ~[?:?]
	at com.amazon.opendistroforelasticsearch.ad.model.ADTask.<init>(ADTask.java:89) ~[?:?]
	at com.amazon.opendistroforelasticsearch.ad.transport.ADBatchAnomalyResultRequest.<init>(ADBatchAnomalyResultRequest.java:43) ~[?:?]
	at org.elasticsearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:59) ~[elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:195) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.0.jar:7.10.0]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.0.jar:7.10.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]

Changes

This PR fixed serialization of anomaly detector and feature by writing to and reading from stream.

Test

1.Add unit test cases for serialization of anomaly detector and feature
2../gradlew build pass
3../gradlew run and test

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@codecov
Copy link

codecov bot commented Dec 3, 2020

Codecov Report

Merging #322 (e3d5991) into master (3e3d599) will increase coverage by 0.80%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #322      +/-   ##
============================================
+ Coverage     72.11%   72.92%   +0.80%     
- Complexity     1985     2000      +15     
============================================
  Files           201      201              
  Lines          9558     9563       +5     
  Branches        850      852       +2     
============================================
+ Hits           6893     6974      +81     
+ Misses         2256     2176      -80     
- Partials        409      413       +4     
Flag Coverage Δ Complexity Δ
plugin 72.41% <100.00%> (+0.87%) 0.00 <0.00> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ Complexity Δ
...stroforelasticsearch/ad/model/AnomalyDetector.java 87.98% <100.00%> (+22.46%) 66.00 <0.00> (+6.00)
...n/opendistroforelasticsearch/ad/model/Feature.java 96.36% <100.00%> (+20.00%) 15.00 <0.00> (+2.00)
...sticsearch/ad/indices/AnomalyDetectionIndices.java 49.34% <0.00%> (-0.33%) 40.00% <0.00%> (ø%)
...ransport/DeleteAnomalyDetectorTransportAction.java 53.84% <0.00%> (+16.66%) 13.00% <0.00%> (+4.00%)
...sticsearch/ad/model/IntervalTimeConfiguration.java 100.00% <0.00%> (+36.36%) 11.00% <0.00%> (+3.00%)

if (input.readBoolean()) {
this.user = new User(input);
} else {
user = null;
}
if (input.readBoolean()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious: why readBoolean being true indicates uiMetadata can be read from readMap()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@saratvemulapalli saratvemulapalli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for change Yaliang!

@ylwu-amzn ylwu-amzn merged commit 358afe5 into opendistro-for-elasticsearch:master Dec 3, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants