By type
[38;5;6m [38;5;5m14:50:05.79 [0m
[38;5;6m [38;5;5m14:50:05.80 [0m[1mWelcome to the Bitnami elasticsearch container[0m
[38;5;6m [38;5;5m14:50:05.80 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m
[38;5;6m [38;5;5m14:50:05.81 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m
[38;5;6m [38;5;5m14:50:05.89 [0m
[38;5;6m [38;5;5m14:50:05.90 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup **
[38;5;6m [38;5;5m14:50:06.11 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch...
[38;5;6m [38;5;5m14:50:06.41 [0m[38;5;2mINFO [0m ==> Setting default configuration
[38;5;6m [38;5;5m14:50:06.59 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings...
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[38;5;6m [38;5;5m14:50:23.31 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! **
[38;5;6m [38;5;5m14:50:23.50 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-10-14T14:50:54,403][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS]
[2022-10-14T14:50:54,497][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] JVM home [/opt/bitnami/java]
[2022-10-14T14:50:54,500][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-5197005119438229371, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-10-14T14:51:10,205][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [aggs-matrix-stats]
[2022-10-14T14:51:10,206][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [analysis-common]
[2022-10-14T14:51:10,206][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [geo]
[2022-10-14T14:51:10,206][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [ingest-common]
[2022-10-14T14:51:10,207][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [ingest-geoip]
[2022-10-14T14:51:10,207][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [ingest-user-agent]
[2022-10-14T14:51:10,207][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [kibana]
[2022-10-14T14:51:10,207][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [lang-expression]
[2022-10-14T14:51:10,207][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [lang-mustache]
[2022-10-14T14:51:10,208][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [lang-painless]
[2022-10-14T14:51:10,294][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [mapper-extras]
[2022-10-14T14:51:10,295][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [parent-join]
[2022-10-14T14:51:10,296][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [percolator]
[2022-10-14T14:51:10,296][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [rank-eval]
[2022-10-14T14:51:10,296][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [reindex]
[2022-10-14T14:51:10,296][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [repository-url]
[2022-10-14T14:51:10,297][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [tasks]
[2022-10-14T14:51:10,297][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [transport-netty4]
[2022-10-14T14:51:10,298][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded plugin [repository-s3]
[2022-10-14T14:51:11,396][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-2] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.13.82:/dockerdata-nfs/onap/elastic-master-2)]], net usable_space [95.4gb], net total_space [99.9gb], types [nfs4]
[2022-10-14T14:51:11,397][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-2] heap size [123.7mb], compressed ordinary object pointers [true]
[2022-10-14T14:51:11,999][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] node name [onap-sdnrdb-master-2], node ID [No94EsJOTM-kk9Fq8fSO9g], cluster name [sdnrdb-cluster]
[2022-10-14T14:51:51,195][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-master-2] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}]
[2022-10-14T14:51:52,103][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-master-2] using discovery type [zen] and seed hosts providers [settings]
[2022-10-14T14:51:55,307][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-10-14T14:51:57,101][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] initialized
[2022-10-14T14:51:57,102][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] starting ...
[2022-10-14T14:51:58,204][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][1] overhead, spent [301ms] collecting in the last [1s]
[2022-10-14T14:51:59,104][INFO ][o.e.t.TransportService ] [onap-sdnrdb-master-2] publish_address {10.233.69.23:9300}, bound_addresses {0.0.0.0:9300}
[2022-10-14T14:52:00,298][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.69.23:9300, remoteAddress=/127.0.0.6:40355}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-10-14T14:52:00,499][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.69.23:9300, remoteAddress=/127.0.0.6:44685}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-10-14T14:52:00,797][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.69.23:9300, remoteAddress=/127.0.0.6:36853}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-10-14T14:52:01,502][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.69.23:9300, remoteAddress=/127.0.0.6:40013}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-10-14T14:52:01,600][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-master-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-10-14T14:52:03,397][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-onap-sdnrdb-master-0,mPUdSz6LTq-eI9TuIEvEqg,No94EsJOTM-kk9Fq8fSO9g}
[2022-10-14T14:52:05,594][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-2] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr} elect leader, {onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 1, delta: master node changed {previous [], current [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T14:52:05,599][WARN ][o.e.c.s.MasterService ] [onap-sdnrdb-master-2] failing [elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr} elect leader, {onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 2 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-10-14T14:52:05,604][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}, targetNode={onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-2][10.233.69.23:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 2 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-10-14T14:52:06,408][INFO ][o.e.c.c.CoordinationState] [onap-sdnrdb-master-2] cluster UUID set to [9Dq5nEzRSyiQvLMtK0U5qQ]
[2022-10-14T14:52:06,815][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}, term: 2, version: 1, reason: ApplyCommitRequest{term=2, version=1, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T14:52:06,907][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-2] publish_address {10.233.69.23:9200}, bound_addresses {0.0.0.0:9200}
[2022-10-14T14:52:06,908][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] started
[2022-10-14T14:52:07,794][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2, reason: ApplyCommitRequest{term=2, version=2, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T14:52:22,609][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4, reason: ApplyCommitRequest{term=2, version=4, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T14:52:31,802][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-master-2] updating [action.auto_create_index] from [true] to [false]
[2022-10-14T14:52:41,401][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][44] overhead, spent [286ms] collecting in the last [1s]
[2022-10-14T15:52:03,009][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T15:52:03,018][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 86, reason: becoming candidate: onLeaderFailure
[2022-10-14T15:52:03,500][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 87, reason: ApplyCommitRequest{term=2, version=87, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:06,268][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 88, reason: ApplyCommitRequest{term=2, version=88, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:08,781][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 89, reason: ApplyCommitRequest{term=2, version=89, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:23,547][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 91, reason: ApplyCommitRequest{term=2, version=91, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:24,197][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,204][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,301][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,307][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,400][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,801][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,895][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T15:52:24,996][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T15:52:25,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T15:52:25,114][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T15:52:25,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T15:52:25,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T15:52:25,304][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T15:52:25,311][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T15:52:26,335][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 92, reason: ApplyCommitRequest{term=2, version=92, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:27,042][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 93, reason: ApplyCommitRequest{term=2, version=93, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:27,895][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 94, reason: ApplyCommitRequest{term=2, version=94, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:45,136][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T15:53:45,144][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 162, reason: becoming candidate: onLeaderFailure
[2022-10-14T15:53:51,013][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 168, reason: ApplyCommitRequest{term=2, version=168, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:03,107][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T16:52:03,112][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 236, reason: becoming candidate: onLeaderFailure
[2022-10-14T16:52:03,421][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 237, reason: ApplyCommitRequest{term=2, version=237, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:08,677][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 238, reason: ApplyCommitRequest{term=2, version=238, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:11,504][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 239, reason: ApplyCommitRequest{term=2, version=239, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:28,213][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 241, reason: ApplyCommitRequest{term=2, version=241, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:28,394][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,493][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,503][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,597][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,808][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,906][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T16:52:28,916][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,006][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,105][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,200][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,207][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,217][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,299][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,316][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,398][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,416][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,502][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,503][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,602][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T16:52:29,611][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T16:52:30,069][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 242, reason: ApplyCommitRequest{term=2, version=242, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:53:52,801][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T16:53:52,894][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 311, reason: becoming candidate: onLeaderFailure
[2022-10-14T16:53:53,635][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 313, reason: ApplyCommitRequest{term=2, version=313, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:54:10,494][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][7328] overhead, spent [314ms] collecting in the last [1.2s]
[2022-10-14T17:52:03,210][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T17:52:03,216][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 381, reason: becoming candidate: onLeaderFailure
[2022-10-14T17:52:03,609][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 382, reason: ApplyCommitRequest{term=2, version=382, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:11,517][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 383, reason: ApplyCommitRequest{term=2, version=383, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:14,611][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 384, reason: ApplyCommitRequest{term=2, version=384, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:30,400][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 386, reason: ApplyCommitRequest{term=2, version=386, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:30,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,604][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,702][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,802][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,896][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,906][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:30,998][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,105][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,110][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,218][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,224][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,231][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,310][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,405][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,496][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T17:52:31,700][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T17:52:32,913][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 387, reason: ApplyCommitRequest{term=2, version=387, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:53:55,961][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T17:53:55,969][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 457, reason: becoming candidate: onLeaderFailure
[2022-10-14T17:53:56,356][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 459, reason: ApplyCommitRequest{term=2, version=459, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:03,302][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T18:52:03,306][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 527, reason: becoming candidate: onLeaderFailure
[2022-10-14T18:52:03,443][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 528, reason: ApplyCommitRequest{term=2, version=528, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:14,548][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 529, reason: ApplyCommitRequest{term=2, version=529, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:17,630][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 530, reason: ApplyCommitRequest{term=2, version=530, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:33,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 532, reason: ApplyCommitRequest{term=2, version=532, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:33,393][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,401][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,409][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,509][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,600][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,608][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,694][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,704][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,795][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,802][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T18:52:33,907][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,003][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,010][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,018][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,205][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,396][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,404][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,408][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T18:52:34,982][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 533, reason: ApplyCommitRequest{term=2, version=533, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:53:58,969][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T18:53:58,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 602, reason: becoming candidate: onLeaderFailure
[2022-10-14T18:53:59,352][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 604, reason: ApplyCommitRequest{term=2, version=604, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:54:25,595][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][14539] overhead, spent [279ms] collecting in the last [1s]
[2022-10-14T19:52:03,321][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T19:52:03,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 671, reason: becoming candidate: onLeaderFailure
[2022-10-14T19:52:03,805][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 672, reason: ApplyCommitRequest{term=2, version=672, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:17,582][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 673, reason: ApplyCommitRequest{term=2, version=673, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:19,740][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 675, reason: ApplyCommitRequest{term=2, version=675, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:35,478][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 676, reason: ApplyCommitRequest{term=2, version=676, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:35,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,599][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,700][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,710][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,796][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,804][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,899][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T19:52:35,997][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,098][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,101][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,296][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,308][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,402][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,414][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,505][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,594][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,605][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,793][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,895][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T19:52:36,997][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:37,006][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T19:52:37,095][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T19:52:37,962][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 677, reason: ApplyCommitRequest{term=2, version=677, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:54:01,155][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T19:54:01,163][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 744, reason: becoming candidate: onLeaderFailure
[2022-10-14T19:54:04,017][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 750, reason: ApplyCommitRequest{term=2, version=750, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:03,338][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T20:52:03,343][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 819, reason: becoming candidate: onLeaderFailure
[2022-10-14T20:52:03,621][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 820, reason: ApplyCommitRequest{term=2, version=820, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:19,719][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 822, reason: ApplyCommitRequest{term=2, version=822, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:22,800][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 823, reason: ApplyCommitRequest{term=2, version=823, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:38,451][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 824, reason: ApplyCommitRequest{term=2, version=824, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:38,516][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,598][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,610][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,703][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,797][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,806][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,906][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,908][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T20:52:38,918][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,107][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,201][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,299][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,399][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,408][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,697][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,709][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,910][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,912][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T20:52:39,928][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T20:52:40,583][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 825, reason: ApplyCommitRequest{term=2, version=825, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:54:06,145][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T20:54:06,152][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 894, reason: becoming candidate: onLeaderFailure
[2022-10-14T20:54:06,526][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 896, reason: ApplyCommitRequest{term=2, version=896, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:03,360][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T21:52:03,364][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 966, reason: becoming candidate: onLeaderFailure
[2022-10-14T21:52:03,500][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 967, reason: ApplyCommitRequest{term=2, version=967, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:22,721][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 969, reason: ApplyCommitRequest{term=2, version=969, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:25,768][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 970, reason: ApplyCommitRequest{term=2, version=970, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:41,124][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 971, reason: ApplyCommitRequest{term=2, version=971, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:41,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,203][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,304][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,401][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,404][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,498][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,509][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,615][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,706][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,793][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,807][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,817][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,897][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,909][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,914][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:41,995][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T21:52:42,003][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:42,018][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:42,026][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T21:52:43,404][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 972, reason: ApplyCommitRequest{term=2, version=972, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:54:08,873][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T21:54:08,882][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1040, reason: becoming candidate: onLeaderFailure
[2022-10-14T21:54:09,341][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1042, reason: ApplyCommitRequest{term=2, version=1042, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:03,381][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T22:52:03,385][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1113, reason: becoming candidate: onLeaderFailure
[2022-10-14T22:52:03,750][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1114, reason: ApplyCommitRequest{term=2, version=1114, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:25,776][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1116, reason: ApplyCommitRequest{term=2, version=1116, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:28,719][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1117, reason: ApplyCommitRequest{term=2, version=1117, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:43,717][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1118, reason: ApplyCommitRequest{term=2, version=1118, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:43,995][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,097][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,109][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,205][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,218][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,307][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,405][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,513][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,524][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,609][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,693][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,697][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,720][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,809][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:44,995][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T22:52:45,005][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T22:52:46,537][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1119, reason: ApplyCommitRequest{term=2, version=1119, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:54:11,243][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T22:54:11,252][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1187, reason: becoming candidate: onLeaderFailure
[2022-10-14T22:54:12,854][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1191, reason: ApplyCommitRequest{term=2, version=1191, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:03,411][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T23:52:03,415][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1261, reason: becoming candidate: onLeaderFailure
[2022-10-14T23:52:03,714][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1262, reason: ApplyCommitRequest{term=2, version=1262, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:28,690][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1264, reason: ApplyCommitRequest{term=2, version=1264, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:31,777][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1265, reason: ApplyCommitRequest{term=2, version=1265, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:46,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1266, reason: ApplyCommitRequest{term=2, version=1266, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:46,793][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:46,795][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T23:52:46,893][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T23:52:46,899][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,008][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,198][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,199][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,305][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,311][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,402][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,598][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,609][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,619][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,701][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,718][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,800][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,811][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-14T23:52:47,903][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-14T23:52:49,080][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1267, reason: ApplyCommitRequest{term=2, version=1267, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:54:15,168][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T23:54:15,184][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1336, reason: becoming candidate: onLeaderFailure
[2022-10-14T23:54:15,642][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1338, reason: ApplyCommitRequest{term=2, version=1338, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:03,423][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T00:52:03,698][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1405, reason: becoming candidate: onLeaderFailure
[2022-10-15T00:52:04,127][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1406, reason: ApplyCommitRequest{term=2, version=1406, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:32,117][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1408, reason: ApplyCommitRequest{term=2, version=1408, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:34,811][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1409, reason: ApplyCommitRequest{term=2, version=1409, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:49,361][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1410, reason: ApplyCommitRequest{term=2, version=1410, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:49,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,496][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,596][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,598][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,606][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,708][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,799][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,807][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,898][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,899][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,912][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T00:52:49,920][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,008][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,198][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,210][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,305][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,401][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,409][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T00:52:50,502][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T00:52:51,064][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1411, reason: ApplyCommitRequest{term=2, version=1411, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:54:17,607][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T00:54:17,615][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1478, reason: becoming candidate: onLeaderFailure
[2022-10-15T00:54:17,937][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1480, reason: ApplyCommitRequest{term=2, version=1480, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:03,820][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T01:52:03,897][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1552, reason: becoming candidate: onLeaderFailure
[2022-10-15T01:52:04,198][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1553, reason: ApplyCommitRequest{term=2, version=1553, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:34,781][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1555, reason: ApplyCommitRequest{term=2, version=1555, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:37,854][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1556, reason: ApplyCommitRequest{term=2, version=1556, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:51,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1557, reason: ApplyCommitRequest{term=2, version=1557, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:51,598][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,606][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,701][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,716][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,796][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,893][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:51,994][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,095][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,195][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,315][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,403][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,505][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,594][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,599][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,609][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,705][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T01:52:52,709][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T01:52:53,472][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1558, reason: ApplyCommitRequest{term=2, version=1558, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:54:20,587][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T01:54:20,595][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1626, reason: becoming candidate: onLeaderFailure
[2022-10-15T01:54:20,950][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1628, reason: ApplyCommitRequest{term=2, version=1628, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:03,914][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T02:52:03,918][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1697, reason: becoming candidate: onLeaderFailure
[2022-10-15T02:52:04,084][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1698, reason: ApplyCommitRequest{term=2, version=1698, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:37,877][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1700, reason: ApplyCommitRequest{term=2, version=1700, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:39,974][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1701, reason: ApplyCommitRequest{term=2, version=1701, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:53,760][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1702, reason: ApplyCommitRequest{term=2, version=1702, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:53,894][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T02:52:53,902][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,011][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,103][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,110][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,205][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,297][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,404][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,413][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,504][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,518][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,609][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,696][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,705][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,715][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,804][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T02:52:54,814][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T02:52:56,267][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1703, reason: ApplyCommitRequest{term=2, version=1703, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:54:23,606][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T02:54:23,621][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1771, reason: becoming candidate: onLeaderFailure
[2022-10-15T02:54:23,976][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1773, reason: ApplyCommitRequest{term=2, version=1773, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:03,947][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T03:52:03,954][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1843, reason: becoming candidate: onLeaderFailure
[2022-10-15T03:52:04,222][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1844, reason: ApplyCommitRequest{term=2, version=1844, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:40,001][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1846, reason: ApplyCommitRequest{term=2, version=1846, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:42,808][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1847, reason: ApplyCommitRequest{term=2, version=1847, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:56,560][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1848, reason: ApplyCommitRequest{term=2, version=1848, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:56,618][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,697][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,708][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,796][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,805][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,898][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,908][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T03:52:56,995][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,005][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,013][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,104][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,204][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,213][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,297][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,313][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,397][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,399][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,504][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,596][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T03:52:57,604][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T03:52:58,876][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1849, reason: ApplyCommitRequest{term=2, version=1849, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:54:26,582][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T03:54:26,600][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1916, reason: becoming candidate: onLeaderFailure
[2022-10-15T03:54:27,900][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1920, reason: ApplyCommitRequest{term=2, version=1920, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:03,968][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T04:52:03,977][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1991, reason: becoming candidate: onLeaderFailure
[2022-10-15T04:52:04,322][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1992, reason: ApplyCommitRequest{term=2, version=1992, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:42,913][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1994, reason: ApplyCommitRequest{term=2, version=1994, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:45,842][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 1995, reason: ApplyCommitRequest{term=2, version=1995, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:59,071][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1996, reason: ApplyCommitRequest{term=2, version=1996, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:59,119][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,202][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,215][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,309][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,394][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,400][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,406][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,498][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,509][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,611][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,705][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,707][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,902][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T04:52:59,996][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:53:00,006][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T04:53:00,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:53:00,106][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T04:53:00,300][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T04:53:00,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T04:53:00,828][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1997, reason: ApplyCommitRequest{term=2, version=1997, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:54:29,821][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T04:54:29,895][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2065, reason: becoming candidate: onLeaderFailure
[2022-10-15T04:54:32,034][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2069, reason: ApplyCommitRequest{term=2, version=2069, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:52:03,995][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T05:52:03,998][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2139, reason: becoming candidate: onLeaderFailure
[2022-10-15T05:52:04,317][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2140, reason: ApplyCommitRequest{term=2, version=2140, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:52:45,853][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2142, reason: ApplyCommitRequest{term=2, version=2142, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:52:48,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2143, reason: ApplyCommitRequest{term=2, version=2143, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:53:00,913][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2144, reason: ApplyCommitRequest{term=2, version=2144, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:53:00,996][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,003][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,106][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,196][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,206][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,309][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,396][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,405][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,412][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,499][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,507][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,514][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,597][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,696][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,795][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,894][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,902][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T05:53:01,993][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:02,104][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T05:53:02,105][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T05:53:03,703][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2145, reason: ApplyCommitRequest{term=2, version=2145, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:54:34,872][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T05:54:34,882][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2213, reason: becoming candidate: onLeaderFailure
[2022-10-15T05:54:35,247][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2215, reason: ApplyCommitRequest{term=2, version=2215, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:52:04,014][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T06:52:04,021][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2284, reason: becoming candidate: onLeaderFailure
[2022-10-15T06:52:04,379][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2285, reason: ApplyCommitRequest{term=2, version=2285, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:52:48,088][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2287, reason: ApplyCommitRequest{term=2, version=2287, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:52:50,901][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2288, reason: ApplyCommitRequest{term=2, version=2288, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:53:03,860][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2289, reason: ApplyCommitRequest{term=2, version=2289, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:53:03,914][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:03,995][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,105][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,308][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,402][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,417][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,499][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,510][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,614][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,625][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,712][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,800][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,802][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,895][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,897][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,996][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T06:53:04,999][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T06:53:05,581][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2290, reason: ApplyCommitRequest{term=2, version=2290, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:54:37,901][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T06:54:37,908][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2358, reason: becoming candidate: onLeaderFailure
[2022-10-15T06:54:38,288][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2360, reason: ApplyCommitRequest{term=2, version=2360, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:54:58,202][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][57748] overhead, spent [289ms] collecting in the last [1s]
[2022-10-15T07:52:04,031][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T07:52:04,037][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2429, reason: becoming candidate: onLeaderFailure
[2022-10-15T07:52:04,208][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2430, reason: ApplyCommitRequest{term=2, version=2430, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:52:50,924][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2432, reason: ApplyCommitRequest{term=2, version=2432, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:52:53,933][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2433, reason: ApplyCommitRequest{term=2, version=2433, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:53:05,745][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2434, reason: ApplyCommitRequest{term=2, version=2434, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:53:05,811][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T07:53:05,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T07:53:05,911][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:05,995][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,008][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,016][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,102][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,196][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,199][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,208][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,214][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,314][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,320][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,334][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,402][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,411][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,505][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,512][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,520][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,600][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,615][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T07:53:06,625][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T07:53:07,873][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2435, reason: ApplyCommitRequest{term=2, version=2435, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:54:40,718][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T07:54:40,731][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2503, reason: becoming candidate: onLeaderFailure
[2022-10-15T07:54:41,373][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2505, reason: ApplyCommitRequest{term=2, version=2505, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:52:04,048][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T08:52:04,052][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2576, reason: becoming candidate: onLeaderFailure
[2022-10-15T08:52:04,217][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2577, reason: ApplyCommitRequest{term=2, version=2577, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:52:53,948][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2579, reason: ApplyCommitRequest{term=2, version=2579, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:52:56,970][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2580, reason: ApplyCommitRequest{term=2, version=2580, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:53:08,639][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2581, reason: ApplyCommitRequest{term=2, version=2581, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:53:08,668][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,704][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,713][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,720][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,810][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,811][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,898][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T08:53:08,914][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,023][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,108][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,115][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,206][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,294][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,307][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,311][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,405][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,410][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,500][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,506][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T08:53:09,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2582, reason: ApplyCommitRequest{term=2, version=2582, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:54:43,689][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T08:54:43,697][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2651, reason: becoming candidate: onLeaderFailure
[2022-10-15T08:54:44,143][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2653, reason: ApplyCommitRequest{term=2, version=2653, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:52:04,068][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T09:52:04,071][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2723, reason: becoming candidate: onLeaderFailure
[2022-10-15T09:52:04,373][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2724, reason: ApplyCommitRequest{term=2, version=2724, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:52:56,951][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2726, reason: ApplyCommitRequest{term=2, version=2726, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:52:59,985][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2727, reason: ApplyCommitRequest{term=2, version=2727, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:53:10,250][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2728, reason: ApplyCommitRequest{term=2, version=2728, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:53:10,313][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,408][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,414][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,505][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,515][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,598][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,605][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,694][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,709][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,718][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,802][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,896][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:10,904][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,102][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,106][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,202][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,297][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T09:53:11,309][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T09:53:12,502][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2729, reason: ApplyCommitRequest{term=2, version=2729, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:54:45,866][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T09:54:45,872][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2798, reason: becoming candidate: onLeaderFailure
[2022-10-15T09:54:47,137][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2802, reason: ApplyCommitRequest{term=2, version=2802, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:52:04,081][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T10:52:04,086][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2872, reason: becoming candidate: onLeaderFailure
[2022-10-15T10:52:04,327][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2873, reason: ApplyCommitRequest{term=2, version=2873, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:00,010][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2875, reason: ApplyCommitRequest{term=2, version=2875, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:03,046][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 2876, reason: ApplyCommitRequest{term=2, version=2876, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:12,524][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2877, reason: ApplyCommitRequest{term=2, version=2877, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:12,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:12,794][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T10:53:12,806][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T10:53:12,895][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T10:53:12,901][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T10:53:12,998][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,011][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,107][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,193][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,199][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,213][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,295][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,307][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,310][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,411][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,424][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,497][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,503][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,594][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T10:53:13,602][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T10:53:14,569][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2878, reason: ApplyCommitRequest{term=2, version=2878, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:54:49,801][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T10:54:49,812][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2946, reason: becoming candidate: onLeaderFailure
[2022-10-15T10:54:50,310][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2948, reason: ApplyCommitRequest{term=2, version=2948, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:52:04,099][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T11:52:04,108][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3017, reason: becoming candidate: onLeaderFailure
[2022-10-15T11:52:04,404][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3018, reason: ApplyCommitRequest{term=2, version=3018, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:03,035][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3020, reason: ApplyCommitRequest{term=2, version=3020, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:06,023][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3021, reason: ApplyCommitRequest{term=2, version=3021, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:14,724][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3022, reason: ApplyCommitRequest{term=2, version=3022, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:14,822][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:14,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:14,909][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T11:53:14,998][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,008][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,097][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,110][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,196][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,204][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,221][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,300][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,311][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,322][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,399][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,400][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,495][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,606][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,706][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T11:53:15,719][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T11:53:17,075][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3023, reason: ApplyCommitRequest{term=2, version=3023, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:54:52,826][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T11:54:52,836][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3092, reason: becoming candidate: onLeaderFailure
[2022-10-15T11:54:53,310][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3094, reason: ApplyCommitRequest{term=2, version=3094, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:52:04,125][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T12:52:04,128][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3162, reason: becoming candidate: onLeaderFailure
[2022-10-15T12:52:04,400][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3163, reason: ApplyCommitRequest{term=2, version=3163, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:06,010][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3165, reason: ApplyCommitRequest{term=2, version=3165, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:09,085][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3166, reason: ApplyCommitRequest{term=2, version=3166, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:17,362][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3167, reason: ApplyCommitRequest{term=2, version=3167, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:17,407][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,498][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,503][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,602][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,694][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,702][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,796][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,810][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,893][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,905][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,916][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T12:53:17,999][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,010][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,096][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,104][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,203][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,223][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,303][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T12:53:18,400][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T12:53:19,812][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3168, reason: ApplyCommitRequest{term=2, version=3168, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:54:55,873][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T12:54:55,882][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3237, reason: becoming candidate: onLeaderFailure
[2022-10-15T12:54:56,307][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3239, reason: ApplyCommitRequest{term=2, version=3239, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:55:16,599][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][79355] overhead, spent [416ms] collecting in the last [1s]
[2022-10-15T13:52:04,135][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T13:52:04,140][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3308, reason: becoming candidate: onLeaderFailure
[2022-10-15T13:52:04,236][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3309, reason: ApplyCommitRequest{term=2, version=3309, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:09,066][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3311, reason: ApplyCommitRequest{term=2, version=3311, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:12,216][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3312, reason: ApplyCommitRequest{term=2, version=3312, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:19,882][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3313, reason: ApplyCommitRequest{term=2, version=3313, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:19,999][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,112][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,195][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,202][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,296][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,499][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,594][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,694][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,702][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,796][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,805][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,812][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,819][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,894][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,907][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,918][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T13:53:20,926][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T13:53:22,678][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3314, reason: ApplyCommitRequest{term=2, version=3314, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:54:58,880][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T13:54:58,904][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3381, reason: becoming candidate: onLeaderFailure
[2022-10-15T13:54:59,810][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3383, reason: ApplyCommitRequest{term=2, version=3383, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:52:04,152][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T14:52:04,157][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3450, reason: becoming candidate: onLeaderFailure
[2022-10-15T14:52:04,497][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3451, reason: ApplyCommitRequest{term=2, version=3451, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:12,127][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3453, reason: ApplyCommitRequest{term=2, version=3453, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:14,281][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3454, reason: ApplyCommitRequest{term=2, version=3454, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:23,044][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3455, reason: ApplyCommitRequest{term=2, version=3455, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:23,115][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,121][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,197][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,296][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,319][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,397][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,402][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,503][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,509][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,603][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,709][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,794][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,816][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,822][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,909][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T14:53:23,913][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T14:53:24,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T14:53:24,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T14:53:24,820][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3456, reason: ApplyCommitRequest{term=2, version=3456, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:55:01,013][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T14:55:01,023][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3524, reason: becoming candidate: onLeaderFailure
[2022-10-15T14:55:02,808][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3528, reason: ApplyCommitRequest{term=2, version=3528, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:52:04,166][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T15:52:04,169][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3597, reason: becoming candidate: onLeaderFailure
[2022-10-15T15:52:04,235][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3598, reason: ApplyCommitRequest{term=2, version=3598, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:14,306][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3600, reason: ApplyCommitRequest{term=2, version=3600, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:17,198][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3601, reason: ApplyCommitRequest{term=2, version=3601, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:24,995][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3602, reason: ApplyCommitRequest{term=2, version=3602, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:25,101][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,106][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,127][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,208][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,213][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,298][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,310][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,406][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,413][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,500][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,512][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,599][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,613][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,696][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,697][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,720][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,799][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,808][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,819][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,904][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T15:53:25,915][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T15:53:27,481][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3603, reason: ApplyCommitRequest{term=2, version=3603, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:55:04,733][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T15:55:04,741][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3670, reason: becoming candidate: onLeaderFailure
[2022-10-15T15:55:05,997][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3672, reason: ApplyCommitRequest{term=2, version=3672, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:52:04,178][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T16:52:04,181][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3743, reason: becoming candidate: onLeaderFailure
[2022-10-15T16:52:04,322][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3744, reason: ApplyCommitRequest{term=2, version=3744, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:17,185][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3746, reason: ApplyCommitRequest{term=2, version=3746, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:20,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3747, reason: ApplyCommitRequest{term=2, version=3747, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:28,194][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3748, reason: ApplyCommitRequest{term=2, version=3748, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:28,403][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:28,501][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T16:53:28,593][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T16:53:28,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T16:53:28,707][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T16:53:28,803][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:28,902][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,007][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,103][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,293][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,394][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,795][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T16:53:29,894][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,004][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,193][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,205][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,303][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,506][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,597][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T16:53:30,606][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3749, reason: ApplyCommitRequest{term=2, version=3749, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:30,905][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3750, reason: ApplyCommitRequest{term=2, version=3750, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:31,893][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3751, reason: ApplyCommitRequest{term=2, version=3751, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:55:07,851][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T16:55:07,857][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3821, reason: becoming candidate: onLeaderFailure
[2022-10-15T16:55:08,268][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3823, reason: ApplyCommitRequest{term=2, version=3823, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:55:23,594][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][93754] overhead, spent [276ms] collecting in the last [1s]
[2022-10-15T17:52:04,195][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T17:52:04,201][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3892, reason: becoming candidate: onLeaderFailure
[2022-10-15T17:52:04,505][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3893, reason: ApplyCommitRequest{term=2, version=3893, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:20,170][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3895, reason: ApplyCommitRequest{term=2, version=3895, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:23,240][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 3896, reason: ApplyCommitRequest{term=2, version=3896, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:31,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3897, reason: ApplyCommitRequest{term=2, version=3897, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:31,912][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:31,919][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,010][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,108][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,200][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,210][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,297][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,304][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,394][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,398][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,412][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,496][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,500][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,602][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,696][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,795][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,803][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:32,913][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T17:53:33,005][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T17:53:33,731][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3898, reason: ApplyCommitRequest{term=2, version=3898, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:55:10,208][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T17:55:10,215][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3964, reason: becoming candidate: onLeaderFailure
[2022-10-15T17:55:10,701][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3966, reason: ApplyCommitRequest{term=2, version=3966, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:52:04,208][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T18:52:04,211][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4035, reason: becoming candidate: onLeaderFailure
[2022-10-15T18:52:04,419][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4036, reason: ApplyCommitRequest{term=2, version=4036, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:23,242][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4038, reason: ApplyCommitRequest{term=2, version=4038, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:26,434][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4039, reason: ApplyCommitRequest{term=2, version=4039, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:33,896][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4040, reason: ApplyCommitRequest{term=2, version=4040, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:34,003][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,008][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,094][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,108][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,119][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,204][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,204][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,219][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,303][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,397][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,508][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,699][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,793][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,800][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:34,999][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T18:53:35,097][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:35,298][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T18:53:35,301][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T18:53:35,310][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T18:53:36,686][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4041, reason: ApplyCommitRequest{term=2, version=4041, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:55:12,958][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T18:55:12,966][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4109, reason: becoming candidate: onLeaderFailure
[2022-10-15T18:55:13,386][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4111, reason: ApplyCommitRequest{term=2, version=4111, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:52:04,221][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T19:52:04,225][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4181, reason: becoming candidate: onLeaderFailure
[2022-10-15T19:52:04,342][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4182, reason: ApplyCommitRequest{term=2, version=4182, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:26,268][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4184, reason: ApplyCommitRequest{term=2, version=4184, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:29,257][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4185, reason: ApplyCommitRequest{term=2, version=4185, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:36,925][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4186, reason: ApplyCommitRequest{term=2, version=4186, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:37,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,007][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,013][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,098][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,108][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,114][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,198][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,298][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,306][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,313][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,400][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,411][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,419][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,502][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,509][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,522][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,602][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,613][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,695][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,705][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,710][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T19:53:37,795][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T19:53:39,384][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4187, reason: ApplyCommitRequest{term=2, version=4187, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:55:15,925][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T19:55:15,934][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4255, reason: becoming candidate: onLeaderFailure
[2022-10-15T19:55:16,385][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4257, reason: ApplyCommitRequest{term=2, version=4257, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:52:04,244][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T20:52:04,248][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4324, reason: becoming candidate: onLeaderFailure
[2022-10-15T20:52:04,335][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4325, reason: ApplyCommitRequest{term=2, version=4325, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:29,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4327, reason: ApplyCommitRequest{term=2, version=4327, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:32,309][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4328, reason: ApplyCommitRequest{term=2, version=4328, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:39,689][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4329, reason: ApplyCommitRequest{term=2, version=4329, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:39,719][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:39,724][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:39,810][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T20:53:39,817][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:39,896][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:39,994][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,006][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,106][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,108][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,194][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,204][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,214][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,309][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,317][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,399][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,404][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,416][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,501][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,510][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:40,610][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T20:53:42,120][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4330, reason: ApplyCommitRequest{term=2, version=4330, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:55:18,797][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T20:55:18,804][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4398, reason: becoming candidate: onLeaderFailure
[2022-10-15T20:55:19,393][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4400, reason: ApplyCommitRequest{term=2, version=4400, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:52:04,256][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T21:52:04,266][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4469, reason: becoming candidate: onLeaderFailure
[2022-10-15T21:52:04,618][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4470, reason: ApplyCommitRequest{term=2, version=4470, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:32,414][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4472, reason: ApplyCommitRequest{term=2, version=4472, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:35,332][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}}, term: 2, version: 4473, reason: ApplyCommitRequest{term=2, version=4473, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:42,171][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4474, reason: ApplyCommitRequest{term=2, version=4474, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:42,209][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,298][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,310][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,395][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,405][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,497][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,505][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,513][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,595][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,601][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,698][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,894][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:42,999][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,008][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,107][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,116][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,199][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,223][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,396][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][1] primary-replica resync completed with 0 operations
[2022-10-15T21:53:43,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations
[2022-10-15T21:53:45,023][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4475, reason: ApplyCommitRequest{term=2, version=4475, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:55:21,226][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.Ab