By type
[38;5;6m [38;5;5m23:15:56.36 [0m
[38;5;6m [38;5;5m23:15:56.36 [0m[1mWelcome to the Bitnami elasticsearch container[0m
[38;5;6m [38;5;5m23:15:56.45 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m
[38;5;6m [38;5;5m23:15:56.45 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m
[38;5;6m [38;5;5m23:15:56.46 [0m
[38;5;6m [38;5;5m23:15:56.47 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup **
[38;5;6m [38;5;5m23:15:56.76 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch...
[38;5;6m [38;5;5m23:15:57.06 [0m[38;5;2mINFO [0m ==> Setting default configuration
[38;5;6m [38;5;5m23:15:57.17 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings...
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[38;5;6m [38;5;5m23:16:16.45 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! **
[38;5;6m [38;5;5m23:16:16.56 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2023-01-23T23:16:55,848][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS]
[2023-01-23T23:16:55,851][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] JVM home [/opt/bitnami/java]
[2023-01-23T23:16:55,949][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-14760059311962475914, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2023-01-23T23:17:13,954][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [aggs-matrix-stats]
[2023-01-23T23:17:13,955][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [analysis-common]
[2023-01-23T23:17:13,955][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [geo]
[2023-01-23T23:17:13,956][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [ingest-common]
[2023-01-23T23:17:13,956][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [ingest-geoip]
[2023-01-23T23:17:13,957][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [ingest-user-agent]
[2023-01-23T23:17:13,957][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [kibana]
[2023-01-23T23:17:13,957][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [lang-expression]
[2023-01-23T23:17:13,958][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [lang-mustache]
[2023-01-23T23:17:14,046][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [lang-painless]
[2023-01-23T23:17:14,047][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [mapper-extras]
[2023-01-23T23:17:14,048][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [parent-join]
[2023-01-23T23:17:14,048][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [percolator]
[2023-01-23T23:17:14,048][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [rank-eval]
[2023-01-23T23:17:14,049][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [reindex]
[2023-01-23T23:17:14,049][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [repository-url]
[2023-01-23T23:17:14,049][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [tasks]
[2023-01-23T23:17:14,050][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded module [transport-netty4]
[2023-01-23T23:17:14,051][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-0] loaded plugin [repository-s3]
[2023-01-23T23:17:14,849][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-0] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.13.180:/dockerdata-nfs/onap/elastic-master-1)]], net usable_space [95.4gb], net total_space [99.9gb], types [nfs4]
[2023-01-23T23:17:14,850][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-0] heap size [123.7mb], compressed ordinary object pointers [true]
[2023-01-23T23:17:15,650][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] node name [onap-sdnrdb-master-0], node ID [GJGug0r1SpqSVP5pIZvzbg], cluster name [sdnrdb-cluster]
[2023-01-23T23:18:04,955][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-master-0] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}]
[2023-01-23T23:18:06,049][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-master-0] using discovery type [zen] and seed hosts providers [settings]
[2023-01-23T23:18:09,848][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-01-23T23:18:12,850][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] initialized
[2023-01-23T23:18:12,851][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] starting ...
[2023-01-23T23:18:14,953][INFO ][o.e.t.TransportService ] [onap-sdnrdb-master-0] publish_address {10.233.67.239:9300}, bound_addresses {0.0.0.0:9300}
[2023-01-23T23:18:16,349][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.239:9300, remoteAddress=/127.0.0.6:56231}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-01-23T23:18:16,647][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.239:9300, remoteAddress=/127.0.0.6:50463}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-01-23T23:18:17,649][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.239:9300, remoteAddress=/127.0.0.6:33573}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-01-23T23:18:18,255][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-23T23:18:28,460][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}]; discovery will continue using [10.233.65.115:9300, 10.233.69.48:9300, 10.233.72.143:9300] from hosts providers and [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-01-23T23:18:38,464][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}]; discovery will continue using [10.233.65.115:9300, 10.233.69.48:9300, 10.233.72.143:9300] from hosts providers and [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-01-23T23:18:41,391][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] setting initial configuration to VotingConfiguration{DDUyZf3zRMuBIsKEfy6avg,{bootstrap-placeholder}-onap-sdnrdb-master-2,GJGug0r1SpqSVP5pIZvzbg}
[2023-01-23T23:18:44,449][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-0] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr} elect leader, {onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-23T23:18:44,549][WARN ][o.e.c.s.MasterService ] [onap-sdnrdb-master-0] failing [elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr} elect leader, {onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 1 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-01-23T23:18:44,554][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-0] failed to join {onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.239:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 1 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-01-23T23:18:45,748][INFO ][o.e.c.c.CoordinationState] [onap-sdnrdb-master-0] cluster UUID set to [UEEU1zaHTfuaLwra2e7BVA]
[2023-01-23T23:18:46,163][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}, term: 2, version: 1, reason: ApplyCommitRequest{term=2, version=1, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-23T23:18:46,250][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-0] publish_address {10.233.67.239:9200}, bound_addresses {0.0.0.0:9200}
[2023-01-23T23:18:46,251][INFO ][o.e.n.Node ] [onap-sdnrdb-master-0] started
[2023-01-23T23:18:47,190][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 2, reason: ApplyCommitRequest{term=2, version=2, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-23T23:19:21,500][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-master-0] updating [action.auto_create_index] from [true] to [false]
[2023-01-23T23:20:02,797][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 41, reason: ApplyCommitRequest{term=2, version=41, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:18:41,215][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T00:18:41,549][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 137, reason: becoming candidate: onLeaderFailure
[2023-01-24T00:18:50,873][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 142, reason: ApplyCommitRequest{term=2, version=142, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:18:51,691][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 143, reason: ApplyCommitRequest{term=2, version=143, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:32,989][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 210, reason: ApplyCommitRequest{term=2, version=210, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:33,254][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,350][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,352][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,552][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,747][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,849][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,862][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:33,959][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,048][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,062][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,155][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,253][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,268][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,357][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,361][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,547][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,559][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,752][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,759][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:34,949][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T00:21:35,050][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T00:21:35,147][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T00:21:36,298][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 211, reason: ApplyCommitRequest{term=2, version=211, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:36,600][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 212, reason: ApplyCommitRequest{term=2, version=212, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:36,776][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 213, reason: ApplyCommitRequest{term=2, version=213, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:37,191][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 214, reason: ApplyCommitRequest{term=2, version=214, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:37,904][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 215, reason: ApplyCommitRequest{term=2, version=215, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T00:21:59,959][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-0] [gc][3824] overhead, spent [478ms] collecting in the last [1s]
[2023-01-24T01:18:42,071][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T01:18:42,078][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 282, reason: becoming candidate: onLeaderFailure
[2023-01-24T01:18:42,557][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 283, reason: ApplyCommitRequest{term=2, version=283, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T01:18:51,583][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T01:18:51,597][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 283, reason: becoming candidate: onLeaderFailure
[2023-01-24T01:18:55,085][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 288, reason: ApplyCommitRequest{term=2, version=288, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T01:18:57,247][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 290, reason: ApplyCommitRequest{term=2, version=290, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T01:21:38,491][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 357, reason: ApplyCommitRequest{term=2, version=357, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T01:21:38,547][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,568][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,659][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,667][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,848][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,857][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,952][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:38,961][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,055][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,060][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,069][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,248][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,257][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,263][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,349][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,358][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,456][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,465][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,552][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,558][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T01:21:39,666][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T01:21:40,865][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 358, reason: ApplyCommitRequest{term=2, version=358, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T02:18:42,149][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T02:18:42,154][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 427, reason: becoming candidate: onLeaderFailure
[2023-01-24T02:18:42,412][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 428, reason: ApplyCommitRequest{term=2, version=428, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T02:18:57,458][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T02:18:57,472][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 428, reason: becoming candidate: onLeaderFailure
[2023-01-24T02:18:58,347][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 431, reason: ApplyCommitRequest{term=2, version=431, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T02:19:00,347][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 433, reason: ApplyCommitRequest{term=2, version=433, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T02:19:28,648][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-0] [gc][10869] overhead, spent [426ms] collecting in the last [1.1s]
[2023-01-24T02:21:41,658][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 501, reason: ApplyCommitRequest{term=2, version=501, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T02:21:41,846][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:41,860][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:41,970][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,047][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,072][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,154][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,247][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,256][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,349][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,361][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,447][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,550][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,554][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,565][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,660][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,670][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,758][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,771][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,847][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,860][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T02:21:42,879][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T02:21:43,597][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 502, reason: ApplyCommitRequest{term=2, version=502, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T03:18:42,269][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T03:18:42,279][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 572, reason: becoming candidate: onLeaderFailure
[2023-01-24T03:18:42,587][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 573, reason: ApplyCommitRequest{term=2, version=573, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T03:19:00,630][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T03:19:00,640][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 573, reason: becoming candidate: onLeaderFailure
[2023-01-24T03:19:03,092][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 578, reason: ApplyCommitRequest{term=2, version=578, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T03:19:04,447][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 580, reason: ApplyCommitRequest{term=2, version=580, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T03:21:44,561][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 648, reason: ApplyCommitRequest{term=2, version=648, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T03:21:44,656][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:44,746][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:44,759][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:44,861][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:44,871][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:44,961][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T03:21:44,969][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,148][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,266][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,412][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,444][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,459][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,475][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,566][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,571][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,686][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,772][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T03:21:45,850][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T03:21:46,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T03:21:46,053][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T03:21:46,354][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 649, reason: ApplyCommitRequest{term=2, version=649, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T04:18:42,370][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T04:18:42,378][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 718, reason: becoming candidate: onLeaderFailure
[2023-01-24T04:18:42,585][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 719, reason: ApplyCommitRequest{term=2, version=719, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T04:19:05,641][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T04:19:05,652][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 719, reason: becoming candidate: onLeaderFailure
[2023-01-24T04:19:07,993][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 724, reason: ApplyCommitRequest{term=2, version=724, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T04:19:09,647][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 726, reason: ApplyCommitRequest{term=2, version=726, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T04:21:46,643][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 794, reason: ApplyCommitRequest{term=2, version=794, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T04:21:46,748][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T04:21:46,859][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:46,948][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:46,961][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:46,969][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,060][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,148][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,158][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,167][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,260][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,261][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,363][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,450][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,461][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,467][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,478][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,556][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,565][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,652][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T04:21:47,772][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T04:21:48,954][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 795, reason: ApplyCommitRequest{term=2, version=795, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T05:18:42,469][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T05:18:42,474][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 865, reason: becoming candidate: onLeaderFailure
[2023-01-24T05:18:42,866][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 866, reason: ApplyCommitRequest{term=2, version=866, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T05:19:09,932][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T05:19:09,942][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 866, reason: becoming candidate: onLeaderFailure
[2023-01-24T05:19:11,285][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 871, reason: ApplyCommitRequest{term=2, version=871, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T05:19:12,958][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 873, reason: ApplyCommitRequest{term=2, version=873, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T05:21:49,469][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 939, reason: ApplyCommitRequest{term=2, version=939, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T05:21:49,555][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:49,565][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T05:21:49,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:49,661][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:49,750][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:49,848][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:49,948][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,147][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,153][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,154][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,265][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,356][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,369][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,453][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,467][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,548][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,567][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,650][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,661][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,671][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T05:21:50,750][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T05:21:51,152][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 940, reason: ApplyCommitRequest{term=2, version=940, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T06:18:42,544][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T06:18:42,550][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1009, reason: becoming candidate: onLeaderFailure
[2023-01-24T06:18:42,959][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1010, reason: ApplyCommitRequest{term=2, version=1010, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T06:19:13,121][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T06:19:13,132][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1010, reason: becoming candidate: onLeaderFailure
[2023-01-24T06:19:15,488][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1015, reason: ApplyCommitRequest{term=2, version=1015, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T06:19:17,056][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1017, reason: ApplyCommitRequest{term=2, version=1017, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T06:21:51,801][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1084, reason: ApplyCommitRequest{term=2, version=1084, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T06:21:51,963][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,054][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,150][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,164][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,257][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,353][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,364][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,465][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,553][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,648][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,651][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,669][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,758][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,779][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,847][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,861][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:52,958][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T06:21:53,049][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:53,061][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:53,150][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T06:21:53,173][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T06:21:54,097][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1085, reason: ApplyCommitRequest{term=2, version=1085, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T07:18:42,573][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T07:18:42,580][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1152, reason: becoming candidate: onLeaderFailure
[2023-01-24T07:18:43,178][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1153, reason: ApplyCommitRequest{term=2, version=1153, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T07:19:17,311][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T07:19:17,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1153, reason: becoming candidate: onLeaderFailure
[2023-01-24T07:19:20,799][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1158, reason: ApplyCommitRequest{term=2, version=1158, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T07:19:22,251][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1160, reason: ApplyCommitRequest{term=2, version=1160, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T07:21:54,428][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1229, reason: ApplyCommitRequest{term=2, version=1229, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T07:21:54,551][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:54,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T07:21:54,756][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T07:21:54,771][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T07:21:54,862][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:54,865][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:54,964][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,080][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,084][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,154][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,167][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,250][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,263][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,275][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,359][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,471][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,553][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,565][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,651][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,747][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,760][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T07:21:55,769][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T07:21:56,167][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1230, reason: ApplyCommitRequest{term=2, version=1230, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T08:18:42,756][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T08:18:42,762][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1299, reason: becoming candidate: onLeaderFailure
[2023-01-24T08:18:43,161][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1300, reason: ApplyCommitRequest{term=2, version=1300, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T08:19:23,300][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T08:19:23,310][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1300, reason: becoming candidate: onLeaderFailure
[2023-01-24T08:19:24,015][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1303, reason: ApplyCommitRequest{term=2, version=1303, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T08:19:26,421][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1307, reason: ApplyCommitRequest{term=2, version=1307, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T08:21:56,721][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1372, reason: ApplyCommitRequest{term=2, version=1372, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T08:21:56,759][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:56,768][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:56,853][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:56,864][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:56,951][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:56,960][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:56,974][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,049][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,065][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,151][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,163][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,249][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,257][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,351][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,447][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,565][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,651][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,658][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,674][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,758][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T08:21:57,764][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T08:21:58,832][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1373, reason: ApplyCommitRequest{term=2, version=1373, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T09:18:42,842][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T09:18:42,852][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1441, reason: becoming candidate: onLeaderFailure
[2023-01-24T09:18:43,163][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1442, reason: ApplyCommitRequest{term=2, version=1442, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T09:19:26,268][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T09:19:26,284][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1442, reason: becoming candidate: onLeaderFailure
[2023-01-24T09:19:27,601][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1447, reason: ApplyCommitRequest{term=2, version=1447, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T09:19:29,157][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1449, reason: ApplyCommitRequest{term=2, version=1449, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T09:21:59,309][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1516, reason: ApplyCommitRequest{term=2, version=1516, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T09:21:59,358][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,372][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,458][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,551][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,565][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,653][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,659][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,755][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,759][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,852][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,857][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,954][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:21:59,963][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,047][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,058][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,068][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,146][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,157][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,167][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,176][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,251][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T09:22:00,272][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T09:22:01,659][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1517, reason: ApplyCommitRequest{term=2, version=1517, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T10:18:42,867][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T10:18:42,875][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1586, reason: becoming candidate: onLeaderFailure
[2023-01-24T10:18:43,269][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1587, reason: ApplyCommitRequest{term=2, version=1587, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T10:19:29,482][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T10:19:29,497][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1587, reason: becoming candidate: onLeaderFailure
[2023-01-24T10:19:31,942][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1592, reason: ApplyCommitRequest{term=2, version=1592, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T10:19:33,154][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1594, reason: ApplyCommitRequest{term=2, version=1594, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T10:19:54,764][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-0] [gc][39681] overhead, spent [301ms] collecting in the last [1s]
[2023-01-24T10:22:02,041][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1661, reason: ApplyCommitRequest{term=2, version=1661, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T10:22:02,147][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,158][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,250][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,264][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,369][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,462][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,562][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,658][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,747][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,759][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,787][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,847][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,859][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,947][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,970][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:02,983][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T10:22:03,053][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T10:22:03,147][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T10:22:03,148][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T10:22:03,150][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T10:22:04,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1662, reason: ApplyCommitRequest{term=2, version=1662, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T11:18:42,957][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T11:18:42,966][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1732, reason: becoming candidate: onLeaderFailure
[2023-01-24T11:18:43,466][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1733, reason: ApplyCommitRequest{term=2, version=1733, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T11:19:33,668][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T11:19:33,679][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1733, reason: becoming candidate: onLeaderFailure
[2023-01-24T11:19:36,942][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1738, reason: ApplyCommitRequest{term=2, version=1738, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T11:19:38,562][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1740, reason: ApplyCommitRequest{term=2, version=1740, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T11:22:04,768][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1806, reason: ApplyCommitRequest{term=2, version=1806, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T11:22:04,851][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:04,866][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:04,950][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:04,962][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,048][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,063][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,148][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,163][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,259][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,447][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,547][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,559][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,568][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,662][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,749][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,764][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,849][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,858][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,957][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T11:22:05,968][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T11:22:06,051][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T11:22:06,072][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T11:22:06,597][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1807, reason: ApplyCommitRequest{term=2, version=1807, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T12:18:43,054][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T12:18:43,060][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1876, reason: becoming candidate: onLeaderFailure
[2023-01-24T12:18:43,370][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 1877, reason: ApplyCommitRequest{term=2, version=1877, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T12:19:39,554][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T12:19:39,566][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 1877, reason: becoming candidate: onLeaderFailure
[2023-01-24T12:19:40,051][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1880, reason: ApplyCommitRequest{term=2, version=1880, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T12:19:42,448][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 1884, reason: ApplyCommitRequest{term=2, version=1884, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T12:20:02,356][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-0] [gc][46885] overhead, spent [291ms] collecting in the last [1s]
[2023-01-24T12:22:07,083][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1948, reason: ApplyCommitRequest{term=2, version=1948, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T12:22:07,166][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,350][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,351][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,458][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,656][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,753][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,847][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,854][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,867][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,951][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:07,960][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,051][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,074][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,088][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,153][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,159][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,248][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,260][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:08,271][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T12:22:09,499][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 1949, reason: ApplyCommitRequest{term=2, version=1949, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T13:18:43,085][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.115:9300][disconnected] disconnected
[2023-01-24T13:18:43,092][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 2017, reason: becoming candidate: onLeaderFailure
[2023-01-24T13:18:43,463][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, term: 2, version: 2018, reason: ApplyCommitRequest{term=2, version=2018, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T13:19:42,620][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-0] master node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.115:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-0}{GJGug0r1SpqSVP5pIZvzbg}{1RcGeLw-TcKtooc7ktzyXw}{10.233.67.239}{10.233.67.239:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-01-24T13:19:42,637][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}], current []}, term: 2, version: 2018, reason: becoming candidate: onLeaderFailure
[2023-01-24T13:19:45,000][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] master node changed {previous [], current [{onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 2023, reason: ApplyCommitRequest{term=2, version=2023, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T13:19:46,250][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-coordinating-only-54c85fc557-clw5j}{Nluuiri_QU-QkkKTEReL3g}{qy087GCmRgq6ZQZOmrQwUw}{10.233.69.48}{10.233.69.48:9300}{r}}, term: 2, version: 2025, reason: ApplyCommitRequest{term=2, version=2025, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T13:22:09,914][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] removed {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 2091, reason: ApplyCommitRequest{term=2, version=2091, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}
[2023-01-24T13:22:09,956][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:09,966][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [userdata-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,050][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,156][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [maintenancemode-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,247][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,446][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,458][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [historicalperformance24h-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,471][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [networkelement-connection-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,550][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,655][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,747][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,859][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [guicutthrough-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:10,949][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,052][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,249][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [eventlog-v7][2] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,448][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,546][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [cmlog-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,554][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][1] primary-replica resync completed with 0 operations
[2023-01-24T13:22:11,655][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-0] [faultcurrent-v7][3] primary-replica resync completed with 0 operations
[2023-01-24T13:22:12,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-0] added {{onap-sdnrdb-master-2}{TVrcxSuQT2KjwLCcYPny1w}{NguB91VwQteshz91brw05Q}{10.233.72.143}{10.233.72.143:9300}{dmr}}, term: 2, version: 2092, reason: ApplyCommitRequest{term=2, version=2092, sourceNode={onap-sdnrdb-master-1}{DDUyZf3zRMuBIsKEfy6avg}{kqPw9gDjS7qpHdNDKJAFIQ}{10.233.65.115}{10.233.65.115:9300}{dmr}}