Results

By type
Logs
raw version
          [38;5;6m [38;5;5m03:50:14.67 [0m
[38;5;6m [38;5;5m03:50:14.67 [0m[1mWelcome to the Bitnami elasticsearch container[0m
[38;5;6m [38;5;5m03:50:14.69 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m
[38;5;6m [38;5;5m03:50:14.76 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m
[38;5;6m [38;5;5m03:50:14.77 [0m
[38;5;6m [38;5;5m03:50:14.77 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup **
[38;5;6m [38;5;5m03:50:15.16 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch...
[38;5;6m [38;5;5m03:50:15.58 [0m[38;5;2mINFO [0m ==> Setting default configuration
[38;5;6m [38;5;5m03:50:15.77 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings...
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

[38;5;6m [38;5;5m03:50:39.68 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! **
[38;5;6m [38;5;5m03:50:39.98 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2023-04-01T03:51:32,562][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS]
[2023-04-01T03:51:32,568][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] JVM home [/opt/bitnami/java]
[2023-04-01T03:51:32,663][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-4199102135364366184, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2023-04-01T03:51:52,862][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [aggs-matrix-stats]
[2023-04-01T03:51:52,863][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [analysis-common]
[2023-04-01T03:51:52,864][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [geo]
[2023-04-01T03:51:52,864][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [ingest-common]
[2023-04-01T03:51:52,864][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [ingest-geoip]
[2023-04-01T03:51:52,864][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [ingest-user-agent]
[2023-04-01T03:51:52,880][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [kibana]
[2023-04-01T03:51:52,880][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [lang-expression]
[2023-04-01T03:51:52,880][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [lang-mustache]
[2023-04-01T03:51:52,881][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [lang-painless]
[2023-04-01T03:51:52,906][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [mapper-extras]
[2023-04-01T03:51:52,907][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [parent-join]
[2023-04-01T03:51:52,907][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [percolator]
[2023-04-01T03:51:52,907][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [rank-eval]
[2023-04-01T03:51:52,908][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [reindex]
[2023-04-01T03:51:52,908][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [repository-url]
[2023-04-01T03:51:52,908][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [tasks]
[2023-04-01T03:51:52,908][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded module [transport-netty4]
[2023-04-01T03:51:52,909][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-master-1] loaded plugin [repository-s3]
[2023-04-01T03:51:53,685][INFO ][o.e.e.NodeEnvironment    ] [onap-sdnrdb-master-1] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.5.249:/dockerdata-nfs/onap/elastic-master-2)]], net usable_space [95.1gb], net total_space [99.9gb], types [nfs4]
[2023-04-01T03:51:53,763][INFO ][o.e.e.NodeEnvironment    ] [onap-sdnrdb-master-1] heap size [123.7mb], compressed ordinary object pointers [true]
[2023-04-01T03:51:54,663][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] node name [onap-sdnrdb-master-1], node ID [oBVWcRcCQa2pTEZnFAZuXQ], cluster name [sdnrdb-cluster]
[2023-04-01T03:52:50,465][INFO ][o.e.t.NettyAllocator     ] [onap-sdnrdb-master-1] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}]
[2023-04-01T03:52:51,281][INFO ][o.e.d.DiscoveryModule    ] [onap-sdnrdb-master-1] using discovery type [zen] and seed hosts providers [settings]
[2023-04-01T03:52:55,072][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-04-01T03:52:57,767][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] initialized
[2023-04-01T03:52:57,768][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] starting ...
[2023-04-01T03:52:58,962][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][1] overhead, spent [297ms] collecting in the last [1s]
[2023-04-01T03:53:00,373][INFO ][o.e.t.TransportService   ] [onap-sdnrdb-master-1] publish_address {10.233.68.45:9300}, bound_addresses {0.0.0.0:9300}
[2023-04-01T03:53:02,365][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:35537}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:02,469][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:55643}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:02,775][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:42025}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:03,067][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:51985}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:03,775][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:50697}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:04,067][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:35981}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:04,772][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:40975}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:05,068][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.68.45:9300, remoteAddress=/127.0.0.6:40881}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:05,181][INFO ][o.e.b.BootstrapChecks    ] [onap-sdnrdb-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-04-01T03:53:08,563][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-onap-sdnrdb-master-2,oBVWcRcCQa2pTEZnFAZuXQ,UYq294gRQ3W6BCfWxYv4Rw}
[2023-04-01T03:53:13,265][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}
	at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:459) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:533) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:375) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:800) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:14,364][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][15] overhead, spent [407ms] collecting in the last [1.1s]
[2023-04-01T03:53:14,463][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=1, optionalJoin=Optional[Join{term=2, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}
	at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:459) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:533) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:375) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:800) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:15,144][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=3, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 4 does not match current term 5
	at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1013) ~[elasticsearch-7.9.3.jar:7.9.3]
	at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
	at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:532) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:120) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:378) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:15,370][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=2, optionalJoin=Optional[Join{term=3, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}
	at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:459) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:533) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:375) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:800) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:15,463][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] last failed join attempt was 207ms ago, failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=3, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: incoming term 4 does not match current term 5
	at org.elasticsearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1013) ~[elasticsearch-7.9.3.jar:7.9.3]
	at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
	at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:532) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:120) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:378) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:15,872][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=4, optionalJoin=Optional[Join{term=5, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}
	at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:459) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:533) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:375) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:800) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:16,200][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered or elected yet, an election requires 2 nodes with ids [oBVWcRcCQa2pTEZnFAZuXQ, UYq294gRQ3W6BCfWxYv4Rw], have discovered [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] which is a quorum; discovery will continue using [10.233.66.44:9300, 10.233.67.181:9300, 10.233.65.70:9300] from hosts providers and [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] from last-known cluster state; node term 7, last-accepted version 0 in term 0
[2023-04-01T03:53:16,574][INFO ][o.e.c.s.MasterService    ] [onap-sdnrdb-master-1] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 7, version: 1, delta: master node changed {previous [], current [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T03:53:16,576][WARN ][o.e.c.s.MasterService    ] [onap-sdnrdb-master-1] failing [elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 7 while handling publication
	at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:16,664][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=5, optionalJoin=Optional[Join{term=6, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 7 while handling publication
	at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T03:53:18,235][INFO ][o.e.c.c.CoordinationState] [onap-sdnrdb-master-1] cluster UUID set to [POwqk8uWTCO0E8avziKD8Q]
[2023-04-01T03:53:18,620][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}, term: 7, version: 1, reason: ApplyCommitRequest{term=7, version=1, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T03:53:18,772][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-1] publish_address {10.233.68.45:9200}, bound_addresses {0.0.0.0:9200}
[2023-04-01T03:53:18,862][INFO ][o.e.n.Node               ] [onap-sdnrdb-master-1] started
[2023-04-01T03:53:20,037][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 2, reason: ApplyCommitRequest{term=7, version=2, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T03:53:29,566][INFO ][o.e.c.s.ClusterSettings  ] [onap-sdnrdb-master-1] updating [action.auto_create_index] from [true] to [false]
[2023-04-01T03:54:01,169][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 16, reason: ApplyCommitRequest{term=7, version=16, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:53:07,874][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T04:53:07,963][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 95, reason: becoming candidate: onLeaderFailure
[2023-04-01T04:53:15,694][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 99, reason: ApplyCommitRequest{term=7, version=99, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:53:18,563][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 101, reason: ApplyCommitRequest{term=7, version=101, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:53:21,885][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 105, reason: ApplyCommitRequest{term=7, version=105, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:55:54,140][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 169, reason: ApplyCommitRequest{term=7, version=169, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:55:54,375][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T04:55:54,466][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T04:55:54,766][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T04:55:54,972][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,163][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,280][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,397][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,470][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,605][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,764][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,770][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T04:55:55,887][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T04:55:56,073][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T04:55:56,367][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T04:55:56,571][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T04:55:56,768][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T04:55:57,082][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T04:55:57,104][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T04:55:57,267][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T04:55:57,364][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T04:55:57,420][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T04:55:57,496][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T04:55:59,420][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 170, reason: ApplyCommitRequest{term=7, version=170, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:55:59,876][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 171, reason: ApplyCommitRequest{term=7, version=171, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:56:01,175][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 172, reason: ApplyCommitRequest{term=7, version=172, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T04:56:14,180][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][3791] overhead, spent [330ms] collecting in the last [1.2s]
[2023-04-01T05:53:08,112][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T05:53:08,119][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 240, reason: becoming candidate: onLeaderFailure
[2023-04-01T05:53:08,452][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 241, reason: ApplyCommitRequest{term=7, version=241, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:53:16,512][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T05:53:16,563][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 241, reason: becoming candidate: onLeaderFailure
[2023-04-01T05:53:19,456][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 245, reason: ApplyCommitRequest{term=7, version=245, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:53:20,973][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 246, reason: ApplyCommitRequest{term=7, version=246, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:53:26,314][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 250, reason: ApplyCommitRequest{term=7, version=250, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:56:02,321][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 313, reason: ApplyCommitRequest{term=7, version=313, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:56:02,469][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T05:56:02,585][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T05:56:02,691][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T05:56:02,764][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T05:56:02,864][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T05:56:02,970][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T05:56:03,068][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T05:56:03,172][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T05:56:03,470][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T05:56:03,577][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T05:56:03,965][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T05:56:03,977][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,070][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,283][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,482][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,562][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,590][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,680][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,880][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T05:56:04,987][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T05:56:05,070][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T05:56:05,173][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T05:56:05,269][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 314, reason: ApplyCommitRequest{term=7, version=314, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:56:05,800][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 315, reason: ApplyCommitRequest{term=7, version=315, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:56:05,974][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 316, reason: ApplyCommitRequest{term=7, version=316, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:56:06,299][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 317, reason: ApplyCommitRequest{term=7, version=317, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T05:56:06,935][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 318, reason: ApplyCommitRequest{term=7, version=318, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T06:01:20,097][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:36212}], closing connection
[2023-04-01T06:53:08,171][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T06:53:08,174][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 387, reason: becoming candidate: onLeaderFailure
[2023-04-01T06:53:08,578][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 388, reason: ApplyCommitRequest{term=7, version=388, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T06:53:20,698][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T06:53:20,715][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 388, reason: becoming candidate: onLeaderFailure
[2023-04-01T06:53:23,859][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 392, reason: ApplyCommitRequest{term=7, version=392, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T06:53:25,469][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 393, reason: ApplyCommitRequest{term=7, version=393, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T06:53:28,898][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 396, reason: ApplyCommitRequest{term=7, version=396, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T06:56:07,894][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 460, reason: ApplyCommitRequest{term=7, version=460, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T06:56:08,008][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,063][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,092][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,202][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,268][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,299][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,402][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,502][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,597][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,678][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,872][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,901][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T06:56:08,996][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,063][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,105][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,213][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,289][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,378][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,396][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,472][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,587][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T06:56:09,612][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T06:56:10,429][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 461, reason: ApplyCommitRequest{term=7, version=461, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T07:06:12,694][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:34930}], closing connection
[2023-04-01T07:53:08,201][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T07:53:08,204][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 530, reason: becoming candidate: onLeaderFailure
[2023-04-01T07:53:08,568][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 531, reason: ApplyCommitRequest{term=7, version=531, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T07:53:25,648][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T07:53:25,662][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 531, reason: becoming candidate: onLeaderFailure
[2023-04-01T07:53:26,800][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 533, reason: ApplyCommitRequest{term=7, version=533, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T07:53:27,990][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 534, reason: ApplyCommitRequest{term=7, version=534, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T07:53:32,183][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 537, reason: ApplyCommitRequest{term=7, version=537, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T07:56:11,526][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 601, reason: ApplyCommitRequest{term=7, version=601, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T07:56:11,668][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T07:56:11,765][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T07:56:11,867][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T07:56:11,901][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T07:56:11,999][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,080][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,206][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,267][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,305][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,385][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,568][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,676][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,763][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,795][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,881][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,921][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T07:56:12,972][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T07:56:13,074][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T07:56:13,178][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T07:56:13,266][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T07:56:13,371][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T07:56:13,571][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T07:56:13,767][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 602, reason: ApplyCommitRequest{term=7, version=602, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:06:09,311][WARN ][o.e.t.TcpTransport       ] [onap-sdnrdb-master-1] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:50854}], closing connection
[2023-04-01T08:53:08,225][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T08:53:08,227][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 670, reason: becoming candidate: onLeaderFailure
[2023-04-01T08:53:08,477][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 671, reason: ApplyCommitRequest{term=7, version=671, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:53:28,665][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T08:53:28,672][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 671, reason: becoming candidate: onLeaderFailure
[2023-04-01T08:53:29,503][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 673, reason: ApplyCommitRequest{term=7, version=673, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:53:30,805][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 674, reason: ApplyCommitRequest{term=7, version=674, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:53:33,450][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 678, reason: ApplyCommitRequest{term=7, version=678, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:56:14,514][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 742, reason: ApplyCommitRequest{term=7, version=742, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:56:14,662][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T08:56:14,687][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T08:56:14,772][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T08:56:14,794][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T08:56:14,875][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T08:56:14,970][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,081][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,163][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,165][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,223][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,364][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,383][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,466][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,499][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,571][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,616][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,682][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,727][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,762][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,866][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T08:56:15,963][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T08:56:16,082][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T08:56:16,180][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 743, reason: ApplyCommitRequest{term=7, version=743, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:56:16,533][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 744, reason: ApplyCommitRequest{term=7, version=744, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T08:56:16,893][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 745, reason: ApplyCommitRequest{term=7, version=745, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:53:08,260][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T09:53:08,262][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 813, reason: becoming candidate: onLeaderFailure
[2023-04-01T09:53:08,479][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 814, reason: ApplyCommitRequest{term=7, version=814, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:53:31,573][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T09:53:31,581][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 814, reason: becoming candidate: onLeaderFailure
[2023-04-01T09:53:32,540][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 816, reason: ApplyCommitRequest{term=7, version=816, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:53:33,785][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 817, reason: ApplyCommitRequest{term=7, version=817, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:53:36,901][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 821, reason: ApplyCommitRequest{term=7, version=821, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:56:17,786][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 885, reason: ApplyCommitRequest{term=7, version=885, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:56:17,897][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T09:56:17,926][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T09:56:17,976][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,022][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,101][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,176][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,273][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,382][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,464][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,495][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,602][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,692][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,724][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,785][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T09:56:18,870][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,163][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,207][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,268][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,464][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,665][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,763][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,788][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T09:56:19,797][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 886, reason: ApplyCommitRequest{term=7, version=886, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:56:20,176][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 887, reason: ApplyCommitRequest{term=7, version=887, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T09:56:20,863][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 888, reason: ApplyCommitRequest{term=7, version=888, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T10:53:08,273][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T10:53:08,275][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 957, reason: becoming candidate: onLeaderFailure
[2023-04-01T10:53:08,566][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 958, reason: ApplyCommitRequest{term=7, version=958, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T10:53:34,677][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T10:53:34,682][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 958, reason: becoming candidate: onLeaderFailure
[2023-04-01T10:53:36,685][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 962, reason: ApplyCommitRequest{term=7, version=962, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T10:53:37,687][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 963, reason: ApplyCommitRequest{term=7, version=963, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T10:53:40,433][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 968, reason: ApplyCommitRequest{term=7, version=968, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T10:56:21,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1031, reason: ApplyCommitRequest{term=7, version=1031, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T10:56:21,476][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T10:56:21,519][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T10:56:21,697][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T10:56:21,762][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T10:56:21,799][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T10:56:21,869][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T10:56:21,906][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,066][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,087][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,137][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,263][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,291][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,344][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,383][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,424][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,478][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,511][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,567][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,605][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,664][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,685][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T10:56:22,714][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T10:56:23,088][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1032, reason: ApplyCommitRequest{term=7, version=1032, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:53:08,293][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T11:53:08,296][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1101, reason: becoming candidate: onLeaderFailure
[2023-04-01T11:53:08,655][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1102, reason: ApplyCommitRequest{term=7, version=1102, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:53:37,817][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T11:53:37,824][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1102, reason: becoming candidate: onLeaderFailure
[2023-04-01T11:53:40,524][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1107, reason: ApplyCommitRequest{term=7, version=1107, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:53:43,067][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1110, reason: ApplyCommitRequest{term=7, version=1110, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:56:23,964][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1176, reason: ApplyCommitRequest{term=7, version=1176, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:56:24,100][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,178][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,264][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,299][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,464][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,495][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,606][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,693][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,864][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T11:56:24,866][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,069][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,163][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,197][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,225][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,301][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,365][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,573][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,672][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T11:56:25,964][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T11:56:26,002][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T11:56:26,074][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T11:56:26,173][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T11:56:26,263][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1177, reason: ApplyCommitRequest{term=7, version=1177, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:56:26,549][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1178, reason: ApplyCommitRequest{term=7, version=1178, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:56:26,663][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1179, reason: ApplyCommitRequest{term=7, version=1179, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:56:26,858][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1180, reason: ApplyCommitRequest{term=7, version=1180, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T11:56:27,630][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1181, reason: ApplyCommitRequest{term=7, version=1181, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T12:53:08,315][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T12:53:08,317][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1248, reason: becoming candidate: onLeaderFailure
[2023-04-01T12:53:08,769][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1249, reason: ApplyCommitRequest{term=7, version=1249, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T12:53:41,921][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T12:53:41,934][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1249, reason: becoming candidate: onLeaderFailure
[2023-04-01T12:53:44,598][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1254, reason: ApplyCommitRequest{term=7, version=1254, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T12:53:46,463][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1256, reason: ApplyCommitRequest{term=7, version=1256, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T12:56:28,603][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1323, reason: ApplyCommitRequest{term=7, version=1323, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T12:56:28,732][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T12:56:28,784][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T12:56:28,862][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T12:56:28,926][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T12:56:28,999][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,073][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,112][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,205][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,234][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,292][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,380][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,415][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,489][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,563][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,606][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,672][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,791][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,872][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,900][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T12:56:29,986][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T12:56:30,086][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T12:56:30,129][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T12:56:30,242][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1324, reason: ApplyCommitRequest{term=7, version=1324, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T13:53:08,384][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T13:53:08,389][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1393, reason: becoming candidate: onLeaderFailure
[2023-04-01T13:53:08,699][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1394, reason: ApplyCommitRequest{term=7, version=1394, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T13:53:46,869][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T13:53:46,879][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1394, reason: becoming candidate: onLeaderFailure
[2023-04-01T13:53:49,529][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1399, reason: ApplyCommitRequest{term=7, version=1399, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T13:53:50,661][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1400, reason: ApplyCommitRequest{term=7, version=1400, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T13:56:31,128][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1467, reason: ApplyCommitRequest{term=7, version=1467, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T13:56:31,264][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,364][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,396][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,505][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,584][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,674][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,710][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,796][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,890][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T13:56:31,999][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,164][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,297][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,397][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,490][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,568][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,628][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,683][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,764][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,796][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,867][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,896][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T13:56:32,938][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T13:56:33,236][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1468, reason: ApplyCommitRequest{term=7, version=1468, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T14:53:08,410][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T14:53:08,413][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1536, reason: becoming candidate: onLeaderFailure
[2023-04-01T14:53:08,595][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1537, reason: ApplyCommitRequest{term=7, version=1537, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T14:53:51,793][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T14:53:51,805][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1537, reason: becoming candidate: onLeaderFailure
[2023-04-01T14:53:55,164][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
	at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join/validate] disconnected
[2023-04-01T14:53:55,169][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
	at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.68.45:9300][internal:cluster/coordination/join/validate] disconnected
[2023-04-01T14:53:56,595][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1547, reason: ApplyCommitRequest{term=7, version=1547, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T14:56:34,455][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1615, reason: ApplyCommitRequest{term=7, version=1615, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T14:56:34,590][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T14:56:34,792][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T14:56:34,870][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T14:56:34,903][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T14:56:34,995][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,063][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,109][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,163][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,268][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,397][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,479][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,568][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,595][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,764][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T14:56:35,870][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,164][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,373][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,562][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,627][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,694][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,791][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,805][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T14:56:36,837][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1616, reason: ApplyCommitRequest{term=7, version=1616, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T14:56:37,048][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1617, reason: ApplyCommitRequest{term=7, version=1617, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T14:56:38,017][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 7, version: 1618, reason: ApplyCommitRequest{term=7, version=1618, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T15:53:08,419][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T15:53:08,427][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1686, reason: becoming candidate: onLeaderFailure
[2023-04-01T15:53:08,755][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1687, reason: ApplyCommitRequest{term=7, version=1687, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T15:53:55,065][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1688, reason: ApplyCommitRequest{term=7, version=1688, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T15:53:58,056][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T15:53:58,064][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1688, reason: becoming candidate: onLeaderFailure
[2023-04-01T15:53:58,351][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.NotMasterException: Node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] not master for join request
[2023-04-01T15:53:59,150][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1690, reason: ApplyCommitRequest{term=9, version=1690, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T15:53:59,796][INFO ][o.e.c.c.JoinHelper       ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=8, lastAcceptedTerm=7, lastAcceptedVersion=1688, sourceNode={onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, targetNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 9 while handling publication
	at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.9.3.jar:7.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2023-04-01T15:54:00,159][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1691, reason: ApplyCommitRequest{term=9, version=1691, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T15:56:38,703][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 9, version: 1758, reason: ApplyCommitRequest{term=9, version=1758, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T15:56:38,812][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T15:56:38,894][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T15:56:38,906][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T15:56:38,977][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,062][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,089][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,189][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,310][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,407][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,493][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,590][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,680][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,863][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T15:56:39,907][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,001][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,028][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,102][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,165][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,203][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,281][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,393][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T15:56:40,419][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T15:56:41,045][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 9, version: 1759, reason: ApplyCommitRequest{term=9, version=1759, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T16:53:08,456][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T16:53:08,459][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1828, reason: becoming candidate: onLeaderFailure
[2023-04-01T16:53:08,718][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1829, reason: ApplyCommitRequest{term=9, version=1829, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T16:53:57,585][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1831, reason: ApplyCommitRequest{term=9, version=1831, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T16:54:00,978][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T16:54:00,989][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1831, reason: becoming candidate: onLeaderFailure
[2023-04-01T16:54:01,828][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1834, reason: ApplyCommitRequest{term=9, version=1834, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T16:56:41,372][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 9, version: 1900, reason: ApplyCommitRequest{term=9, version=1900, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T16:56:41,507][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T16:56:41,567][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T16:56:41,609][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T16:56:41,870][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,063][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,071][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,179][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,212][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,299][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,381][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,464][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,508][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,539][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,603][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,663][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,697][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,738][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,806][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,839][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,884][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,918][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T16:56:42,946][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T16:56:43,365][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 9, version: 1901, reason: ApplyCommitRequest{term=9, version=1901, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T17:53:08,491][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected
[2023-04-01T17:53:08,495][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1970, reason: becoming candidate: onLeaderFailure
[2023-04-01T17:53:08,701][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1971, reason: ApplyCommitRequest{term=9, version=1971, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T17:54:00,229][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1973, reason: ApplyCommitRequest{term=9, version=1973, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T17:54:03,947][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2023-04-01T17:54:03,959][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1973, reason: becoming candidate: onLeaderFailure
[2023-04-01T17:54:07,601][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1981, reason: ApplyCommitRequest{term=9, version=1981, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T17:56:44,246][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 9, version: 2047, reason: ApplyCommitRequest{term=9, version=2047, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}
[2023-04-01T17:56:44,377][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,470][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [connectionlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,508][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,664][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,699][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,786][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,880][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T17:56:44,962][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,005][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,079][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,164][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,272][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,380][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,764][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T17:56:45,985][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,095][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,164][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,188][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][1] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,296][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [cmlog-v7][4] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,365][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,404][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations
[2023-04-01T17:56:46,569][INFO ][o.e.i.s.IndexShard       ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations
[2023-04-01T17:56:47,307][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}, term: 9, version: 2048, reason: ApplyCommitRequest{term=9, version=2048, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}