18:47:44.36   18:47:44.37 Welcome to the Bitnami elasticsearch container  18:47:44.38 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch  18:47:44.40 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues  18:47:44.41   18:47:44.41 INFO  ==> ** Starting Elasticsearch setup **  18:47:44.54 INFO  ==> Configuring/Initializing Elasticsearch...  18:47:44.63 INFO  ==> Setting default configuration  18:47:44.66 INFO  ==> Configuring Elasticsearch cluster settings... OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  18:47:48.25 INFO  ==> ** Elasticsearch setup finished! **  18:47:48.28 INFO  ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2022-09-22T18:47:52,225][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2022-09-22T18:47:52,227][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] JVM home [/opt/bitnami/java] [2022-09-22T18:47:52,228][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-846221801874921186, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2022-09-22T18:47:53,520][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [aggs-matrix-stats] [2022-09-22T18:47:53,521][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [analysis-common] [2022-09-22T18:47:53,521][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [geo] [2022-09-22T18:47:53,521][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [ingest-common] [2022-09-22T18:47:53,521][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [ingest-geoip] [2022-09-22T18:47:53,522][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [ingest-user-agent] [2022-09-22T18:47:53,522][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [kibana] [2022-09-22T18:47:53,522][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [lang-expression] [2022-09-22T18:47:53,522][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [lang-mustache] [2022-09-22T18:47:53,522][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [lang-painless] [2022-09-22T18:47:53,522][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [mapper-extras] [2022-09-22T18:47:53,523][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [parent-join] [2022-09-22T18:47:53,523][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [percolator] [2022-09-22T18:47:53,523][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [rank-eval] [2022-09-22T18:47:53,523][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [reindex] [2022-09-22T18:47:53,523][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [repository-url] [2022-09-22T18:47:53,524][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [tasks] [2022-09-22T18:47:53,524][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded module [transport-netty4] [2022-09-22T18:47:53,525][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] loaded plugin [repository-s3] [2022-09-22T18:47:53,590][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/vda1)]], net usable_space [147.3gb], net total_space [154.8gb], types [ext4] [2022-09-22T18:47:53,591][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] heap size [123.7mb], compressed ordinary object pointers [true] [2022-09-22T18:47:53,644][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] node name [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx], node ID [xskgo6YPQNSIUGw6-kTfiQ], cluster name [sdnrdb-cluster] [2022-09-22T18:47:57,428][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2022-09-22T18:47:57,528][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] using discovery type [zen] and seed hosts providers [settings] [2022-09-22T18:47:57,868][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2022-09-22T18:47:58,354][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] initialized [2022-09-22T18:47:58,355][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] starting ... [2022-09-22T18:47:58,893][INFO ][o.e.t.TransportService ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] publish_address {10.233.70.232:9300}, bound_addresses {0.0.0.0:9300} [2022-09-22T18:47:59,092][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] bound or publishing to a non-loopback address, enforcing bootstrap checks [2022-09-22T18:50:05,653][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr},{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T18:50:05,671][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] publish_address {10.233.70.232:9000}, bound_addresses {0.0.0.0:9000} [2022-09-22T18:50:05,672][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] started [2022-09-22T18:50:32,673][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] updating [action.auto_create_index] from [true] to [false] [2022-09-22T18:50:40,742][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T19:49:35,551][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-22T19:49:35,575][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 84, reason: becoming candidate: onLeaderFailure [2022-09-22T19:49:35,774][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 85, reason: ApplyCommitRequest{term=1, version=85, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T19:50:02,440][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 86, reason: ApplyCommitRequest{term=1, version=86, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T19:50:08,052][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-22T19:50:08,069][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 86, reason: becoming candidate: onLeaderFailure [2022-09-22T19:50:09,172][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 92, reason: ApplyCommitRequest{term=1, version=92, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T19:50:40,558][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 161, reason: ApplyCommitRequest{term=1, version=161, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T19:50:43,353][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 164, reason: ApplyCommitRequest{term=1, version=164, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:49:35,601][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-22T20:49:35,621][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 232, reason: becoming candidate: onLeaderFailure [2022-09-22T20:49:35,765][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 233, reason: ApplyCommitRequest{term=1, version=233, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:50:07,716][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 235, reason: ApplyCommitRequest{term=1, version=235, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:50:11,775][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-22T20:50:11,786][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 235, reason: becoming candidate: onLeaderFailure [2022-09-22T20:50:12,256][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 239, reason: ApplyCommitRequest{term=1, version=239, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:50:44,789][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 298, reason: ApplyCommitRequest{term=1, version=298, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:50:46,756][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 300, reason: ApplyCommitRequest{term=1, version=300, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:50:47,245][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 301, reason: ApplyCommitRequest{term=1, version=301, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T20:50:48,255][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 304, reason: ApplyCommitRequest{term=1, version=304, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T21:49:35,631][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-22T21:49:35,641][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 372, reason: becoming candidate: onLeaderFailure [2022-09-22T21:49:35,790][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 373, reason: ApplyCommitRequest{term=1, version=373, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T21:50:10,210][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 375, reason: ApplyCommitRequest{term=1, version=375, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T21:50:14,864][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-22T21:50:14,873][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 375, reason: becoming candidate: onLeaderFailure [2022-09-22T21:50:15,901][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 381, reason: ApplyCommitRequest{term=1, version=381, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T21:50:48,171][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 448, reason: ApplyCommitRequest{term=1, version=448, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T21:50:50,095][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 449, reason: ApplyCommitRequest{term=1, version=449, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T22:49:35,663][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-22T22:49:35,678][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 515, reason: becoming candidate: onLeaderFailure [2022-09-22T22:49:35,746][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 516, reason: ApplyCommitRequest{term=1, version=516, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T22:50:13,343][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 518, reason: ApplyCommitRequest{term=1, version=518, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T22:50:17,841][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-22T22:50:17,856][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 518, reason: becoming candidate: onLeaderFailure [2022-09-22T22:50:19,039][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 525, reason: ApplyCommitRequest{term=1, version=525, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T22:50:51,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 591, reason: ApplyCommitRequest{term=1, version=591, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T22:50:53,385][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 592, reason: ApplyCommitRequest{term=1, version=592, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:49:35,683][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-22T23:49:35,692][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 661, reason: becoming candidate: onLeaderFailure [2022-09-22T23:49:35,778][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 662, reason: ApplyCommitRequest{term=1, version=662, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:50:16,485][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 664, reason: ApplyCommitRequest{term=1, version=664, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:50:20,874][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-22T23:50:20,883][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 664, reason: becoming candidate: onLeaderFailure [2022-09-22T23:50:21,442][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 670, reason: ApplyCommitRequest{term=1, version=670, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:50:53,794][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 739, reason: ApplyCommitRequest{term=1, version=739, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:50:56,060][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 740, reason: ApplyCommitRequest{term=1, version=740, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:50:56,458][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 741, reason: ApplyCommitRequest{term=1, version=741, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-22T23:50:56,663][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 742, reason: ApplyCommitRequest{term=1, version=742, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T00:49:35,718][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T00:49:35,754][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 808, reason: becoming candidate: onLeaderFailure [2022-09-23T00:49:35,853][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 809, reason: ApplyCommitRequest{term=1, version=809, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T00:50:20,673][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 811, reason: ApplyCommitRequest{term=1, version=811, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T00:50:24,953][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T00:50:24,964][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 811, reason: becoming candidate: onLeaderFailure [2022-09-23T00:50:25,091][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 816, reason: ApplyCommitRequest{term=1, version=816, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T00:50:57,277][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 883, reason: ApplyCommitRequest{term=1, version=883, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T00:50:58,955][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 884, reason: ApplyCommitRequest{term=1, version=884, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T01:49:35,774][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T01:49:35,786][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 950, reason: becoming candidate: onLeaderFailure [2022-09-23T01:49:35,864][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 951, reason: ApplyCommitRequest{term=1, version=951, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T01:50:23,230][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 953, reason: ApplyCommitRequest{term=1, version=953, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T01:50:27,935][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T01:50:27,943][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 953, reason: becoming candidate: onLeaderFailure [2022-09-23T01:50:28,871][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 959, reason: ApplyCommitRequest{term=1, version=959, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T01:50:59,385][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1025, reason: ApplyCommitRequest{term=1, version=1025, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T01:51:01,901][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1026, reason: ApplyCommitRequest{term=1, version=1026, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T02:49:35,811][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T02:49:35,821][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1094, reason: becoming candidate: onLeaderFailure [2022-09-23T02:49:35,896][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 1095, reason: ApplyCommitRequest{term=1, version=1095, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T02:50:26,344][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1097, reason: ApplyCommitRequest{term=1, version=1097, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T02:50:29,995][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T02:50:30,011][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1097, reason: becoming candidate: onLeaderFailure [2022-09-23T02:50:30,214][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1100, reason: ApplyCommitRequest{term=1, version=1100, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T02:51:02,123][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1171, reason: ApplyCommitRequest{term=1, version=1171, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T02:51:04,756][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1172, reason: ApplyCommitRequest{term=1, version=1172, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T03:49:35,833][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T03:49:35,840][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1240, reason: becoming candidate: onLeaderFailure [2022-09-23T03:49:35,991][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 1241, reason: ApplyCommitRequest{term=1, version=1241, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T03:50:29,393][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1243, reason: ApplyCommitRequest{term=1, version=1243, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T03:50:33,135][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T03:50:33,143][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1243, reason: becoming candidate: onLeaderFailure [2022-09-23T03:50:33,827][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1248, reason: ApplyCommitRequest{term=1, version=1248, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T03:51:05,211][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1316, reason: ApplyCommitRequest{term=1, version=1316, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T03:51:06,950][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1317, reason: ApplyCommitRequest{term=1, version=1317, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T04:49:35,851][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T04:49:35,859][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1383, reason: becoming candidate: onLeaderFailure [2022-09-23T04:49:35,952][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 1384, reason: ApplyCommitRequest{term=1, version=1384, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T04:50:32,121][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1386, reason: ApplyCommitRequest{term=1, version=1386, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T04:50:36,086][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T04:50:36,102][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1386, reason: becoming candidate: onLeaderFailure [2022-09-23T04:50:36,398][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1391, reason: ApplyCommitRequest{term=1, version=1391, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T04:51:07,493][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1458, reason: ApplyCommitRequest{term=1, version=1458, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T04:51:09,098][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1459, reason: ApplyCommitRequest{term=1, version=1459, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T05:49:35,875][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T05:49:35,892][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1527, reason: becoming candidate: onLeaderFailure [2022-09-23T05:49:36,039][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 1528, reason: ApplyCommitRequest{term=1, version=1528, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T05:50:35,247][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1530, reason: ApplyCommitRequest{term=1, version=1530, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T05:50:39,121][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T05:50:39,134][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1530, reason: becoming candidate: onLeaderFailure [2022-09-23T05:50:39,291][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1535, reason: ApplyCommitRequest{term=1, version=1535, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T05:51:09,584][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1604, reason: ApplyCommitRequest{term=1, version=1604, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T05:51:12,067][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1605, reason: ApplyCommitRequest{term=1, version=1605, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T06:49:35,930][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.70.159:9300][disconnected] disconnected [2022-09-23T06:49:35,944][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1672, reason: becoming candidate: onLeaderFailure [2022-09-23T06:49:36,251][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, term: 1, version: 1673, reason: ApplyCommitRequest{term=1, version=1673, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T06:50:37,837][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1675, reason: ApplyCommitRequest{term=1, version=1675, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T06:50:41,352][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.70.159:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx}{xskgo6YPQNSIUGw6-kTfiQ}{7ckh1NRCQ9aHOMBWNPkmOQ}{10.233.70.232}{10.233.70.232:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2022-09-23T06:50:41,364][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}], current []}, term: 1, version: 1675, reason: becoming candidate: onLeaderFailure [2022-09-23T06:50:41,573][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] master node changed {previous [], current [{onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{B6HouXcOS2iloacEKZJWfg}{JoXR_g1KSri4bumeg2qGxA}{10.233.65.193}{10.233.65.193:9300}{dmr}}, term: 1, version: 1678, reason: ApplyCommitRequest{term=1, version=1678, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T06:51:12,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] removed {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1747, reason: ApplyCommitRequest{term=1, version=1747, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}} [2022-09-23T06:51:15,055][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-pbjvx] added {{onap-sdnrdb-master-1}{tX3NrHU1QsC5azulbtbKng}{Ugq52laFSwuF3yYTwocw6g}{10.233.72.1}{10.233.72.1:9300}{dmr}}, term: 1, version: 1748, reason: ApplyCommitRequest{term=1, version=1748, sourceNode={onap-sdnrdb-master-0}{u9CHUCTcRACBq5B5-2LB0g}{dmMhGJxvQ_OKujPi78jgsQ}{10.233.70.159}{10.233.70.159:9300}{dmr}}