11:46:53.76   11:46:53.76 Welcome to the Bitnami elasticsearch container  11:46:53.77 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch  11:46:53.77 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues  11:46:53.78   11:46:53.78 INFO  ==> ** Starting Elasticsearch setup **  11:46:53.87 INFO  ==> Configuring/Initializing Elasticsearch...  11:46:53.96 INFO  ==> Setting default configuration  11:46:53.99 INFO  ==> Configuring Elasticsearch cluster settings... OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  11:46:56.44 INFO  ==> ** Elasticsearch setup finished! **  11:46:56.47 INFO  ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2023-03-21T11:47:00,374][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2023-03-21T11:47:00,376][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] JVM home [/opt/bitnami/java] [2023-03-21T11:47:00,377][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-7011307390337174510, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2023-03-21T11:47:01,705][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [aggs-matrix-stats] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [analysis-common] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [geo] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [ingest-common] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [ingest-geoip] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [ingest-user-agent] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [kibana] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [lang-expression] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [lang-mustache] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [lang-painless] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [mapper-extras] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [parent-join] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [percolator] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [rank-eval] [2023-03-21T11:47:01,709][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [reindex] [2023-03-21T11:47:01,709][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [repository-url] [2023-03-21T11:47:01,709][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [tasks] [2023-03-21T11:47:01,710][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [transport-netty4] [2023-03-21T11:47:01,710][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded plugin [repository-s3] [2023-03-21T11:47:01,758][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/sda1)]], net usable_space [152.1gb], net total_space [154.8gb], types [ext4] [2023-03-21T11:47:01,759][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] heap size [123.7mb], compressed ordinary object pointers [true] [2023-03-21T11:47:01,807][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] node name [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr], node ID [vZNn6r6kRHyfyDE4L_1YiA], cluster name [sdnrdb-cluster] [2023-03-21T11:47:05,258][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2023-03-21T11:47:05,331][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] using discovery type [zen] and seed hosts providers [settings] [2023-03-21T11:47:05,645][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-03-21T11:47:05,830][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] initialized [2023-03-21T11:47:05,831][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] starting ... [2023-03-21T11:47:06,068][INFO ][o.e.t.TransportService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] publish_address {10.233.72.244:9300}, bound_addresses {0.0.0.0:9300} [2023-03-21T11:47:06,386][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-03-21T11:50:21,062][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr},{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T11:50:21,089][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] publish_address {10.233.72.244:9000}, bound_addresses {0.0.0.0:9000} [2023-03-21T11:50:21,090][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] started [2023-03-21T11:50:27,211][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] updating [action.auto_create_index] from [true] to [false] [2023-03-21T11:50:39,150][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:49:44,028][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T12:49:44,042][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 88, reason: becoming candidate: onLeaderFailure [2023-03-21T12:49:44,589][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 89, reason: ApplyCommitRequest{term=1, version=89, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:14,870][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 90, reason: ApplyCommitRequest{term=1, version=90, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:21,843][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T12:50:21,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 90, reason: becoming candidate: onLeaderFailure [2023-03-21T12:50:22,857][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 96, reason: ApplyCommitRequest{term=1, version=96, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:23,490][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 97, reason: ApplyCommitRequest{term=1, version=97, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:39,755][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 110, reason: ApplyCommitRequest{term=1, version=110, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:43,158][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 111, reason: ApplyCommitRequest{term=1, version=111, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:44,020][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 112, reason: ApplyCommitRequest{term=1, version=112, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:46,159][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 114, reason: ApplyCommitRequest{term=1, version=114, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:46,949][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 115, reason: ApplyCommitRequest{term=1, version=115, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:48,243][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 117, reason: ApplyCommitRequest{term=1, version=117, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:49,446][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 118, reason: ApplyCommitRequest{term=1, version=118, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:51,150][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 119, reason: ApplyCommitRequest{term=1, version=119, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:51,736][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 120, reason: ApplyCommitRequest{term=1, version=120, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:55,036][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 123, reason: ApplyCommitRequest{term=1, version=123, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:00:19,255][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:39464}], closing connection [2023-03-21T13:24:11,863][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:36332}], closing connection [2023-03-21T13:49:08,978][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:36236}], closing connection [2023-03-21T13:49:44,116][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T13:49:44,120][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 191, reason: becoming candidate: onLeaderFailure [2023-03-21T13:49:44,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 192, reason: ApplyCommitRequest{term=1, version=192, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:24,633][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T13:50:24,648][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 193, reason: becoming candidate: onLeaderFailure [2023-03-21T13:50:26,412][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 196, reason: ApplyCommitRequest{term=1, version=196, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:26,933][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 197, reason: ApplyCommitRequest{term=1, version=197, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:27,933][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 198, reason: ApplyCommitRequest{term=1, version=198, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:53,290][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 232, reason: ApplyCommitRequest{term=1, version=232, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:56,039][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 234, reason: ApplyCommitRequest{term=1, version=234, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:49:44,207][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T14:49:44,217][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 301, reason: becoming candidate: onLeaderFailure [2023-03-21T14:49:44,821][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 302, reason: ApplyCommitRequest{term=1, version=302, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:27,571][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T14:50:27,583][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 303, reason: becoming candidate: onLeaderFailure [2023-03-21T14:50:27,899][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 305, reason: ApplyCommitRequest{term=1, version=305, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:28,956][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 306, reason: ApplyCommitRequest{term=1, version=306, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:31,284][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 307, reason: ApplyCommitRequest{term=1, version=307, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:56,859][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 345, reason: ApplyCommitRequest{term=1, version=345, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:59,616][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 347, reason: ApplyCommitRequest{term=1, version=347, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:49:44,235][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T15:49:44,238][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 415, reason: becoming candidate: onLeaderFailure [2023-03-21T15:49:44,464][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 416, reason: ApplyCommitRequest{term=1, version=416, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:30,501][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T15:50:30,513][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 417, reason: becoming candidate: onLeaderFailure [2023-03-21T15:50:30,718][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 419, reason: ApplyCommitRequest{term=1, version=419, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:31,866][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 420, reason: ApplyCommitRequest{term=1, version=420, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:33,988][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 421, reason: ApplyCommitRequest{term=1, version=421, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:59,612][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 461, reason: ApplyCommitRequest{term=1, version=461, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:51:01,567][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 464, reason: ApplyCommitRequest{term=1, version=464, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:49:44,270][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T16:49:44,275][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 529, reason: becoming candidate: onLeaderFailure [2023-03-21T16:49:44,420][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 530, reason: ApplyCommitRequest{term=1, version=530, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:50:33,527][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T16:50:33,536][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 531, reason: becoming candidate: onLeaderFailure [2023-03-21T16:50:33,706][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 533, reason: ApplyCommitRequest{term=1, version=533, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:50:34,873][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 534, reason: ApplyCommitRequest{term=1, version=534, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:50:37,186][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 536, reason: ApplyCommitRequest{term=1, version=536, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:03,171][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 575, reason: ApplyCommitRequest{term=1, version=575, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:05,416][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 577, reason: ApplyCommitRequest{term=1, version=577, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:06,072][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 578, reason: ApplyCommitRequest{term=1, version=578, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:08,025][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 580, reason: ApplyCommitRequest{term=1, version=580, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:08,290][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 581, reason: ApplyCommitRequest{term=1, version=581, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:09,992][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 583, reason: ApplyCommitRequest{term=1, version=583, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:49:44,290][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T17:49:44,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 650, reason: becoming candidate: onLeaderFailure [2023-03-21T17:49:44,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 651, reason: ApplyCommitRequest{term=1, version=651, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:36,575][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T17:50:36,584][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 652, reason: becoming candidate: onLeaderFailure [2023-03-21T17:50:36,803][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 655, reason: ApplyCommitRequest{term=1, version=655, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:37,644][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 656, reason: ApplyCommitRequest{term=1, version=656, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:39,756][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 657, reason: ApplyCommitRequest{term=1, version=657, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:39,954][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 658, reason: ApplyCommitRequest{term=1, version=658, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:40,038][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 659, reason: ApplyCommitRequest{term=1, version=659, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:40,353][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 660, reason: ApplyCommitRequest{term=1, version=660, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:41,051][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 661, reason: ApplyCommitRequest{term=1, version=661, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:10,024][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 700, reason: ApplyCommitRequest{term=1, version=700, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:12,178][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 701, reason: ApplyCommitRequest{term=1, version=701, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:13,190][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 702, reason: ApplyCommitRequest{term=1, version=702, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:14,607][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 704, reason: ApplyCommitRequest{term=1, version=704, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:14,922][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 705, reason: ApplyCommitRequest{term=1, version=705, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:16,465][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 708, reason: ApplyCommitRequest{term=1, version=708, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:49:44,320][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T18:49:44,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 777, reason: becoming candidate: onLeaderFailure [2023-03-21T18:49:44,453][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 778, reason: ApplyCommitRequest{term=1, version=778, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:50:38,653][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T18:50:38,664][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 780, reason: becoming candidate: onLeaderFailure [2023-03-21T18:50:38,836][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 782, reason: ApplyCommitRequest{term=1, version=782, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:50:42,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 783, reason: ApplyCommitRequest{term=1, version=783, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:50:43,950][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 784, reason: ApplyCommitRequest{term=1, version=784, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:17,559][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 842, reason: ApplyCommitRequest{term=1, version=842, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:19,689][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 843, reason: ApplyCommitRequest{term=1, version=843, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:20,378][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 844, reason: ApplyCommitRequest{term=1, version=844, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:20,883][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 845, reason: ApplyCommitRequest{term=1, version=845, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:21,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 846, reason: ApplyCommitRequest{term=1, version=846, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:22,066][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 848, reason: ApplyCommitRequest{term=1, version=848, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:49:44,331][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T19:49:44,334][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 914, reason: becoming candidate: onLeaderFailure [2023-03-21T19:49:44,509][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 915, reason: ApplyCommitRequest{term=1, version=915, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:50:41,608][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T19:50:41,624][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 917, reason: becoming candidate: onLeaderFailure [2023-03-21T19:50:41,832][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 919, reason: ApplyCommitRequest{term=1, version=919, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:50:44,555][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 920, reason: ApplyCommitRequest{term=1, version=920, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:50:46,056][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 921, reason: ApplyCommitRequest{term=1, version=921, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:51:23,131][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 981, reason: ApplyCommitRequest{term=1, version=981, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:51:25,746][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 984, reason: ApplyCommitRequest{term=1, version=984, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:49:44,350][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T20:49:44,352][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1051, reason: becoming candidate: onLeaderFailure [2023-03-21T20:49:44,536][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1052, reason: ApplyCommitRequest{term=1, version=1052, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:50:44,661][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T20:50:44,672][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1054, reason: becoming candidate: onLeaderFailure [2023-03-21T20:50:44,864][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1056, reason: ApplyCommitRequest{term=1, version=1056, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:50:46,735][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1057, reason: ApplyCommitRequest{term=1, version=1057, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:50:49,202][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1058, reason: ApplyCommitRequest{term=1, version=1058, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:27,335][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1120, reason: ApplyCommitRequest{term=1, version=1120, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:29,807][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1122, reason: ApplyCommitRequest{term=1, version=1122, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:30,329][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1123, reason: ApplyCommitRequest{term=1, version=1123, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:30,833][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1124, reason: ApplyCommitRequest{term=1, version=1124, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:31,211][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1125, reason: ApplyCommitRequest{term=1, version=1125, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:32,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1127, reason: ApplyCommitRequest{term=1, version=1127, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:49:44,366][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T21:49:44,370][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1192, reason: becoming candidate: onLeaderFailure [2023-03-21T21:49:44,523][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1193, reason: ApplyCommitRequest{term=1, version=1193, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:47,675][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T21:50:47,684][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1195, reason: becoming candidate: onLeaderFailure [2023-03-21T21:50:47,855][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1197, reason: ApplyCommitRequest{term=1, version=1197, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:50,408][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1198, reason: ApplyCommitRequest{term=1, version=1198, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:52,037][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1199, reason: ApplyCommitRequest{term=1, version=1199, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:52,395][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1200, reason: ApplyCommitRequest{term=1, version=1200, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:52,846][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1201, reason: ApplyCommitRequest{term=1, version=1201, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:51:33,060][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1268, reason: ApplyCommitRequest{term=1, version=1268, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:51:35,190][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1269, reason: ApplyCommitRequest{term=1, version=1269, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:49:44,382][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T22:49:44,384][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1335, reason: becoming candidate: onLeaderFailure [2023-03-21T22:49:44,523][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1336, reason: ApplyCommitRequest{term=1, version=1336, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:50:50,717][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T22:50:50,726][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1338, reason: becoming candidate: onLeaderFailure [2023-03-21T22:50:50,870][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1340, reason: ApplyCommitRequest{term=1, version=1340, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:50:54,115][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1341, reason: ApplyCommitRequest{term=1, version=1341, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:50:55,976][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1342, reason: ApplyCommitRequest{term=1, version=1342, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:51:36,609][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1408, reason: ApplyCommitRequest{term=1, version=1408, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:51:38,371][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1409, reason: ApplyCommitRequest{term=1, version=1409, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:51:38,517][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1410, reason: ApplyCommitRequest{term=1, version=1410, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T22:51:39,129][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1411, reason: ApplyCommitRequest{term=1, version=1411, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:49:44,402][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T23:49:44,405][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1477, reason: becoming candidate: onLeaderFailure [2023-03-21T23:49:44,662][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1478, reason: ApplyCommitRequest{term=1, version=1478, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:50:52,862][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T23:50:52,878][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1480, reason: becoming candidate: onLeaderFailure [2023-03-21T23:50:53,118][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1482, reason: ApplyCommitRequest{term=1, version=1482, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:50:56,640][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1483, reason: ApplyCommitRequest{term=1, version=1483, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:50:59,173][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1484, reason: ApplyCommitRequest{term=1, version=1484, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:51:40,025][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1552, reason: ApplyCommitRequest{term=1, version=1552, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:51:42,323][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1553, reason: ApplyCommitRequest{term=1, version=1553, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:51:42,604][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1554, reason: ApplyCommitRequest{term=1, version=1554, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T23:51:43,621][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1555, reason: ApplyCommitRequest{term=1, version=1555, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:49:44,458][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T00:49:44,483][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1621, reason: becoming candidate: onLeaderFailure [2023-03-22T00:49:44,608][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1622, reason: ApplyCommitRequest{term=1, version=1622, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:50:55,855][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T00:50:55,868][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1624, reason: becoming candidate: onLeaderFailure [2023-03-22T00:50:56,032][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1626, reason: ApplyCommitRequest{term=1, version=1626, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:00,209][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1627, reason: ApplyCommitRequest{term=1, version=1627, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:02,092][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1628, reason: ApplyCommitRequest{term=1, version=1628, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:02,299][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1629, reason: ApplyCommitRequest{term=1, version=1629, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:02,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1630, reason: ApplyCommitRequest{term=1, version=1630, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:44,466][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1696, reason: ApplyCommitRequest{term=1, version=1696, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:46,206][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1697, reason: ApplyCommitRequest{term=1, version=1697, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:46,407][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1698, reason: ApplyCommitRequest{term=1, version=1698, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T00:51:46,765][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1699, reason: ApplyCommitRequest{term=1, version=1699, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:49:44,501][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T01:49:44,507][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1764, reason: becoming candidate: onLeaderFailure [2023-03-22T01:49:44,644][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1765, reason: ApplyCommitRequest{term=1, version=1765, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:50:58,893][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T01:50:58,905][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1767, reason: becoming candidate: onLeaderFailure [2023-03-22T01:50:59,108][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1769, reason: ApplyCommitRequest{term=1, version=1769, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:03,205][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1770, reason: ApplyCommitRequest{term=1, version=1770, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:05,178][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1771, reason: ApplyCommitRequest{term=1, version=1771, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:47,405][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1838, reason: ApplyCommitRequest{term=1, version=1838, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:50,853][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1839, reason: ApplyCommitRequest{term=1, version=1839, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:51,553][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1840, reason: ApplyCommitRequest{term=1, version=1840, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:51,869][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1841, reason: ApplyCommitRequest{term=1, version=1841, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:52,346][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1842, reason: ApplyCommitRequest{term=1, version=1842, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:52,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1843, reason: ApplyCommitRequest{term=1, version=1843, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:53,057][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1844, reason: ApplyCommitRequest{term=1, version=1844, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T01:51:53,660][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1845, reason: ApplyCommitRequest{term=1, version=1845, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:49:44,521][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T02:49:44,524][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1911, reason: becoming candidate: onLeaderFailure [2023-03-22T02:49:44,841][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1912, reason: ApplyCommitRequest{term=1, version=1912, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:01,903][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T02:51:01,914][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1914, reason: becoming candidate: onLeaderFailure [2023-03-22T02:51:02,068][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1916, reason: ApplyCommitRequest{term=1, version=1916, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:06,148][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1917, reason: ApplyCommitRequest{term=1, version=1917, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:08,654][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1918, reason: ApplyCommitRequest{term=1, version=1918, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:09,127][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1919, reason: ApplyCommitRequest{term=1, version=1919, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:10,429][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1920, reason: ApplyCommitRequest{term=1, version=1920, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:54,059][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1988, reason: ApplyCommitRequest{term=1, version=1988, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T02:51:55,702][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1989, reason: ApplyCommitRequest{term=1, version=1989, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:49:44,539][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T03:49:44,541][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2055, reason: becoming candidate: onLeaderFailure [2023-03-22T03:49:44,670][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2056, reason: ApplyCommitRequest{term=1, version=2056, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:04,929][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T03:51:04,941][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2058, reason: becoming candidate: onLeaderFailure [2023-03-22T03:51:05,101][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2060, reason: ApplyCommitRequest{term=1, version=2060, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:11,824][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2061, reason: ApplyCommitRequest{term=1, version=2061, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:13,931][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2062, reason: ApplyCommitRequest{term=1, version=2062, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:14,113][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2063, reason: ApplyCommitRequest{term=1, version=2063, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:15,128][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2064, reason: ApplyCommitRequest{term=1, version=2064, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:58,324][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2130, reason: ApplyCommitRequest{term=1, version=2130, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:51:59,836][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2131, reason: ApplyCommitRequest{term=1, version=2131, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:52:00,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2132, reason: ApplyCommitRequest{term=1, version=2132, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T03:52:00,929][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2134, reason: ApplyCommitRequest{term=1, version=2134, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T04:49:44,550][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T04:49:44,553][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2200, reason: becoming candidate: onLeaderFailure [2023-03-22T04:49:44,763][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2201, reason: ApplyCommitRequest{term=1, version=2201, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T04:51:07,963][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T04:51:07,974][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2203, reason: becoming candidate: onLeaderFailure [2023-03-22T04:51:08,184][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2205, reason: ApplyCommitRequest{term=1, version=2205, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T04:51:16,093][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2206, reason: ApplyCommitRequest{term=1, version=2206, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T04:51:17,736][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2207, reason: ApplyCommitRequest{term=1, version=2207, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T04:52:01,919][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2273, reason: ApplyCommitRequest{term=1, version=2273, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T04:52:03,689][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2274, reason: ApplyCommitRequest{term=1, version=2274, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T05:49:44,566][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T05:49:44,570][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2339, reason: becoming candidate: onLeaderFailure [2023-03-22T05:49:44,718][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2340, reason: ApplyCommitRequest{term=1, version=2340, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T05:51:10,990][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T05:51:11,003][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2342, reason: becoming candidate: onLeaderFailure [2023-03-22T05:51:11,158][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2344, reason: ApplyCommitRequest{term=1, version=2344, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T05:51:18,713][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2345, reason: ApplyCommitRequest{term=1, version=2345, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T05:51:20,493][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2346, reason: ApplyCommitRequest{term=1, version=2346, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T05:52:04,185][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2413, reason: ApplyCommitRequest{term=1, version=2413, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T05:52:06,649][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2414, reason: ApplyCommitRequest{term=1, version=2414, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:49:44,584][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T06:49:44,587][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2480, reason: becoming candidate: onLeaderFailure [2023-03-22T06:49:44,724][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2481, reason: ApplyCommitRequest{term=1, version=2481, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:51:14,007][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T06:51:14,017][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2483, reason: becoming candidate: onLeaderFailure [2023-03-22T06:51:14,191][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2485, reason: ApplyCommitRequest{term=1, version=2485, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:51:21,153][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2486, reason: ApplyCommitRequest{term=1, version=2486, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:51:23,363][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2487, reason: ApplyCommitRequest{term=1, version=2487, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:52:07,810][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2553, reason: ApplyCommitRequest{term=1, version=2553, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:52:09,373][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2554, reason: ApplyCommitRequest{term=1, version=2554, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:52:09,710][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2555, reason: ApplyCommitRequest{term=1, version=2555, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:52:09,793][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2556, reason: ApplyCommitRequest{term=1, version=2556, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:52:09,926][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2557, reason: ApplyCommitRequest{term=1, version=2557, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T06:52:10,821][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2558, reason: ApplyCommitRequest{term=1, version=2558, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:49:44,603][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T07:49:44,607][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2624, reason: becoming candidate: onLeaderFailure [2023-03-22T07:49:44,741][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2625, reason: ApplyCommitRequest{term=1, version=2625, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:51:17,054][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T07:51:17,068][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2627, reason: becoming candidate: onLeaderFailure [2023-03-22T07:51:17,219][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2629, reason: ApplyCommitRequest{term=1, version=2629, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:51:24,358][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2630, reason: ApplyCommitRequest{term=1, version=2630, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:51:28,351][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2631, reason: ApplyCommitRequest{term=1, version=2631, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:51:28,792][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2632, reason: ApplyCommitRequest{term=1, version=2632, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:51:29,335][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2633, reason: ApplyCommitRequest{term=1, version=2633, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:52:11,356][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2700, reason: ApplyCommitRequest{term=1, version=2700, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T07:52:12,958][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2701, reason: ApplyCommitRequest{term=1, version=2701, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T08:49:44,631][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T08:49:44,634][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2767, reason: becoming candidate: onLeaderFailure [2023-03-22T08:49:44,776][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2768, reason: ApplyCommitRequest{term=1, version=2768, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T08:51:20,102][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T08:51:20,112][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2770, reason: becoming candidate: onLeaderFailure [2023-03-22T08:51:20,273][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2772, reason: ApplyCommitRequest{term=1, version=2772, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T08:51:30,242][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2773, reason: ApplyCommitRequest{term=1, version=2773, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T08:51:32,345][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2774, reason: ApplyCommitRequest{term=1, version=2774, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T08:52:13,670][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2840, reason: ApplyCommitRequest{term=1, version=2840, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T08:52:15,754][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2841, reason: ApplyCommitRequest{term=1, version=2841, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T09:49:44,659][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T09:49:44,672][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2906, reason: becoming candidate: onLeaderFailure [2023-03-22T09:49:44,820][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2907, reason: ApplyCommitRequest{term=1, version=2907, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T09:51:22,156][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T09:51:22,168][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 2909, reason: becoming candidate: onLeaderFailure [2023-03-22T09:51:22,325][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 2911, reason: ApplyCommitRequest{term=1, version=2911, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T09:51:32,874][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2912, reason: ApplyCommitRequest{term=1, version=2912, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T09:51:35,341][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2913, reason: ApplyCommitRequest{term=1, version=2913, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T09:52:16,833][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2980, reason: ApplyCommitRequest{term=1, version=2980, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T09:52:18,874][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 2981, reason: ApplyCommitRequest{term=1, version=2981, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T10:49:44,688][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T10:49:44,693][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3047, reason: becoming candidate: onLeaderFailure [2023-03-22T10:49:44,816][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3048, reason: ApplyCommitRequest{term=1, version=3048, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T10:51:25,171][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T10:51:25,182][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3050, reason: becoming candidate: onLeaderFailure [2023-03-22T10:51:25,342][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3052, reason: ApplyCommitRequest{term=1, version=3052, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T10:51:36,433][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3053, reason: ApplyCommitRequest{term=1, version=3053, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T10:51:38,350][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3054, reason: ApplyCommitRequest{term=1, version=3054, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T10:52:19,625][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3121, reason: ApplyCommitRequest{term=1, version=3121, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T10:52:21,778][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3122, reason: ApplyCommitRequest{term=1, version=3122, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T11:49:44,716][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T11:49:44,721][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3188, reason: becoming candidate: onLeaderFailure [2023-03-22T11:49:44,887][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3189, reason: ApplyCommitRequest{term=1, version=3189, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T11:51:27,231][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T11:51:27,239][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3191, reason: becoming candidate: onLeaderFailure [2023-03-22T11:51:27,411][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3193, reason: ApplyCommitRequest{term=1, version=3193, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T11:51:39,077][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3194, reason: ApplyCommitRequest{term=1, version=3194, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T11:51:41,317][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3195, reason: ApplyCommitRequest{term=1, version=3195, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T11:52:22,317][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3262, reason: ApplyCommitRequest{term=1, version=3262, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T11:52:24,059][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3263, reason: ApplyCommitRequest{term=1, version=3263, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T12:49:44,743][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T12:49:44,746][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3329, reason: becoming candidate: onLeaderFailure [2023-03-22T12:49:44,879][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3330, reason: ApplyCommitRequest{term=1, version=3330, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T12:51:30,243][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T12:51:30,253][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3332, reason: becoming candidate: onLeaderFailure [2023-03-22T12:51:30,417][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3334, reason: ApplyCommitRequest{term=1, version=3334, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T12:51:41,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3335, reason: ApplyCommitRequest{term=1, version=3335, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T12:51:44,473][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3336, reason: ApplyCommitRequest{term=1, version=3336, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T12:52:25,170][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3402, reason: ApplyCommitRequest{term=1, version=3402, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T12:52:27,221][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3403, reason: ApplyCommitRequest{term=1, version=3403, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T13:49:44,763][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T13:49:44,766][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3469, reason: becoming candidate: onLeaderFailure [2023-03-22T13:49:44,922][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3470, reason: ApplyCommitRequest{term=1, version=3470, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T13:51:33,270][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T13:51:33,292][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3472, reason: becoming candidate: onLeaderFailure [2023-03-22T13:51:33,425][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3474, reason: ApplyCommitRequest{term=1, version=3474, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T13:51:45,019][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3475, reason: ApplyCommitRequest{term=1, version=3475, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T13:51:46,751][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3476, reason: ApplyCommitRequest{term=1, version=3476, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T13:52:27,778][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3544, reason: ApplyCommitRequest{term=1, version=3544, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T13:52:29,856][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3545, reason: ApplyCommitRequest{term=1, version=3545, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T14:49:44,781][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T14:49:44,786][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3611, reason: becoming candidate: onLeaderFailure [2023-03-22T14:49:44,937][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3612, reason: ApplyCommitRequest{term=1, version=3612, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T14:51:35,360][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T14:51:35,370][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3614, reason: becoming candidate: onLeaderFailure [2023-03-22T14:51:35,530][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3616, reason: ApplyCommitRequest{term=1, version=3616, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T14:51:47,515][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3617, reason: ApplyCommitRequest{term=1, version=3617, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T14:51:49,624][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3618, reason: ApplyCommitRequest{term=1, version=3618, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T14:52:30,352][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3685, reason: ApplyCommitRequest{term=1, version=3685, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T14:52:32,152][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3686, reason: ApplyCommitRequest{term=1, version=3686, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T15:49:44,799][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T15:49:44,809][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3752, reason: becoming candidate: onLeaderFailure [2023-03-22T15:49:44,955][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3753, reason: ApplyCommitRequest{term=1, version=3753, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T15:51:38,367][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T15:51:38,377][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3755, reason: becoming candidate: onLeaderFailure [2023-03-22T15:51:38,534][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3757, reason: ApplyCommitRequest{term=1, version=3757, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T15:51:50,132][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3758, reason: ApplyCommitRequest{term=1, version=3758, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T15:51:52,532][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3759, reason: ApplyCommitRequest{term=1, version=3759, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T15:52:32,610][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3827, reason: ApplyCommitRequest{term=1, version=3827, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T15:52:35,251][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3828, reason: ApplyCommitRequest{term=1, version=3828, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T16:49:44,824][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T16:49:44,827][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3894, reason: becoming candidate: onLeaderFailure [2023-03-22T16:49:44,961][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3895, reason: ApplyCommitRequest{term=1, version=3895, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T16:51:41,381][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T16:51:41,389][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 3897, reason: becoming candidate: onLeaderFailure [2023-03-22T16:51:41,600][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 3899, reason: ApplyCommitRequest{term=1, version=3899, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T16:51:53,074][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3900, reason: ApplyCommitRequest{term=1, version=3900, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T16:51:55,542][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 3901, reason: ApplyCommitRequest{term=1, version=3901, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T16:52:35,656][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3968, reason: ApplyCommitRequest{term=1, version=3968, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T16:52:37,976][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 3969, reason: ApplyCommitRequest{term=1, version=3969, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:49:44,839][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T17:49:44,845][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4034, reason: becoming candidate: onLeaderFailure [2023-03-22T17:49:45,046][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4035, reason: ApplyCommitRequest{term=1, version=4035, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:51:44,419][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T17:51:44,428][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4037, reason: becoming candidate: onLeaderFailure [2023-03-22T17:51:44,634][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4039, reason: ApplyCommitRequest{term=1, version=4039, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:51:56,186][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4040, reason: ApplyCommitRequest{term=1, version=4040, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:51:58,435][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4041, reason: ApplyCommitRequest{term=1, version=4041, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:52:38,671][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4108, reason: ApplyCommitRequest{term=1, version=4108, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:52:40,350][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4109, reason: ApplyCommitRequest{term=1, version=4109, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:52:40,501][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4110, reason: ApplyCommitRequest{term=1, version=4110, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T17:52:41,067][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4111, reason: ApplyCommitRequest{term=1, version=4111, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:49:44,861][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T18:49:44,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4177, reason: becoming candidate: onLeaderFailure [2023-03-22T18:49:45,017][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4178, reason: ApplyCommitRequest{term=1, version=4178, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:51:47,444][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T18:51:47,454][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4180, reason: becoming candidate: onLeaderFailure [2023-03-22T18:51:47,649][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4182, reason: ApplyCommitRequest{term=1, version=4182, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:51:58,996][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4183, reason: ApplyCommitRequest{term=1, version=4183, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:52:00,670][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4184, reason: ApplyCommitRequest{term=1, version=4184, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:52:42,788][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4250, reason: ApplyCommitRequest{term=1, version=4250, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:52:44,293][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4251, reason: ApplyCommitRequest{term=1, version=4251, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:52:44,527][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4252, reason: ApplyCommitRequest{term=1, version=4252, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T18:52:45,026][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4253, reason: ApplyCommitRequest{term=1, version=4253, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T19:49:44,893][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T19:49:44,897][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4319, reason: becoming candidate: onLeaderFailure [2023-03-22T19:49:45,038][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4320, reason: ApplyCommitRequest{term=1, version=4320, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T19:51:50,472][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T19:51:50,481][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4322, reason: becoming candidate: onLeaderFailure [2023-03-22T19:51:50,645][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4324, reason: ApplyCommitRequest{term=1, version=4324, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T19:52:01,275][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4325, reason: ApplyCommitRequest{term=1, version=4325, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T19:52:03,460][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4326, reason: ApplyCommitRequest{term=1, version=4326, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T19:52:45,775][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4394, reason: ApplyCommitRequest{term=1, version=4394, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T19:52:47,213][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4395, reason: ApplyCommitRequest{term=1, version=4395, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T20:49:44,907][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T20:49:44,912][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4461, reason: becoming candidate: onLeaderFailure [2023-03-22T20:49:45,064][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4462, reason: ApplyCommitRequest{term=1, version=4462, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T20:51:53,505][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T20:51:53,516][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4464, reason: becoming candidate: onLeaderFailure [2023-03-22T20:51:53,637][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4466, reason: ApplyCommitRequest{term=1, version=4466, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T20:52:04,107][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4467, reason: ApplyCommitRequest{term=1, version=4467, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T20:52:05,670][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4468, reason: ApplyCommitRequest{term=1, version=4468, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T20:52:48,329][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4535, reason: ApplyCommitRequest{term=1, version=4535, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T20:52:50,275][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4536, reason: ApplyCommitRequest{term=1, version=4536, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T21:49:44,933][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T21:49:44,936][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4602, reason: becoming candidate: onLeaderFailure [2023-03-22T21:49:45,073][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4603, reason: ApplyCommitRequest{term=1, version=4603, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T21:51:56,553][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T21:51:56,560][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4605, reason: becoming candidate: onLeaderFailure [2023-03-22T21:51:56,721][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4607, reason: ApplyCommitRequest{term=1, version=4607, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T21:52:06,282][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4608, reason: ApplyCommitRequest{term=1, version=4608, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T21:52:08,499][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4609, reason: ApplyCommitRequest{term=1, version=4609, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T21:52:51,463][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4676, reason: ApplyCommitRequest{term=1, version=4676, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T21:52:53,109][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4677, reason: ApplyCommitRequest{term=1, version=4677, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T22:49:44,942][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T22:49:44,945][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4743, reason: becoming candidate: onLeaderFailure [2023-03-22T22:49:45,086][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4744, reason: ApplyCommitRequest{term=1, version=4744, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T22:51:59,595][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T22:51:59,604][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4746, reason: becoming candidate: onLeaderFailure [2023-03-22T22:51:59,775][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4748, reason: ApplyCommitRequest{term=1, version=4748, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T22:52:09,141][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4749, reason: ApplyCommitRequest{term=1, version=4749, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T22:52:11,493][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4750, reason: ApplyCommitRequest{term=1, version=4750, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T22:52:53,729][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4817, reason: ApplyCommitRequest{term=1, version=4817, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T22:52:55,369][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4818, reason: ApplyCommitRequest{term=1, version=4818, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T23:49:44,962][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-22T23:49:44,967][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4884, reason: becoming candidate: onLeaderFailure [2023-03-22T23:49:45,114][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4885, reason: ApplyCommitRequest{term=1, version=4885, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T23:52:02,640][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-22T23:52:02,646][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 4887, reason: becoming candidate: onLeaderFailure [2023-03-22T23:52:02,793][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 4889, reason: ApplyCommitRequest{term=1, version=4889, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T23:52:12,125][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4890, reason: ApplyCommitRequest{term=1, version=4890, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T23:52:13,779][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 4891, reason: ApplyCommitRequest{term=1, version=4891, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T23:52:55,936][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4958, reason: ApplyCommitRequest{term=1, version=4958, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-22T23:52:58,375][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 4959, reason: ApplyCommitRequest{term=1, version=4959, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:49:44,987][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-23T00:49:45,009][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 5024, reason: becoming candidate: onLeaderFailure [2023-03-23T00:49:45,163][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 5025, reason: ApplyCommitRequest{term=1, version=5025, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:52:05,656][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-23T00:52:05,666][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 5027, reason: becoming candidate: onLeaderFailure [2023-03-23T00:52:05,814][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 5029, reason: ApplyCommitRequest{term=1, version=5029, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:52:14,356][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 5030, reason: ApplyCommitRequest{term=1, version=5030, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:52:15,926][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 5031, reason: ApplyCommitRequest{term=1, version=5031, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:53:00,006][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 5097, reason: ApplyCommitRequest{term=1, version=5097, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:53:01,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 5098, reason: ApplyCommitRequest{term=1, version=5098, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:53:01,726][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 5099, reason: ApplyCommitRequest{term=1, version=5099, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T00:53:02,362][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 5100, reason: ApplyCommitRequest{term=1, version=5100, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T01:49:45,024][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-23T01:49:45,029][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 5167, reason: becoming candidate: onLeaderFailure [2023-03-23T01:49:45,195][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 5168, reason: ApplyCommitRequest{term=1, version=5168, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T01:52:07,706][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-23T01:52:07,716][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 5170, reason: becoming candidate: onLeaderFailure [2023-03-23T01:52:07,910][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 5172, reason: ApplyCommitRequest{term=1, version=5172, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T01:52:16,574][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 5173, reason: ApplyCommitRequest{term=1, version=5173, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T01:52:18,695][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 5174, reason: ApplyCommitRequest{term=1, version=5174, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T01:53:02,942][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 5241, reason: ApplyCommitRequest{term=1, version=5241, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T01:53:05,386][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 5242, reason: ApplyCommitRequest{term=1, version=5242, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T02:49:45,040][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-23T02:49:45,045][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 5307, reason: becoming candidate: onLeaderFailure [2023-03-23T02:49:45,178][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 5308, reason: ApplyCommitRequest{term=1, version=5308, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-23T02:52:10,733][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[