11:46:53.76   11:46:53.76 Welcome to the Bitnami elasticsearch container  11:46:53.77 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch  11:46:53.77 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues  11:46:53.78   11:46:53.78 INFO  ==> ** Starting Elasticsearch setup **  11:46:53.87 INFO  ==> Configuring/Initializing Elasticsearch...  11:46:53.96 INFO  ==> Setting default configuration  11:46:53.99 INFO  ==> Configuring Elasticsearch cluster settings... OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  11:46:56.44 INFO  ==> ** Elasticsearch setup finished! **  11:46:56.47 INFO  ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2023-03-21T11:47:00,374][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2023-03-21T11:47:00,376][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] JVM home [/opt/bitnami/java] [2023-03-21T11:47:00,377][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-7011307390337174510, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2023-03-21T11:47:01,705][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [aggs-matrix-stats] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [analysis-common] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [geo] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [ingest-common] [2023-03-21T11:47:01,706][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [ingest-geoip] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [ingest-user-agent] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [kibana] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [lang-expression] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [lang-mustache] [2023-03-21T11:47:01,707][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [lang-painless] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [mapper-extras] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [parent-join] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [percolator] [2023-03-21T11:47:01,708][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [rank-eval] [2023-03-21T11:47:01,709][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [reindex] [2023-03-21T11:47:01,709][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [repository-url] [2023-03-21T11:47:01,709][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [tasks] [2023-03-21T11:47:01,710][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded module [transport-netty4] [2023-03-21T11:47:01,710][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] loaded plugin [repository-s3] [2023-03-21T11:47:01,758][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/sda1)]], net usable_space [152.1gb], net total_space [154.8gb], types [ext4] [2023-03-21T11:47:01,759][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] heap size [123.7mb], compressed ordinary object pointers [true] [2023-03-21T11:47:01,807][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] node name [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr], node ID [vZNn6r6kRHyfyDE4L_1YiA], cluster name [sdnrdb-cluster] [2023-03-21T11:47:05,258][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2023-03-21T11:47:05,331][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] using discovery type [zen] and seed hosts providers [settings] [2023-03-21T11:47:05,645][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-03-21T11:47:05,830][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] initialized [2023-03-21T11:47:05,831][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] starting ... [2023-03-21T11:47:06,068][INFO ][o.e.t.TransportService ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] publish_address {10.233.72.244:9300}, bound_addresses {0.0.0.0:9300} [2023-03-21T11:47:06,386][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-03-21T11:50:21,062][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr},{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T11:50:21,089][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] publish_address {10.233.72.244:9000}, bound_addresses {0.0.0.0:9000} [2023-03-21T11:50:21,090][INFO ][o.e.n.Node ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] started [2023-03-21T11:50:27,211][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] updating [action.auto_create_index] from [true] to [false] [2023-03-21T11:50:39,150][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 6, reason: ApplyCommitRequest{term=1, version=6, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:49:44,028][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T12:49:44,042][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 88, reason: becoming candidate: onLeaderFailure [2023-03-21T12:49:44,589][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 89, reason: ApplyCommitRequest{term=1, version=89, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:14,870][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 90, reason: ApplyCommitRequest{term=1, version=90, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:21,843][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T12:50:21,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 90, reason: becoming candidate: onLeaderFailure [2023-03-21T12:50:22,857][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 96, reason: ApplyCommitRequest{term=1, version=96, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:23,490][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 97, reason: ApplyCommitRequest{term=1, version=97, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:39,755][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 110, reason: ApplyCommitRequest{term=1, version=110, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:43,158][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 111, reason: ApplyCommitRequest{term=1, version=111, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:44,020][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 112, reason: ApplyCommitRequest{term=1, version=112, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:46,159][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 114, reason: ApplyCommitRequest{term=1, version=114, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:46,949][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 115, reason: ApplyCommitRequest{term=1, version=115, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:48,243][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 117, reason: ApplyCommitRequest{term=1, version=117, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:49,446][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 118, reason: ApplyCommitRequest{term=1, version=118, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:51,150][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 119, reason: ApplyCommitRequest{term=1, version=119, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:51,736][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 120, reason: ApplyCommitRequest{term=1, version=120, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T12:50:55,036][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 123, reason: ApplyCommitRequest{term=1, version=123, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:00:19,255][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:39464}], closing connection [2023-03-21T13:24:11,863][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:36332}], closing connection [2023-03-21T13:49:08,978][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:36236}], closing connection [2023-03-21T13:49:44,116][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T13:49:44,120][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 191, reason: becoming candidate: onLeaderFailure [2023-03-21T13:49:44,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 192, reason: ApplyCommitRequest{term=1, version=192, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:24,633][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T13:50:24,648][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 193, reason: becoming candidate: onLeaderFailure [2023-03-21T13:50:26,412][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 196, reason: ApplyCommitRequest{term=1, version=196, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:26,933][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 197, reason: ApplyCommitRequest{term=1, version=197, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:27,933][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 198, reason: ApplyCommitRequest{term=1, version=198, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:53,290][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 232, reason: ApplyCommitRequest{term=1, version=232, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T13:50:56,039][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 234, reason: ApplyCommitRequest{term=1, version=234, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:49:44,207][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T14:49:44,217][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 301, reason: becoming candidate: onLeaderFailure [2023-03-21T14:49:44,821][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 302, reason: ApplyCommitRequest{term=1, version=302, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:27,571][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T14:50:27,583][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 303, reason: becoming candidate: onLeaderFailure [2023-03-21T14:50:27,899][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 305, reason: ApplyCommitRequest{term=1, version=305, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:28,956][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 306, reason: ApplyCommitRequest{term=1, version=306, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:31,284][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 307, reason: ApplyCommitRequest{term=1, version=307, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:56,859][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 345, reason: ApplyCommitRequest{term=1, version=345, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T14:50:59,616][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 347, reason: ApplyCommitRequest{term=1, version=347, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:49:44,235][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T15:49:44,238][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 415, reason: becoming candidate: onLeaderFailure [2023-03-21T15:49:44,464][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 416, reason: ApplyCommitRequest{term=1, version=416, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:30,501][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T15:50:30,513][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 417, reason: becoming candidate: onLeaderFailure [2023-03-21T15:50:30,718][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 419, reason: ApplyCommitRequest{term=1, version=419, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:31,866][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 420, reason: ApplyCommitRequest{term=1, version=420, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:33,988][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 421, reason: ApplyCommitRequest{term=1, version=421, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:50:59,612][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 461, reason: ApplyCommitRequest{term=1, version=461, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T15:51:01,567][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 464, reason: ApplyCommitRequest{term=1, version=464, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:49:44,270][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T16:49:44,275][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 529, reason: becoming candidate: onLeaderFailure [2023-03-21T16:49:44,420][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 530, reason: ApplyCommitRequest{term=1, version=530, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:50:33,527][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T16:50:33,536][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 531, reason: becoming candidate: onLeaderFailure [2023-03-21T16:50:33,706][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 533, reason: ApplyCommitRequest{term=1, version=533, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:50:34,873][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 534, reason: ApplyCommitRequest{term=1, version=534, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:50:37,186][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 536, reason: ApplyCommitRequest{term=1, version=536, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:03,171][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 575, reason: ApplyCommitRequest{term=1, version=575, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:05,416][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 577, reason: ApplyCommitRequest{term=1, version=577, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:06,072][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 578, reason: ApplyCommitRequest{term=1, version=578, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:08,025][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 580, reason: ApplyCommitRequest{term=1, version=580, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:08,290][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 581, reason: ApplyCommitRequest{term=1, version=581, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T16:51:09,992][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 583, reason: ApplyCommitRequest{term=1, version=583, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:49:44,290][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T17:49:44,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 650, reason: becoming candidate: onLeaderFailure [2023-03-21T17:49:44,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 651, reason: ApplyCommitRequest{term=1, version=651, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:36,575][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T17:50:36,584][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 652, reason: becoming candidate: onLeaderFailure [2023-03-21T17:50:36,803][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 655, reason: ApplyCommitRequest{term=1, version=655, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:37,644][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 656, reason: ApplyCommitRequest{term=1, version=656, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:39,756][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 657, reason: ApplyCommitRequest{term=1, version=657, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:39,954][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 658, reason: ApplyCommitRequest{term=1, version=658, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:40,038][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 659, reason: ApplyCommitRequest{term=1, version=659, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:40,353][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 660, reason: ApplyCommitRequest{term=1, version=660, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:50:41,051][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 661, reason: ApplyCommitRequest{term=1, version=661, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:10,024][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 700, reason: ApplyCommitRequest{term=1, version=700, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:12,178][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 701, reason: ApplyCommitRequest{term=1, version=701, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:13,190][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 702, reason: ApplyCommitRequest{term=1, version=702, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:14,607][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 704, reason: ApplyCommitRequest{term=1, version=704, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:14,922][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 705, reason: ApplyCommitRequest{term=1, version=705, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T17:51:16,465][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 708, reason: ApplyCommitRequest{term=1, version=708, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:49:44,320][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T18:49:44,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 777, reason: becoming candidate: onLeaderFailure [2023-03-21T18:49:44,453][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 778, reason: ApplyCommitRequest{term=1, version=778, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:50:38,653][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T18:50:38,664][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 780, reason: becoming candidate: onLeaderFailure [2023-03-21T18:50:38,836][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 782, reason: ApplyCommitRequest{term=1, version=782, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:50:42,326][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 783, reason: ApplyCommitRequest{term=1, version=783, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:50:43,950][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 784, reason: ApplyCommitRequest{term=1, version=784, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:17,559][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 842, reason: ApplyCommitRequest{term=1, version=842, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:19,689][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 843, reason: ApplyCommitRequest{term=1, version=843, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:20,378][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 844, reason: ApplyCommitRequest{term=1, version=844, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:20,883][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 845, reason: ApplyCommitRequest{term=1, version=845, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:21,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 846, reason: ApplyCommitRequest{term=1, version=846, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T18:51:22,066][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 848, reason: ApplyCommitRequest{term=1, version=848, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:49:44,331][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T19:49:44,334][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 914, reason: becoming candidate: onLeaderFailure [2023-03-21T19:49:44,509][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 915, reason: ApplyCommitRequest{term=1, version=915, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:50:41,608][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T19:50:41,624][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 917, reason: becoming candidate: onLeaderFailure [2023-03-21T19:50:41,832][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 919, reason: ApplyCommitRequest{term=1, version=919, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:50:44,555][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 920, reason: ApplyCommitRequest{term=1, version=920, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:50:46,056][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 921, reason: ApplyCommitRequest{term=1, version=921, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:51:23,131][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 981, reason: ApplyCommitRequest{term=1, version=981, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T19:51:25,746][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 984, reason: ApplyCommitRequest{term=1, version=984, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:49:44,350][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T20:49:44,352][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1051, reason: becoming candidate: onLeaderFailure [2023-03-21T20:49:44,536][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1052, reason: ApplyCommitRequest{term=1, version=1052, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:50:44,661][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T20:50:44,672][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1054, reason: becoming candidate: onLeaderFailure [2023-03-21T20:50:44,864][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1056, reason: ApplyCommitRequest{term=1, version=1056, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:50:46,735][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1057, reason: ApplyCommitRequest{term=1, version=1057, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:50:49,202][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1058, reason: ApplyCommitRequest{term=1, version=1058, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:27,335][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1120, reason: ApplyCommitRequest{term=1, version=1120, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:29,807][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1122, reason: ApplyCommitRequest{term=1, version=1122, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:30,329][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1123, reason: ApplyCommitRequest{term=1, version=1123, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:30,833][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1124, reason: ApplyCommitRequest{term=1, version=1124, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:31,211][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1125, reason: ApplyCommitRequest{term=1, version=1125, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T20:51:32,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1127, reason: ApplyCommitRequest{term=1, version=1127, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:49:44,366][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.69.216:9300][disconnected] disconnected [2023-03-21T21:49:44,370][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1192, reason: becoming candidate: onLeaderFailure [2023-03-21T21:49:44,523][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1193, reason: ApplyCommitRequest{term=1, version=1193, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:47,675][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.69.216:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-54c85fc557-g9ncr}{vZNn6r6kRHyfyDE4L_1YiA}{UPUaZWnxTD6Zfn7HDPY98w}{10.233.72.244}{10.233.72.244:9300}{r}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-03-21T21:50:47,684][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}], current []}, term: 1, version: 1195, reason: becoming candidate: onLeaderFailure [2023-03-21T21:50:47,855][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] master node changed {previous [], current [{onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}]}, term: 1, version: 1197, reason: ApplyCommitRequest{term=1, version=1197, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:50,408][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1198, reason: ApplyCommitRequest{term=1, version=1198, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:52,037][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1199, reason: ApplyCommitRequest{term=1, version=1199, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:52,395][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1200, reason: ApplyCommitRequest{term=1, version=1200, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:50:52,846][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-1}{U_yhvesnS2i-RAKDRtpMmw}{Bh7bpHVoQIC7MbIxwYjHqQ}{10.233.71.242}{10.233.71.242:9300}{dmr}}, term: 1, version: 1201, reason: ApplyCommitRequest{term=1, version=1201, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:51:33,060][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] removed {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1268, reason: ApplyCommitRequest{term=1, version=1268, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}} [2023-03-21T21:51:35,190][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-54c85fc557-g9ncr] added {{onap-sdnrdb-master-2}{rMotoWCKQouj0wC84rH0Rg}{t46xUV7jTIeDWF-Pgp65Xw}{10.233.70.251}{10.233.70.251:9300}{dmr}}, term: 1, version: 1269, reason: ApplyCommitRequest{term=1, version=1269, sourceNode={onap-sdnrdb-master-0}{483FQrbQQCWG4xEhg5RI9g}{NHD-zOyhSx-VWY6EUTf9Pg}{10.233.69.216}{10.233.69.216:9300}{dmr}}