03:50:41.77   03:50:41.78 Welcome to the Bitnami elasticsearch container  03:50:41.79 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch  03:50:41.87 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues  03:50:41.89   03:50:41.98 INFO  ==> ** Starting Elasticsearch setup **  03:50:42.47 INFO  ==> Configuring/Initializing Elasticsearch...  03:50:42.79 INFO  ==> Setting default configuration  03:50:42.97 INFO  ==> Configuring Elasticsearch cluster settings... OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  03:51:08.67 INFO  ==> ** Elasticsearch setup finished! **  03:51:08.88 INFO  ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2023-04-01T03:52:05,171][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2023-04-01T03:52:05,175][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] JVM home [/opt/bitnami/java] [2023-04-01T03:52:05,176][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-18104087176374184171, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2023-04-01T03:52:26,372][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [aggs-matrix-stats] [2023-04-01T03:52:26,373][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [analysis-common] [2023-04-01T03:52:26,374][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [geo] [2023-04-01T03:52:26,375][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [ingest-common] [2023-04-01T03:52:26,376][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [ingest-geoip] [2023-04-01T03:52:26,376][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [ingest-user-agent] [2023-04-01T03:52:26,377][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [kibana] [2023-04-01T03:52:26,474][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [lang-expression] [2023-04-01T03:52:26,475][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [lang-mustache] [2023-04-01T03:52:26,475][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [lang-painless] [2023-04-01T03:52:26,475][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [mapper-extras] [2023-04-01T03:52:26,476][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [parent-join] [2023-04-01T03:52:26,476][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [percolator] [2023-04-01T03:52:26,478][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [rank-eval] [2023-04-01T03:52:26,478][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [reindex] [2023-04-01T03:52:26,478][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [repository-url] [2023-04-01T03:52:26,479][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [tasks] [2023-04-01T03:52:26,479][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded module [transport-netty4] [2023-04-01T03:52:26,480][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-2] loaded plugin [repository-s3] [2023-04-01T03:52:27,391][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-2] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.5.249:/dockerdata-nfs/onap/elastic-master-1)]], net usable_space [95.2gb], net total_space [99.9gb], types [nfs4] [2023-04-01T03:52:27,474][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-2] heap size [123.7mb], compressed ordinary object pointers [true] [2023-04-01T03:52:28,193][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] node name [onap-sdnrdb-master-2], node ID [y71Y2EKcTU6nD12jJL8k8A], cluster name [sdnrdb-cluster] [2023-04-01T03:53:29,981][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-master-2] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2023-04-01T03:53:31,679][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-master-2] using discovery type [zen] and seed hosts providers [settings] [2023-04-01T03:53:38,078][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-04-01T03:53:41,474][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] initialized [2023-04-01T03:53:41,475][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] starting ... [2023-04-01T03:53:44,271][INFO ][o.e.t.TransportService ] [onap-sdnrdb-master-2] publish_address {10.233.65.70:9300}, bound_addresses {0.0.0.0:9300} [2023-04-01T03:53:49,581][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-master-2] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-04-01T03:53:54,373][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] setting initial configuration to VotingConfiguration{oBVWcRcCQa2pTEZnFAZuXQ,{bootstrap-placeholder}-onap-sdnrdb-master-0,y71Y2EKcTU6nD12jJL8k8A} [2023-04-01T03:53:59,979][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-2] master not discovered or elected yet, an election requires 2 nodes with ids [oBVWcRcCQa2pTEZnFAZuXQ, y71Y2EKcTU6nD12jJL8k8A], have discovered [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}, {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] which is a quorum; discovery will continue using [10.233.66.44:9300, 10.233.68.45:9300, 10.233.67.181:9300] from hosts providers and [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] from last-known cluster state; node term 7, last-accepted version 0 in term 0 [2023-04-01T03:54:02,773][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr},{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr},{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 16, reason: ApplyCommitRequest{term=7, version=16, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T03:54:03,173][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-master-2] updating [action.auto_create_index] from [true] to [false] [2023-04-01T03:54:03,671][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-2] publish_address {10.233.65.70:9200}, bound_addresses {0.0.0.0:9200} [2023-04-01T03:54:03,673][INFO ][o.e.n.Node ] [onap-sdnrdb-master-2] started [2023-04-01T03:54:11,471][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][28] overhead, spent [293ms] collecting in the last [1s] [2023-04-01T04:53:09,699][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 96, reason: ApplyCommitRequest{term=7, version=96, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:53:10,270][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T04:53:10,277][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T04:53:10,482][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T04:53:10,591][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T04:53:10,877][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T04:53:10,974][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T04:53:11,179][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T04:53:11,471][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T04:53:11,777][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T04:53:11,975][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T04:53:12,172][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T04:53:12,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T04:53:13,431][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 97, reason: ApplyCommitRequest{term=7, version=97, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:53:14,328][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 98, reason: ApplyCommitRequest{term=7, version=98, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:53:15,111][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 99, reason: ApplyCommitRequest{term=7, version=99, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:53:18,807][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 101, reason: ApplyCommitRequest{term=7, version=101, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:53:22,391][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 105, reason: ApplyCommitRequest{term=7, version=105, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:53:54,625][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T04:53:54,676][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 135, reason: becoming candidate: onLeaderFailure [2023-04-01T04:53:55,538][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 137, reason: ApplyCommitRequest{term=7, version=137, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T04:55:55,309][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T04:55:55,372][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 168, reason: becoming candidate: onLeaderFailure [2023-04-01T04:56:00,476][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-01T04:56:00,573][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-01T04:56:03,372][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 172, reason: ApplyCommitRequest{term=7, version=172, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:00:44,186][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][4017] overhead, spent [316ms] collecting in the last [1s] [2023-04-01T05:53:15,964][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 242, reason: ApplyCommitRequest{term=7, version=242, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:53:16,172][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,273][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,374][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,385][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,705][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,790][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,896][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T05:53:16,982][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,083][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,197][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,281][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,312][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,429][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,510][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,602][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,680][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,806][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T05:53:17,900][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T05:53:18,011][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T05:53:18,084][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T05:53:18,193][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T05:53:18,225][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 243, reason: ApplyCommitRequest{term=7, version=243, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:53:18,893][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 244, reason: ApplyCommitRequest{term=7, version=244, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:53:19,250][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 245, reason: ApplyCommitRequest{term=7, version=245, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:53:21,418][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 246, reason: ApplyCommitRequest{term=7, version=246, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:53:26,794][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 250, reason: ApplyCommitRequest{term=7, version=250, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:53:54,871][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T05:53:54,873][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 288, reason: becoming candidate: onLeaderFailure [2023-04-01T05:53:56,337][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 289, reason: ApplyCommitRequest{term=7, version=289, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T05:56:04,002][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T05:56:04,173][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 312, reason: becoming candidate: onLeaderFailure [2023-04-01T05:56:08,226][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 318, reason: ApplyCommitRequest{term=7, version=318, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:01:29,749][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:48852}], closing connection [2023-04-01T06:53:20,341][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 389, reason: ApplyCommitRequest{term=7, version=389, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:53:20,514][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T06:53:20,673][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T06:53:20,683][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T06:53:20,896][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,075][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,201][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,294][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,381][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,471][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,521][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,604][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,671][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,716][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,785][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,826][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T06:53:21,915][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T06:53:22,002][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T06:53:22,085][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T06:53:22,174][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T06:53:22,289][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T06:53:22,385][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T06:53:22,406][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 390, reason: ApplyCommitRequest{term=7, version=390, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:53:22,727][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 391, reason: ApplyCommitRequest{term=7, version=391, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:53:23,347][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 392, reason: ApplyCommitRequest{term=7, version=392, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:53:25,865][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 393, reason: ApplyCommitRequest{term=7, version=393, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:53:29,377][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 396, reason: ApplyCommitRequest{term=7, version=396, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:53:55,326][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T06:53:55,329][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 439, reason: becoming candidate: onLeaderFailure [2023-04-01T06:53:56,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 441, reason: ApplyCommitRequest{term=7, version=441, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T06:56:09,711][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T06:56:09,725][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 459, reason: becoming candidate: onLeaderFailure [2023-04-01T06:56:09,972][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-01T06:56:09,975][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-01T06:56:11,435][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 461, reason: ApplyCommitRequest{term=7, version=461, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T07:06:21,915][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:60614}], closing connection [2023-04-01T07:53:24,602][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 532, reason: ApplyCommitRequest{term=7, version=532, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T07:53:24,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T07:53:24,775][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T07:53:24,811][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T07:53:24,909][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T07:53:24,974][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,109][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,199][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,282][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,372][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,414][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,497][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,579][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,598][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,688][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,799][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,889][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T07:53:25,977][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T07:53:26,097][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T07:53:26,271][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T07:53:26,281][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T07:53:26,755][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 533, reason: ApplyCommitRequest{term=7, version=533, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T07:53:28,549][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 534, reason: ApplyCommitRequest{term=7, version=534, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T07:53:32,663][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 537, reason: ApplyCommitRequest{term=7, version=537, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T07:53:55,421][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T07:53:55,429][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 569, reason: becoming candidate: onLeaderFailure [2023-04-01T07:53:55,801][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 571, reason: ApplyCommitRequest{term=7, version=571, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T07:56:13,518][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T07:56:13,575][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 600, reason: becoming candidate: onLeaderFailure [2023-04-01T07:56:14,574][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 602, reason: ApplyCommitRequest{term=7, version=602, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T08:06:18,490][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-2] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:34602}], closing connection [2023-04-01T08:53:27,558][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 672, reason: ApplyCommitRequest{term=7, version=672, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T08:53:27,673][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T08:53:27,722][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T08:53:27,797][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T08:53:27,871][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T08:53:27,914][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T08:53:27,980][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,078][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,110][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,191][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,272][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,371][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,386][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,482][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,506][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,583][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,609][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,688][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,781][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,816][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,913][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T08:53:28,997][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T08:53:29,026][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T08:53:29,427][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 673, reason: ApplyCommitRequest{term=7, version=673, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T08:53:31,206][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 674, reason: ApplyCommitRequest{term=7, version=674, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T08:53:33,935][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 678, reason: ApplyCommitRequest{term=7, version=678, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T08:53:55,780][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T08:53:55,783][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 712, reason: becoming candidate: onLeaderFailure [2023-04-01T08:53:56,452][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 714, reason: ApplyCommitRequest{term=7, version=714, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T08:56:16,145][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T08:56:16,157][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 741, reason: becoming candidate: onLeaderFailure [2023-04-01T08:56:18,098][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 745, reason: ApplyCommitRequest{term=7, version=745, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T09:53:30,194][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 815, reason: ApplyCommitRequest{term=7, version=815, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T09:53:30,319][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,377][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,413][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,514][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,587][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,691][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,772][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T09:53:30,982][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,176][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,286][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,331][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,374][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,436][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,488][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,582][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,675][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,719][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,809][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,879][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T09:53:31,973][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T09:53:32,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T09:53:32,101][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T09:53:32,434][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 816, reason: ApplyCommitRequest{term=7, version=816, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T09:53:34,174][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 817, reason: ApplyCommitRequest{term=7, version=817, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T09:53:37,466][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 821, reason: ApplyCommitRequest{term=7, version=821, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T09:53:55,971][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T09:53:55,973][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 853, reason: becoming candidate: onLeaderFailure [2023-04-01T09:53:56,177][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 854, reason: ApplyCommitRequest{term=7, version=854, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T09:56:19,904][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T09:56:19,909][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 884, reason: becoming candidate: onLeaderFailure [2023-04-01T09:56:21,562][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 888, reason: ApplyCommitRequest{term=7, version=888, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:33,481][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 959, reason: ApplyCommitRequest{term=7, version=959, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:33,607][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T10:53:33,701][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T10:53:33,780][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T10:53:33,822][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T10:53:33,923][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,009][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,106][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,189][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,293][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,310][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,415][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,508][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,616][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,681][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,788][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T10:53:34,918][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,007][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,079][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,176][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,298][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,343][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T10:53:35,506][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 960, reason: ApplyCommitRequest{term=7, version=960, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:36,104][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 961, reason: ApplyCommitRequest{term=7, version=961, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:36,708][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 962, reason: ApplyCommitRequest{term=7, version=962, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:38,088][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 963, reason: ApplyCommitRequest{term=7, version=963, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:40,914][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 968, reason: ApplyCommitRequest{term=7, version=968, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:53:56,064][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T10:53:56,067][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 992, reason: becoming candidate: onLeaderFailure [2023-04-01T10:53:56,550][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 993, reason: ApplyCommitRequest{term=7, version=993, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T10:56:23,324][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T10:56:23,338][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1030, reason: becoming candidate: onLeaderFailure [2023-04-01T10:56:24,180][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1032, reason: ApplyCommitRequest{term=7, version=1032, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:37,551][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1103, reason: ApplyCommitRequest{term=7, version=1103, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:37,675][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T11:53:37,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T11:53:37,806][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T11:53:37,887][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,001][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,072][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,112][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,271][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,373][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,471][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,507][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,682][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,711][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,806][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,871][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,952][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T11:53:38,982][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T11:53:39,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T11:53:39,110][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T11:53:39,175][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T11:53:39,208][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T11:53:39,272][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T11:53:39,301][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1104, reason: ApplyCommitRequest{term=7, version=1104, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:39,693][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1105, reason: ApplyCommitRequest{term=7, version=1105, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:40,356][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1106, reason: ApplyCommitRequest{term=7, version=1106, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:40,453][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1107, reason: ApplyCommitRequest{term=7, version=1107, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:43,375][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1110, reason: ApplyCommitRequest{term=7, version=1110, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:53:56,096][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T11:53:56,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1135, reason: becoming candidate: onLeaderFailure [2023-04-01T11:53:57,282][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1136, reason: ApplyCommitRequest{term=7, version=1136, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T11:56:25,968][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T11:56:25,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1175, reason: becoming candidate: onLeaderFailure [2023-04-01T11:56:28,701][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1181, reason: ApplyCommitRequest{term=7, version=1181, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:41,375][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1250, reason: ApplyCommitRequest{term=7, version=1250, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:41,509][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T12:53:41,673][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T12:53:41,697][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T12:53:41,872][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T12:53:41,888][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T12:53:41,981][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,072][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,125][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,372][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,416][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,678][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,787][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,976][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T12:53:42,999][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,080][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,108][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,272][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,473][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,571][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,605][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,693][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,723][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T12:53:43,783][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1251, reason: ApplyCommitRequest{term=7, version=1251, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:44,012][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1252, reason: ApplyCommitRequest{term=7, version=1252, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:44,278][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1253, reason: ApplyCommitRequest{term=7, version=1253, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:44,548][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1254, reason: ApplyCommitRequest{term=7, version=1254, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:46,742][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1256, reason: ApplyCommitRequest{term=7, version=1256, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:53:56,244][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T12:53:56,247][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1273, reason: becoming candidate: onLeaderFailure [2023-04-01T12:53:56,607][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1275, reason: ApplyCommitRequest{term=7, version=1275, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T12:56:30,300][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T12:56:30,313][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1322, reason: becoming candidate: onLeaderFailure [2023-04-01T12:56:31,271][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1324, reason: ApplyCommitRequest{term=7, version=1324, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:45,414][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1395, reason: ApplyCommitRequest{term=7, version=1395, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:45,577][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T13:53:45,611][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T13:53:45,707][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T13:53:45,773][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T13:53:45,899][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T13:53:45,993][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,114][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,210][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,392][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,582][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,777][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T13:53:46,980][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,177][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,473][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,675][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,772][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,799][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,883][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T13:53:47,993][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T13:53:48,105][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T13:53:48,179][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1396, reason: ApplyCommitRequest{term=7, version=1396, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:48,361][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1397, reason: ApplyCommitRequest{term=7, version=1397, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:48,964][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1398, reason: ApplyCommitRequest{term=7, version=1398, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:49,488][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1399, reason: ApplyCommitRequest{term=7, version=1399, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:51,126][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1400, reason: ApplyCommitRequest{term=7, version=1400, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:53:56,303][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T13:53:56,305][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1407, reason: becoming candidate: onLeaderFailure [2023-04-01T13:53:56,674][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1408, reason: ApplyCommitRequest{term=7, version=1408, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T13:56:33,487][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T13:56:33,496][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1466, reason: becoming candidate: onLeaderFailure [2023-04-01T13:56:34,239][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1468, reason: ApplyCommitRequest{term=7, version=1468, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:50,549][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1538, reason: ApplyCommitRequest{term=7, version=1538, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:50,676][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T14:53:50,713][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T14:53:50,980][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,188][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,218][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,302][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,379][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,494][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,578][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,600][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,703][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,790][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T14:53:51,988][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,191][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,279][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,373][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,415][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,487][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,593][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,703][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1539, reason: ApplyCommitRequest{term=7, version=1539, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:52,709][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T14:53:52,944][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1540, reason: ApplyCommitRequest{term=7, version=1540, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:53,540][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1541, reason: ApplyCommitRequest{term=7, version=1541, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:53,850][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1542, reason: ApplyCommitRequest{term=7, version=1542, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:54,545][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1543, reason: ApplyCommitRequest{term=7, version=1543, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:54,941][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1544, reason: ApplyCommitRequest{term=7, version=1544, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:55,545][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1545, reason: ApplyCommitRequest{term=7, version=1545, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:55,846][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1546, reason: ApplyCommitRequest{term=7, version=1546, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:53:56,334][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T14:53:56,336][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1546, reason: becoming candidate: onLeaderFailure [2023-04-01T14:53:56,663][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1547, reason: ApplyCommitRequest{term=7, version=1547, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T14:56:36,350][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T14:56:36,361][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1614, reason: becoming candidate: onLeaderFailure [2023-04-01T14:56:39,126][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 7, version: 1618, reason: ApplyCommitRequest{term=7, version=1618, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T15:53:55,549][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 7, version: 1688, reason: ApplyCommitRequest{term=7, version=1688, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T15:53:57,368][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T15:53:57,370][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 7, version: 1688, reason: becoming candidate: onLeaderFailure [2023-04-01T15:53:57,380][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 7, version: 1689, reason: ApplyCommitRequest{term=7, version=1689, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T15:53:57,611][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T15:53:57,783][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T15:53:57,875][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T15:53:57,972][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,073][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,173][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,273][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,373][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,495][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,589][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,701][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,871][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T15:53:58,871][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.NotMasterException: Node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] not master for join request [2023-04-01T15:53:59,079][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,184][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,279][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,407][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,503][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,600][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,704][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,790][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,903][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T15:53:59,974][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T15:54:00,002][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1690, reason: ApplyCommitRequest{term=9, version=1690, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T15:54:00,086][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-2] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 9, version: 1691, delta: master node changed {previous [], current [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}]} [2023-04-01T15:54:00,175][WARN ][o.e.c.s.MasterService ] [onap-sdnrdb-master-2] failing [elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1691] org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 9 while handling publication at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-01T15:54:00,178][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=7, optionalJoin=Optional[Join{term=8, lastAcceptedTerm=7, lastAcceptedVersion=1689, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 9 while handling publication at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-01T15:54:00,639][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1691, reason: ApplyCommitRequest{term=9, version=1691, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T15:56:40,737][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T15:56:40,750][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1757, reason: becoming candidate: onLeaderFailure [2023-04-01T15:56:41,754][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1759, reason: ApplyCommitRequest{term=9, version=1759, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:53:57,400][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T16:53:57,404][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1829, reason: becoming candidate: onLeaderFailure [2023-04-01T16:53:57,686][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1830, reason: ApplyCommitRequest{term=9, version=1830, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:53:58,064][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1831, reason: ApplyCommitRequest{term=9, version=1831, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:53:59,578][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1832, reason: ApplyCommitRequest{term=9, version=1832, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:53:59,973][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,071][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,111][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,211][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,293][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,403][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,487][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,576][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,671][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,723][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,794][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,885][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T16:54:00,920][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,003][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,038][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,100][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,134][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,183][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,233][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,278][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,315][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,362][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T16:54:01,384][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1833, reason: ApplyCommitRequest{term=9, version=1833, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:54:01,860][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1834, reason: ApplyCommitRequest{term=9, version=1834, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:56:43,578][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T16:56:43,589][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1899, reason: becoming candidate: onLeaderFailure [2023-04-01T16:56:44,305][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1901, reason: ApplyCommitRequest{term=9, version=1901, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T16:57:13,383][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-2] [gc][46981] overhead, spent [319ms] collecting in the last [1s] [2023-04-01T17:53:57,425][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.181:9300][disconnected] disconnected [2023-04-01T17:53:57,428][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 1971, reason: becoming candidate: onLeaderFailure [2023-04-01T17:53:57,596][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 1972, reason: ApplyCommitRequest{term=9, version=1972, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:00,693][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1973, reason: ApplyCommitRequest{term=9, version=1973, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:02,361][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1974, reason: ApplyCommitRequest{term=9, version=1974, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:02,490][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T17:54:02,578][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T17:54:02,612][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-04-01T17:54:02,715][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-01T17:54:02,805][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T17:54:02,880][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [cmlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T17:54:02,972][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][2] primary-replica resync completed with 0 operations [2023-04-01T17:54:03,073][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-01T17:54:03,173][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-01T17:54:03,479][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [networkelement-connection-v7][0] primary-replica resync completed with 0 operations [2023-04-01T17:54:03,571][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-04-01T17:54:03,871][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultlog-v7][1] primary-replica resync completed with 0 operations [2023-04-01T17:54:04,072][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-01T17:54:04,183][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][3] primary-replica resync completed with 0 operations [2023-04-01T17:54:04,477][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [connectionlog-v7][0] primary-replica resync completed with 0 operations [2023-04-01T17:54:04,579][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-01T17:54:04,679][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-01T17:54:04,878][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][2] primary-replica resync completed with 0 operations [2023-04-01T17:54:05,017][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [faultcurrent-v7][0] primary-replica resync completed with 0 operations [2023-04-01T17:54:05,117][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-01T17:54:05,199][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-01T17:54:05,230][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-2] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations [2023-04-01T17:54:05,248][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr},{onap-sdnrdb-coordinating-only-54c85fc557-rpdht}{RpZKxuxrSRWWGbSlFMER2Q}{Pw7CHW9yTbqr8U9eZSRxeg}{10.233.66.44}{10.233.66.44:9300}{r}}, term: 9, version: 1975, reason: ApplyCommitRequest{term=9, version=1975, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:05,613][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1976, reason: ApplyCommitRequest{term=9, version=1976, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:06,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1977, reason: ApplyCommitRequest{term=9, version=1977, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:06,478][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1978, reason: ApplyCommitRequest{term=9, version=1978, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:06,619][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1979, reason: ApplyCommitRequest{term=9, version=1979, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:06,812][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] removed {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1980, reason: ApplyCommitRequest{term=9, version=1980, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:54:07,590][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] added {{onap-sdnrdb-master-1}{oBVWcRcCQa2pTEZnFAZuXQ}{acj9if5DQhONf4qfezXo6Q}{10.233.68.45}{10.233.68.45:9300}{dmr}}, term: 9, version: 1981, reason: ApplyCommitRequest{term=9, version=1981, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}} [2023-04-01T17:56:46,629][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-2] master node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}] has been removed from the cluster at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2023-04-01T17:56:46,639][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}], current []}, term: 9, version: 2046, reason: becoming candidate: onLeaderFailure [2023-04-01T17:56:46,872][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=9, optionalJoin=Optional[Join{term=9, lastAcceptedTerm=7, lastAcceptedVersion=1689, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-01T17:56:46,875][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-2] failed to join {onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, minimumTerm=9, optionalJoin=Optional[Join{term=9, lastAcceptedTerm=7, lastAcceptedVersion=1689, sourceNode={onap-sdnrdb-master-2}{y71Y2EKcTU6nD12jJL8k8A}{11UWgd1qTqiWon9k0RmSHg}{10.233.65.70}{10.233.65.70:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.181:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:519) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.65.70:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-01T17:56:48,287][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-2] master node changed {previous [], current [{onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}]}, term: 9, version: 2048, reason: ApplyCommitRequest{term=9, version=2048, sourceNode={onap-sdnrdb-master-0}{UYq294gRQ3W6BCfWxYv4Rw}{BMz-YV3dQbeeqF74zeGo4A}{10.233.67.181}{10.233.67.181:9300}{dmr}}