Results

By type
Logs
raw version
          [38;5;6m [38;5;5m14:49:28.84 [0m
[38;5;6m [38;5;5m14:49:28.85 [0m[1mWelcome to the Bitnami elasticsearch container[0m
[38;5;6m [38;5;5m14:49:28.85 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m
[38;5;6m [38;5;5m14:49:28.86 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m
[38;5;6m [38;5;5m14:49:28.86 [0m
[38;5;6m [38;5;5m14:49:28.87 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup **
[38;5;6m [38;5;5m14:49:28.97 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch...
[38;5;6m [38;5;5m14:49:29.04 [0m[38;5;2mINFO [0m ==> Setting default configuration
[38;5;6m [38;5;5m14:49:29.07 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings...
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[38;5;6m [38;5;5m14:49:33.36 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! **

[38;5;6m [38;5;5m14:49:33.41 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-10-14T14:49:37,510][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS]
[2022-10-14T14:49:37,513][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] JVM home [/opt/bitnami/java]
[2022-10-14T14:49:37,513][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-1933773947394273470, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-10-14T14:49:38,891][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [aggs-matrix-stats]
[2022-10-14T14:49:38,892][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [analysis-common]
[2022-10-14T14:49:38,892][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [geo]
[2022-10-14T14:49:38,893][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [ingest-common]
[2022-10-14T14:49:38,893][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [ingest-geoip]
[2022-10-14T14:49:38,894][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [ingest-user-agent]
[2022-10-14T14:49:38,894][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [kibana]
[2022-10-14T14:49:38,894][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [lang-expression]
[2022-10-14T14:49:38,894][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [lang-mustache]
[2022-10-14T14:49:38,895][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [lang-painless]
[2022-10-14T14:49:38,895][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [mapper-extras]
[2022-10-14T14:49:38,895][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [parent-join]
[2022-10-14T14:49:38,895][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [percolator]
[2022-10-14T14:49:38,895][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [rank-eval]
[2022-10-14T14:49:38,896][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [reindex]
[2022-10-14T14:49:38,896][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [repository-url]
[2022-10-14T14:49:38,896][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [tasks]
[2022-10-14T14:49:38,897][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded module [transport-netty4]
[2022-10-14T14:49:38,898][INFO ][o.e.p.PluginsService     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] loaded plugin [repository-s3]
[2022-10-14T14:49:38,947][INFO ][o.e.e.NodeEnvironment    ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/vda1)]], net usable_space [152.1gb], net total_space [154.8gb], types [ext4]
[2022-10-14T14:49:38,948][INFO ][o.e.e.NodeEnvironment    ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] heap size [123.7mb], compressed ordinary object pointers [true]
[2022-10-14T14:49:39,012][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] node name [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z], node ID [hsjBrGCOSEWNqrJ6xZUj2g], cluster name [sdnrdb-cluster]
[2022-10-14T14:49:42,566][INFO ][o.e.t.NettyAllocator     ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}]
[2022-10-14T14:49:42,640][INFO ][o.e.d.DiscoveryModule    ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] using discovery type [zen] and seed hosts providers [settings]
[2022-10-14T14:49:43,031][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-10-14T14:49:43,231][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] initialized
[2022-10-14T14:49:43,232][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] starting ...
[2022-10-14T14:49:43,395][INFO ][o.e.t.TransportService   ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] publish_address {10.233.69.201:9300}, bound_addresses {0.0.0.0:9300}
[2022-10-14T14:49:43,632][INFO ][o.e.b.BootstrapChecks    ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-10-14T14:52:07,717][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, added {{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr},{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2, reason: ApplyCommitRequest{term=2, version=2, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T14:52:07,739][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] publish_address {10.233.69.201:9000}, bound_addresses {0.0.0.0:9000}
[2022-10-14T14:52:07,739][INFO ][o.e.n.Node               ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] started
[2022-10-14T14:52:22,612][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4, reason: ApplyCommitRequest{term=2, version=4, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T14:52:31,675][INFO ][o.e.c.s.ClusterSettings  ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] updating [action.auto_create_index] from [true] to [false]
[2022-10-14T15:51:59,258][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T15:51:59,279][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 85, reason: becoming candidate: onLeaderFailure
[2022-10-14T15:51:59,665][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 86, reason: ApplyCommitRequest{term=2, version=86, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:08,483][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T15:52:08,498][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 87, reason: becoming candidate: onLeaderFailure
[2022-10-14T15:52:08,781][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 89, reason: ApplyCommitRequest{term=2, version=89, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:23,548][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 91, reason: ApplyCommitRequest{term=2, version=91, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:26,332][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 92, reason: ApplyCommitRequest{term=2, version=92, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:27,042][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 93, reason: ApplyCommitRequest{term=2, version=93, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:52:27,833][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 94, reason: ApplyCommitRequest{term=2, version=94, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:43,842][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 163, reason: ApplyCommitRequest{term=2, version=163, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:48,347][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 164, reason: ApplyCommitRequest{term=2, version=164, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:49,035][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 165, reason: ApplyCommitRequest{term=2, version=165, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:49,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 166, reason: ApplyCommitRequest{term=2, version=166, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:49,738][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 167, reason: ApplyCommitRequest{term=2, version=167, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T15:53:50,332][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 168, reason: ApplyCommitRequest{term=2, version=168, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:51:59,300][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T16:51:59,336][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 235, reason: becoming candidate: onLeaderFailure
[2022-10-14T16:51:59,469][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 236, reason: ApplyCommitRequest{term=2, version=236, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:11,414][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T16:52:11,426][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 237, reason: becoming candidate: onLeaderFailure
[2022-10-14T16:52:11,505][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 239, reason: ApplyCommitRequest{term=2, version=239, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:28,213][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 241, reason: ApplyCommitRequest{term=2, version=241, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:52:30,069][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 242, reason: ApplyCommitRequest{term=2, version=242, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:53:51,010][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 312, reason: ApplyCommitRequest{term=2, version=312, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T16:53:53,097][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 313, reason: ApplyCommitRequest{term=2, version=313, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:51:59,328][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T17:51:59,338][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 380, reason: becoming candidate: onLeaderFailure
[2022-10-14T17:51:59,522][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 381, reason: ApplyCommitRequest{term=2, version=381, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:14,455][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T17:52:14,472][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 382, reason: becoming candidate: onLeaderFailure
[2022-10-14T17:52:14,611][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 384, reason: ApplyCommitRequest{term=2, version=384, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:30,400][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 386, reason: ApplyCommitRequest{term=2, version=386, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:52:32,914][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 387, reason: ApplyCommitRequest{term=2, version=387, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:53:53,945][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 458, reason: ApplyCommitRequest{term=2, version=458, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T17:53:56,056][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 459, reason: ApplyCommitRequest{term=2, version=459, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:51:59,355][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T18:51:59,362][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 526, reason: becoming candidate: onLeaderFailure
[2022-10-14T18:51:59,458][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 527, reason: ApplyCommitRequest{term=2, version=527, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:17,455][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T18:52:17,466][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 528, reason: becoming candidate: onLeaderFailure
[2022-10-14T18:52:17,630][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 530, reason: ApplyCommitRequest{term=2, version=530, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:33,296][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 532, reason: ApplyCommitRequest{term=2, version=532, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:52:34,982][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 533, reason: ApplyCommitRequest{term=2, version=533, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:53:56,681][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 603, reason: ApplyCommitRequest{term=2, version=603, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T18:53:59,148][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 604, reason: ApplyCommitRequest{term=2, version=604, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:51:59,381][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T19:51:59,388][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 670, reason: becoming candidate: onLeaderFailure
[2022-10-14T19:51:59,597][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 671, reason: ApplyCommitRequest{term=2, version=671, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:19,640][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T19:52:19,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 672, reason: becoming candidate: onLeaderFailure
[2022-10-14T19:52:19,740][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 675, reason: ApplyCommitRequest{term=2, version=675, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:35,477][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 676, reason: ApplyCommitRequest{term=2, version=676, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:52:37,962][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 677, reason: ApplyCommitRequest{term=2, version=677, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:53:59,783][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 745, reason: ApplyCommitRequest{term=2, version=745, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:54:01,953][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 746, reason: ApplyCommitRequest{term=2, version=746, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:54:02,250][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 747, reason: ApplyCommitRequest{term=2, version=747, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:54:02,449][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 748, reason: ApplyCommitRequest{term=2, version=748, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:54:02,758][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 749, reason: ApplyCommitRequest{term=2, version=749, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T19:54:03,247][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 750, reason: ApplyCommitRequest{term=2, version=750, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:51:59,406][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T20:51:59,417][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 818, reason: becoming candidate: onLeaderFailure
[2022-10-14T20:51:59,617][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 819, reason: ApplyCommitRequest{term=2, version=819, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:22,633][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T20:52:22,642][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 821, reason: becoming candidate: onLeaderFailure
[2022-10-14T20:52:22,797][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 823, reason: ApplyCommitRequest{term=2, version=823, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:38,452][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 824, reason: ApplyCommitRequest{term=2, version=824, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:52:40,583][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 825, reason: ApplyCommitRequest{term=2, version=825, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:54:03,988][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 895, reason: ApplyCommitRequest{term=2, version=895, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T20:54:06,299][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 896, reason: ApplyCommitRequest{term=2, version=896, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:51:59,428][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T21:51:59,434][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 965, reason: becoming candidate: onLeaderFailure
[2022-10-14T21:51:59,809][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 966, reason: ApplyCommitRequest{term=2, version=966, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:25,678][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T21:52:25,686][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 968, reason: becoming candidate: onLeaderFailure
[2022-10-14T21:52:25,768][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 970, reason: ApplyCommitRequest{term=2, version=970, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:41,124][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 971, reason: ApplyCommitRequest{term=2, version=971, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:52:43,405][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 972, reason: ApplyCommitRequest{term=2, version=972, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:54:07,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1041, reason: ApplyCommitRequest{term=2, version=1041, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T21:54:09,107][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1042, reason: ApplyCommitRequest{term=2, version=1042, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:51:59,450][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T22:51:59,456][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1112, reason: becoming candidate: onLeaderFailure
[2022-10-14T22:51:59,574][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1113, reason: ApplyCommitRequest{term=2, version=1113, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:28,588][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T22:52:28,595][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1115, reason: becoming candidate: onLeaderFailure
[2022-10-14T22:52:28,722][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1117, reason: ApplyCommitRequest{term=2, version=1117, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:43,714][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1118, reason: ApplyCommitRequest{term=2, version=1118, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:52:46,537][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1119, reason: ApplyCommitRequest{term=2, version=1119, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:54:10,098][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1188, reason: ApplyCommitRequest{term=2, version=1188, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:54:11,838][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1189, reason: ApplyCommitRequest{term=2, version=1189, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:54:11,960][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1190, reason: ApplyCommitRequest{term=2, version=1190, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T22:54:12,328][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1191, reason: ApplyCommitRequest{term=2, version=1191, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:51:59,473][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-14T23:51:59,478][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1260, reason: becoming candidate: onLeaderFailure
[2022-10-14T23:51:59,697][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1261, reason: ApplyCommitRequest{term=2, version=1261, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:31,659][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-14T23:52:31,667][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1263, reason: becoming candidate: onLeaderFailure
[2022-10-14T23:52:31,776][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1265, reason: ApplyCommitRequest{term=2, version=1265, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:46,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1266, reason: ApplyCommitRequest{term=2, version=1266, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:52:49,080][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1267, reason: ApplyCommitRequest{term=2, version=1267, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:54:13,116][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1337, reason: ApplyCommitRequest{term=2, version=1337, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-14T23:54:15,382][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1338, reason: ApplyCommitRequest{term=2, version=1338, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:51:59,487][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T00:51:59,513][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1404, reason: becoming candidate: onLeaderFailure
[2022-10-15T00:51:59,808][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1405, reason: ApplyCommitRequest{term=2, version=1405, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:34,695][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T00:52:34,706][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1407, reason: becoming candidate: onLeaderFailure
[2022-10-15T00:52:34,820][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1409, reason: ApplyCommitRequest{term=2, version=1409, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:49,361][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1410, reason: ApplyCommitRequest{term=2, version=1410, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:52:51,064][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1411, reason: ApplyCommitRequest{term=2, version=1411, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:54:16,307][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1479, reason: ApplyCommitRequest{term=2, version=1479, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T00:54:17,805][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1480, reason: ApplyCommitRequest{term=2, version=1480, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:51:59,523][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T01:51:59,532][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1551, reason: becoming candidate: onLeaderFailure
[2022-10-15T01:51:59,673][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1552, reason: ApplyCommitRequest{term=2, version=1552, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:37,738][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T01:52:37,749][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1554, reason: becoming candidate: onLeaderFailure
[2022-10-15T01:52:37,854][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1556, reason: ApplyCommitRequest{term=2, version=1556, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:51,512][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1557, reason: ApplyCommitRequest{term=2, version=1557, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:52:53,473][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1558, reason: ApplyCommitRequest{term=2, version=1558, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:54:18,534][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1627, reason: ApplyCommitRequest{term=2, version=1627, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T01:54:20,669][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1628, reason: ApplyCommitRequest{term=2, version=1628, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:51:59,543][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T02:51:59,548][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1696, reason: becoming candidate: onLeaderFailure
[2022-10-15T02:51:59,783][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1697, reason: ApplyCommitRequest{term=2, version=1697, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:39,865][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T02:52:39,874][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1699, reason: becoming candidate: onLeaderFailure
[2022-10-15T02:52:39,974][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1701, reason: ApplyCommitRequest{term=2, version=1701, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:53,761][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1702, reason: ApplyCommitRequest{term=2, version=1702, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:52:56,267][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1703, reason: ApplyCommitRequest{term=2, version=1703, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:54:21,288][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1772, reason: ApplyCommitRequest{term=2, version=1772, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T02:54:23,771][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1773, reason: ApplyCommitRequest{term=2, version=1773, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:51:59,563][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T03:51:59,569][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1842, reason: becoming candidate: onLeaderFailure
[2022-10-15T03:51:59,788][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1843, reason: ApplyCommitRequest{term=2, version=1843, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:42,716][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T03:52:42,724][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1845, reason: becoming candidate: onLeaderFailure
[2022-10-15T03:52:42,810][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1847, reason: ApplyCommitRequest{term=2, version=1847, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:56,561][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1848, reason: ApplyCommitRequest{term=2, version=1848, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:52:58,876][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1849, reason: ApplyCommitRequest{term=2, version=1849, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:54:25,052][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1917, reason: ApplyCommitRequest{term=2, version=1917, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:54:26,954][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1918, reason: ApplyCommitRequest{term=2, version=1918, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:54:27,289][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1919, reason: ApplyCommitRequest{term=2, version=1919, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T03:54:27,711][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 1920, reason: ApplyCommitRequest{term=2, version=1920, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:51:59,582][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T04:51:59,598][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1990, reason: becoming candidate: onLeaderFailure
[2022-10-15T04:51:59,672][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1991, reason: ApplyCommitRequest{term=2, version=1991, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:45,749][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T04:52:45,757][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 1993, reason: becoming candidate: onLeaderFailure
[2022-10-15T04:52:45,842][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 1995, reason: ApplyCommitRequest{term=2, version=1995, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:52:59,071][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1996, reason: ApplyCommitRequest{term=2, version=1996, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:53:00,828][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 1997, reason: ApplyCommitRequest{term=2, version=1997, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:54:28,528][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2066, reason: ApplyCommitRequest{term=2, version=2066, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:54:31,141][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2067, reason: ApplyCommitRequest{term=2, version=2067, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:54:31,534][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2068, reason: ApplyCommitRequest{term=2, version=2068, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T04:54:32,034][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2069, reason: ApplyCommitRequest{term=2, version=2069, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:51:59,611][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T05:51:59,617][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2138, reason: becoming candidate: onLeaderFailure
[2022-10-15T05:51:59,875][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2139, reason: ApplyCommitRequest{term=2, version=2139, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:52:47,997][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T05:52:48,004][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2141, reason: becoming candidate: onLeaderFailure
[2022-10-15T05:52:48,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2143, reason: ApplyCommitRequest{term=2, version=2143, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:53:00,913][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2144, reason: ApplyCommitRequest{term=2, version=2144, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:53:03,698][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2145, reason: ApplyCommitRequest{term=2, version=2145, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:54:32,759][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2214, reason: ApplyCommitRequest{term=2, version=2214, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T05:54:35,164][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2215, reason: ApplyCommitRequest{term=2, version=2215, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:51:59,627][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T06:51:59,635][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2283, reason: becoming candidate: onLeaderFailure
[2022-10-15T06:51:59,710][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2284, reason: ApplyCommitRequest{term=2, version=2284, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:52:50,807][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T06:52:50,816][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2286, reason: becoming candidate: onLeaderFailure
[2022-10-15T06:52:50,901][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2288, reason: ApplyCommitRequest{term=2, version=2288, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:53:03,859][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2289, reason: ApplyCommitRequest{term=2, version=2289, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:53:05,581][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2290, reason: ApplyCommitRequest{term=2, version=2290, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:54:35,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2359, reason: ApplyCommitRequest{term=2, version=2359, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T06:54:38,075][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2360, reason: ApplyCommitRequest{term=2, version=2360, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:51:59,646][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T07:51:59,652][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2428, reason: becoming candidate: onLeaderFailure
[2022-10-15T07:51:59,727][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2429, reason: ApplyCommitRequest{term=2, version=2429, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:52:53,818][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T07:52:53,826][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2431, reason: becoming candidate: onLeaderFailure
[2022-10-15T07:52:53,931][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2433, reason: ApplyCommitRequest{term=2, version=2433, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:53:05,746][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2434, reason: ApplyCommitRequest{term=2, version=2434, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:53:07,874][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2435, reason: ApplyCommitRequest{term=2, version=2435, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:54:38,701][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2504, reason: ApplyCommitRequest{term=2, version=2504, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T07:54:41,024][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2505, reason: ApplyCommitRequest{term=2, version=2505, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:51:59,664][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T08:51:59,671][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2575, reason: becoming candidate: onLeaderFailure
[2022-10-15T08:51:59,758][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2576, reason: ApplyCommitRequest{term=2, version=2576, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:52:56,865][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T08:52:56,886][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2578, reason: becoming candidate: onLeaderFailure
[2022-10-15T08:52:56,970][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2580, reason: ApplyCommitRequest{term=2, version=2580, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:53:08,640][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2581, reason: ApplyCommitRequest{term=2, version=2581, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:53:09,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2582, reason: ApplyCommitRequest{term=2, version=2582, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:54:41,459][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2652, reason: ApplyCommitRequest{term=2, version=2652, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T08:54:43,941][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2653, reason: ApplyCommitRequest{term=2, version=2653, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:51:59,687][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T09:51:59,699][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2722, reason: becoming candidate: onLeaderFailure
[2022-10-15T09:51:59,772][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2723, reason: ApplyCommitRequest{term=2, version=2723, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:52:59,884][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T09:52:59,893][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2725, reason: becoming candidate: onLeaderFailure
[2022-10-15T09:52:59,985][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2727, reason: ApplyCommitRequest{term=2, version=2727, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:53:10,250][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2728, reason: ApplyCommitRequest{term=2, version=2728, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:53:12,502][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2729, reason: ApplyCommitRequest{term=2, version=2729, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:54:44,467][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2799, reason: ApplyCommitRequest{term=2, version=2799, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:54:46,138][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2800, reason: ApplyCommitRequest{term=2, version=2800, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:54:46,271][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2801, reason: ApplyCommitRequest{term=2, version=2801, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T09:54:46,969][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2802, reason: ApplyCommitRequest{term=2, version=2802, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:51:59,713][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T10:51:59,722][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2871, reason: becoming candidate: onLeaderFailure
[2022-10-15T10:51:59,991][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2872, reason: ApplyCommitRequest{term=2, version=2872, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:02,947][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T10:53:02,958][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 2874, reason: becoming candidate: onLeaderFailure
[2022-10-15T10:53:03,046][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 2876, reason: ApplyCommitRequest{term=2, version=2876, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:12,524][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2877, reason: ApplyCommitRequest{term=2, version=2877, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:53:14,569][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 2878, reason: ApplyCommitRequest{term=2, version=2878, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:54:47,348][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2947, reason: ApplyCommitRequest{term=2, version=2947, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T10:54:50,029][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 2948, reason: ApplyCommitRequest{term=2, version=2948, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:51:59,733][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T11:51:59,740][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3016, reason: becoming candidate: onLeaderFailure
[2022-10-15T11:51:59,817][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3017, reason: ApplyCommitRequest{term=2, version=3017, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:05,933][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T11:53:05,946][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3019, reason: becoming candidate: onLeaderFailure
[2022-10-15T11:53:06,024][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3021, reason: ApplyCommitRequest{term=2, version=3021, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:14,725][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3022, reason: ApplyCommitRequest{term=2, version=3022, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:53:17,075][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3023, reason: ApplyCommitRequest{term=2, version=3023, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:54:50,691][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3093, reason: ApplyCommitRequest{term=2, version=3093, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T11:54:53,034][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3094, reason: ApplyCommitRequest{term=2, version=3094, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:51:59,752][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T12:51:59,760][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3161, reason: becoming candidate: onLeaderFailure
[2022-10-15T12:51:59,834][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3162, reason: ApplyCommitRequest{term=2, version=3162, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:08,970][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T12:53:08,979][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3164, reason: becoming candidate: onLeaderFailure
[2022-10-15T12:53:09,085][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3166, reason: ApplyCommitRequest{term=2, version=3166, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:17,362][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3167, reason: ApplyCommitRequest{term=2, version=3167, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:53:19,813][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3168, reason: ApplyCommitRequest{term=2, version=3168, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:54:53,575][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3238, reason: ApplyCommitRequest{term=2, version=3238, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T12:54:56,043][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3239, reason: ApplyCommitRequest{term=2, version=3239, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:51:59,775][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T13:51:59,787][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3307, reason: becoming candidate: onLeaderFailure
[2022-10-15T13:51:59,994][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3308, reason: ApplyCommitRequest{term=2, version=3308, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:12,038][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T13:53:12,057][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3310, reason: becoming candidate: onLeaderFailure
[2022-10-15T13:53:12,216][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3312, reason: ApplyCommitRequest{term=2, version=3312, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:19,882][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3313, reason: ApplyCommitRequest{term=2, version=3313, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:53:22,678][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3314, reason: ApplyCommitRequest{term=2, version=3314, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:54:57,360][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3382, reason: ApplyCommitRequest{term=2, version=3382, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T13:54:59,005][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3383, reason: ApplyCommitRequest{term=2, version=3383, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:51:59,795][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T14:51:59,808][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3449, reason: becoming candidate: onLeaderFailure
[2022-10-15T14:52:00,109][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3450, reason: ApplyCommitRequest{term=2, version=3450, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:14,187][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T14:53:14,200][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3452, reason: becoming candidate: onLeaderFailure
[2022-10-15T14:53:14,281][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3454, reason: ApplyCommitRequest{term=2, version=3454, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:23,044][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3455, reason: ApplyCommitRequest{term=2, version=3455, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:53:24,818][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3456, reason: ApplyCommitRequest{term=2, version=3456, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:54:59,894][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3525, reason: ApplyCommitRequest{term=2, version=3525, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:55:01,441][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3526, reason: ApplyCommitRequest{term=2, version=3526, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:55:01,613][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3527, reason: ApplyCommitRequest{term=2, version=3527, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T14:55:02,087][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3528, reason: ApplyCommitRequest{term=2, version=3528, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:51:59,820][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T15:51:59,823][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3596, reason: becoming candidate: onLeaderFailure
[2022-10-15T15:51:59,905][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3597, reason: ApplyCommitRequest{term=2, version=3597, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:17,092][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T15:53:17,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3599, reason: becoming candidate: onLeaderFailure
[2022-10-15T15:53:17,201][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3601, reason: ApplyCommitRequest{term=2, version=3601, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:24,968][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3602, reason: ApplyCommitRequest{term=2, version=3602, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:53:27,481][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3603, reason: ApplyCommitRequest{term=2, version=3603, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:55:02,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3671, reason: ApplyCommitRequest{term=2, version=3671, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T15:55:04,933][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3672, reason: ApplyCommitRequest{term=2, version=3672, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:51:59,839][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T16:51:59,847][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3742, reason: becoming candidate: onLeaderFailure
[2022-10-15T16:51:59,920][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3743, reason: ApplyCommitRequest{term=2, version=3743, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:20,081][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T16:53:20,089][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3745, reason: becoming candidate: onLeaderFailure
[2022-10-15T16:53:20,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3747, reason: ApplyCommitRequest{term=2, version=3747, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:28,162][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3748, reason: ApplyCommitRequest{term=2, version=3748, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:30,606][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3749, reason: ApplyCommitRequest{term=2, version=3749, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:30,905][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3750, reason: ApplyCommitRequest{term=2, version=3750, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:53:31,814][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3751, reason: ApplyCommitRequest{term=2, version=3751, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:55:05,328][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3822, reason: ApplyCommitRequest{term=2, version=3822, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T16:55:08,026][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3823, reason: ApplyCommitRequest{term=2, version=3823, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:51:59,860][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T17:51:59,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3891, reason: becoming candidate: onLeaderFailure
[2022-10-15T17:52:00,102][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3892, reason: ApplyCommitRequest{term=2, version=3892, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:23,134][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T17:53:23,147][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 3894, reason: becoming candidate: onLeaderFailure
[2022-10-15T17:53:23,256][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 3896, reason: ApplyCommitRequest{term=2, version=3896, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:31,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3897, reason: ApplyCommitRequest{term=2, version=3897, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:53:33,731][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 3898, reason: ApplyCommitRequest{term=2, version=3898, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:55:08,498][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3965, reason: ApplyCommitRequest{term=2, version=3965, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T17:55:10,429][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 3966, reason: ApplyCommitRequest{term=2, version=3966, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:51:59,879][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T18:51:59,888][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4034, reason: becoming candidate: onLeaderFailure
[2022-10-15T18:51:59,955][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4035, reason: ApplyCommitRequest{term=2, version=4035, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:26,175][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T18:53:26,184][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4037, reason: becoming candidate: onLeaderFailure
[2022-10-15T18:53:26,434][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4039, reason: ApplyCommitRequest{term=2, version=4039, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:33,896][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4040, reason: ApplyCommitRequest{term=2, version=4040, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:53:36,686][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4041, reason: ApplyCommitRequest{term=2, version=4041, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:55:10,868][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4110, reason: ApplyCommitRequest{term=2, version=4110, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T18:55:13,131][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4111, reason: ApplyCommitRequest{term=2, version=4111, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:51:59,901][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T19:51:59,909][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4180, reason: becoming candidate: onLeaderFailure
[2022-10-15T19:52:00,074][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4181, reason: ApplyCommitRequest{term=2, version=4181, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:29,163][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T19:53:29,180][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4183, reason: becoming candidate: onLeaderFailure
[2022-10-15T19:53:29,257][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4185, reason: ApplyCommitRequest{term=2, version=4185, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:36,925][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4186, reason: ApplyCommitRequest{term=2, version=4186, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:53:39,384][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4187, reason: ApplyCommitRequest{term=2, version=4187, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:55:13,563][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4256, reason: ApplyCommitRequest{term=2, version=4256, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T19:55:16,131][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4257, reason: ApplyCommitRequest{term=2, version=4257, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:51:59,919][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T20:51:59,928][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4323, reason: becoming candidate: onLeaderFailure
[2022-10-15T20:52:00,008][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4324, reason: ApplyCommitRequest{term=2, version=4324, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:32,193][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T20:53:32,201][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4326, reason: becoming candidate: onLeaderFailure
[2022-10-15T20:53:32,307][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4328, reason: ApplyCommitRequest{term=2, version=4328, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:39,689][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4329, reason: ApplyCommitRequest{term=2, version=4329, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:53:42,120][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4330, reason: ApplyCommitRequest{term=2, version=4330, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:55:16,471][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4399, reason: ApplyCommitRequest{term=2, version=4399, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T20:55:19,135][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4400, reason: ApplyCommitRequest{term=2, version=4400, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:51:59,940][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T21:51:59,947][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4468, reason: becoming candidate: onLeaderFailure
[2022-10-15T21:52:00,030][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4469, reason: ApplyCommitRequest{term=2, version=4469, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:35,243][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T21:53:35,256][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4471, reason: becoming candidate: onLeaderFailure
[2022-10-15T21:53:35,331][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4473, reason: ApplyCommitRequest{term=2, version=4473, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:42,170][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4474, reason: ApplyCommitRequest{term=2, version=4474, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:53:45,023][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4475, reason: ApplyCommitRequest{term=2, version=4475, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:55:19,599][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4543, reason: ApplyCommitRequest{term=2, version=4543, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T21:55:21,375][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4544, reason: ApplyCommitRequest{term=2, version=4544, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T22:51:59,966][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T22:51:59,973][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4613, reason: becoming candidate: onLeaderFailure
[2022-10-15T22:52:00,048][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4614, reason: ApplyCommitRequest{term=2, version=4614, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T22:53:38,240][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T22:53:38,247][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4616, reason: becoming candidate: onLeaderFailure
[2022-10-15T22:53:38,444][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4618, reason: ApplyCommitRequest{term=2, version=4618, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T22:53:45,211][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4619, reason: ApplyCommitRequest{term=2, version=4619, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T22:53:47,079][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4620, reason: ApplyCommitRequest{term=2, version=4620, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T22:55:22,436][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4689, reason: ApplyCommitRequest{term=2, version=4689, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T22:55:24,328][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4690, reason: ApplyCommitRequest{term=2, version=4690, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T23:51:59,980][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-15T23:51:59,988][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4757, reason: becoming candidate: onLeaderFailure
[2022-10-15T23:52:00,067][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4758, reason: ApplyCommitRequest{term=2, version=4758, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T23:53:41,274][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-15T23:53:41,282][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4760, reason: becoming candidate: onLeaderFailure
[2022-10-15T23:53:41,380][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4762, reason: ApplyCommitRequest{term=2, version=4762, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T23:53:47,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4763, reason: ApplyCommitRequest{term=2, version=4763, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T23:53:49,876][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4764, reason: ApplyCommitRequest{term=2, version=4764, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T23:55:24,867][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4834, reason: ApplyCommitRequest{term=2, version=4834, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-15T23:55:27,364][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4835, reason: ApplyCommitRequest{term=2, version=4835, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T00:52:00,007][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T00:52:00,019][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4904, reason: becoming candidate: onLeaderFailure
[2022-10-16T00:52:00,098][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4905, reason: ApplyCommitRequest{term=2, version=4905, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T00:53:44,318][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T00:53:44,327][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 4907, reason: becoming candidate: onLeaderFailure
[2022-10-16T00:53:44,400][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 4909, reason: ApplyCommitRequest{term=2, version=4909, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T00:53:50,193][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4910, reason: ApplyCommitRequest{term=2, version=4910, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T00:53:51,725][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 4911, reason: ApplyCommitRequest{term=2, version=4911, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T00:55:28,012][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4980, reason: ApplyCommitRequest{term=2, version=4980, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T00:55:29,729][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 4981, reason: ApplyCommitRequest{term=2, version=4981, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:52:00,041][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T01:52:00,048][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5049, reason: becoming candidate: onLeaderFailure
[2022-10-16T01:52:00,121][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5050, reason: ApplyCommitRequest{term=2, version=5050, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:53:47,379][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T01:53:47,389][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5052, reason: becoming candidate: onLeaderFailure
[2022-10-16T01:53:47,468][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5054, reason: ApplyCommitRequest{term=2, version=5054, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:53:51,798][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5055, reason: ApplyCommitRequest{term=2, version=5055, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:53:53,787][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5056, reason: ApplyCommitRequest{term=2, version=5056, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:55:30,648][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5125, reason: ApplyCommitRequest{term=2, version=5125, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:55:32,958][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5126, reason: ApplyCommitRequest{term=2, version=5126, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:55:33,256][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5127, reason: ApplyCommitRequest{term=2, version=5127, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T01:55:33,673][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5128, reason: ApplyCommitRequest{term=2, version=5128, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T02:52:00,060][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T02:52:00,067][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5195, reason: becoming candidate: onLeaderFailure
[2022-10-16T02:52:00,183][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5196, reason: ApplyCommitRequest{term=2, version=5196, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T02:53:50,368][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T02:53:50,374][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5198, reason: becoming candidate: onLeaderFailure
[2022-10-16T02:53:50,571][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5200, reason: ApplyCommitRequest{term=2, version=5200, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T02:53:54,505][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5201, reason: ApplyCommitRequest{term=2, version=5201, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T02:53:55,889][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5202, reason: ApplyCommitRequest{term=2, version=5202, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T02:55:34,210][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5272, reason: ApplyCommitRequest{term=2, version=5272, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T02:55:36,765][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5273, reason: ApplyCommitRequest{term=2, version=5273, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T03:52:00,076][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T03:52:00,086][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5341, reason: becoming candidate: onLeaderFailure
[2022-10-16T03:52:00,182][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5342, reason: ApplyCommitRequest{term=2, version=5342, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T03:53:53,403][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T03:53:53,410][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5344, reason: becoming candidate: onLeaderFailure
[2022-10-16T03:53:53,486][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5346, reason: ApplyCommitRequest{term=2, version=5346, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T03:53:56,201][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5347, reason: ApplyCommitRequest{term=2, version=5347, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T03:53:57,788][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5348, reason: ApplyCommitRequest{term=2, version=5348, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T03:55:37,694][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5418, reason: ApplyCommitRequest{term=2, version=5418, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T03:55:39,606][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5419, reason: ApplyCommitRequest{term=2, version=5419, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T04:52:00,099][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T04:52:00,110][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5486, reason: becoming candidate: onLeaderFailure
[2022-10-16T04:52:00,189][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5487, reason: ApplyCommitRequest{term=2, version=5487, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T04:53:56,433][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T04:53:56,446][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5489, reason: becoming candidate: onLeaderFailure
[2022-10-16T04:53:56,517][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5491, reason: ApplyCommitRequest{term=2, version=5491, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T04:53:58,237][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5492, reason: ApplyCommitRequest{term=2, version=5492, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T04:54:00,587][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5493, reason: ApplyCommitRequest{term=2, version=5493, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T04:55:39,985][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5562, reason: ApplyCommitRequest{term=2, version=5562, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T04:55:42,559][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5563, reason: ApplyCommitRequest{term=2, version=5563, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T05:52:00,121][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T05:52:00,134][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5631, reason: becoming candidate: onLeaderFailure
[2022-10-16T05:52:00,341][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5632, reason: ApplyCommitRequest{term=2, version=5632, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T05:53:59,473][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T05:53:59,482][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5634, reason: becoming candidate: onLeaderFailure
[2022-10-16T05:53:59,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5636, reason: ApplyCommitRequest{term=2, version=5636, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T05:54:00,647][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5637, reason: ApplyCommitRequest{term=2, version=5637, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T05:54:02,520][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5638, reason: ApplyCommitRequest{term=2, version=5638, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T05:55:43,073][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5706, reason: ApplyCommitRequest{term=2, version=5706, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T05:55:45,031][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5707, reason: ApplyCommitRequest{term=2, version=5707, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T06:52:00,146][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T06:52:00,151][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5777, reason: becoming candidate: onLeaderFailure
[2022-10-16T06:52:00,228][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5778, reason: ApplyCommitRequest{term=2, version=5778, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T06:54:02,469][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.93:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z}{hsjBrGCOSEWNqrJ6xZUj2g}{pmlpNj8xQ9yXNfJbD7un3Q}{10.233.69.201}{10.233.69.201:9300}{r}] has been removed from the cluster
	at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-10-16T06:54:02,477][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5780, reason: becoming candidate: onLeaderFailure
[2022-10-16T06:54:03,670][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, removed {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5783, reason: ApplyCommitRequest{term=2, version=5783, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T06:54:05,486][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-0}{7um74vIcSyewFiD9B7M30w}{zXhrBUK1RSq1Cqgv69Keng}{10.233.72.39}{10.233.72.39:9300}{dmr}}, term: 2, version: 5784, reason: ApplyCommitRequest{term=2, version=5784, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T06:55:45,406][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] removed {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5853, reason: ApplyCommitRequest{term=2, version=5853, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T06:55:47,976][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] added {{onap-sdnrdb-master-2}{No94EsJOTM-kk9Fq8fSO9g}{SdvrZ8yeRMS2hmVy4XKRmQ}{10.233.69.23}{10.233.69.23:9300}{dmr}}, term: 2, version: 5854, reason: ApplyCommitRequest{term=2, version=5854, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T07:52:00,167][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-1][10.233.65.93:9300][disconnected] disconnected
[2022-10-16T07:52:00,175][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}], current []}, term: 2, version: 5922, reason: becoming candidate: onLeaderFailure
[2022-10-16T07:52:00,296][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node changed {previous [], current [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}]}, term: 2, version: 5923, reason: ApplyCommitRequest{term=2, version=5923, sourceNode={onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}}
[2022-10-16T07:54:05,541][INFO ][o.e.c.c.Coordinator      ] [onap-sdnrdb-coordinating-only-7c4fc6d7fd-sbh8z] master node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-1}{mPUdSz6LTq-eI9TuIEvEqg}{oqdZdTJjTA-QwvfQxp9DCw}{10.233.65.93}{10.233.65.93:9300}{dmr}] failed [3] consecutive checks
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsea