By type
[38;5;6m [38;5;5m15:21:32.83 [0m
[38;5;6m [38;5;5m15:21:32.83 [0m[1mWelcome to the Bitnami elasticsearch container[0m
[38;5;6m [38;5;5m15:21:32.84 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m
[38;5;6m [38;5;5m15:21:32.84 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m
[38;5;6m [38;5;5m15:21:32.84 [0m
[38;5;6m [38;5;5m15:21:32.85 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup **
[38;5;6m [38;5;5m15:21:33.04 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch...
[38;5;6m [38;5;5m15:21:33.23 [0m[38;5;2mINFO [0m ==> Setting default configuration
[38;5;6m [38;5;5m15:21:33.34 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings...
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[38;5;6m [38;5;5m15:21:50.74 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! **
[38;5;6m [38;5;5m15:21:50.84 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-06-24T15:22:30,735][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS]
[2022-06-24T15:22:30,834][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] JVM home [/opt/bitnami/java]
[2022-06-24T15:22:30,835][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-8718526661653676840, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-06-24T15:22:48,641][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [aggs-matrix-stats]
[2022-06-24T15:22:48,642][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [analysis-common]
[2022-06-24T15:22:48,642][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [geo]
[2022-06-24T15:22:48,643][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-common]
[2022-06-24T15:22:48,643][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-geoip]
[2022-06-24T15:22:48,644][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-user-agent]
[2022-06-24T15:22:48,644][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [kibana]
[2022-06-24T15:22:48,645][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-expression]
[2022-06-24T15:22:48,645][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-mustache]
[2022-06-24T15:22:48,646][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-painless]
[2022-06-24T15:22:48,646][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [mapper-extras]
[2022-06-24T15:22:48,647][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [parent-join]
[2022-06-24T15:22:48,647][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [percolator]
[2022-06-24T15:22:48,647][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [rank-eval]
[2022-06-24T15:22:48,648][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [reindex]
[2022-06-24T15:22:48,648][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [repository-url]
[2022-06-24T15:22:48,648][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [tasks]
[2022-06-24T15:22:48,649][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [transport-netty4]
[2022-06-24T15:22:48,650][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded plugin [repository-s3]
[2022-06-24T15:22:49,535][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-1] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.13.30:/dockerdata-nfs/onap/elastic-master-0)]], net usable_space [98gb], net total_space [99.9gb], types [nfs4]
[2022-06-24T15:22:49,537][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-1] heap size [123.7mb], compressed ordinary object pointers [true]
[2022-06-24T15:22:50,153][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] node name [onap-sdnrdb-master-1], node ID [GGccGa1dSbO4lH8NH0whtw], cluster name [sdnrdb-cluster]
[2022-06-24T15:23:42,140][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-master-1] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}]
[2022-06-24T15:23:43,042][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-master-1] using discovery type [zen] and seed hosts providers [settings]
[2022-06-24T15:23:46,842][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-06-24T15:23:49,040][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] initialized
[2022-06-24T15:23:49,041][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] starting ...
[2022-06-24T15:23:50,838][INFO ][o.e.t.TransportService ] [onap-sdnrdb-master-1] publish_address {10.233.65.72:9300}, bound_addresses {0.0.0.0:9300}
[2022-06-24T15:23:52,236][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.65.72:9300, remoteAddress=/127.0.0.6:36973}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-06-24T15:23:53,236][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.65.72:9300, remoteAddress=/127.0.0.6:50593}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-06-24T15:23:53,449][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-06-24T15:24:03,649][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}]; discovery will continue using [10.233.65.153:9300, 10.233.67.247:9300, 10.233.66.124:9300] from hosts providers and [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-24T15:24:13,661][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}]; discovery will continue using [10.233.65.153:9300, 10.233.67.247:9300, 10.233.66.124:9300] from hosts providers and [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-24T15:24:23,736][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}]; discovery will continue using [10.233.65.153:9300, 10.233.67.247:9300, 10.233.66.124:9300] from hosts providers and [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-24T15:24:33,739][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}]; discovery will continue using [10.233.65.153:9300, 10.233.67.247:9300, 10.233.66.124:9300] from hosts providers and [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-06-24T15:24:36,217][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-onap-sdnrdb-master-2,PuWbKEJXS7i-1HsSfeiHCg,GGccGa1dSbO4lH8NH0whtw}
[2022-06-24T15:24:39,736][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr} elect leader, {onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T15:24:39,838][WARN ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] failing [elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr} elect leader, {onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1]
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 1 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-06-24T15:24:39,936][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}}]}
org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.65.72:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 1 while handling publication
at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2022-06-24T15:24:43,520][INFO ][o.e.c.c.CoordinationState] [onap-sdnrdb-master-1] cluster UUID set to [K9p3OB3ASoaOCt4XfR1YYA]
[2022-06-24T15:24:43,946][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}, term: 2, version: 1, reason: ApplyCommitRequest{term=2, version=1, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T15:24:44,235][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-1] publish_address {10.233.65.72:9200}, bound_addresses {0.0.0.0:9200}
[2022-06-24T15:24:44,236][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] started
[2022-06-24T15:24:45,912][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2, reason: ApplyCommitRequest{term=2, version=2, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T15:24:52,935][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-master-1] updating [action.auto_create_index] from [true] to [false]
[2022-06-24T15:25:42,744][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 38, reason: ApplyCommitRequest{term=2, version=38, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:24:35,062][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T16:24:35,069][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 130, reason: becoming candidate: onLeaderFailure
[2022-06-24T16:24:38,773][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T16:24:45,548][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 136, reason: ApplyCommitRequest{term=2, version=136, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:24:47,839][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 138, reason: ApplyCommitRequest{term=2, version=138, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:25:39,952][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 205, reason: ApplyCommitRequest{term=2, version=205, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:25:40,143][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T16:25:40,234][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T16:25:40,240][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T16:25:40,434][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T16:25:40,534][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T16:25:40,639][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T16:25:40,836][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,034][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,042][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,236][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,337][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,344][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,441][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,536][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,545][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,636][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,656][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,835][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,843][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T16:25:41,943][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T16:25:42,039][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T16:25:42,051][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T16:25:44,167][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 206, reason: ApplyCommitRequest{term=2, version=206, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:25:44,635][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 207, reason: ApplyCommitRequest{term=2, version=207, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:25:44,968][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 208, reason: ApplyCommitRequest{term=2, version=208, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:25:45,351][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 209, reason: ApplyCommitRequest{term=2, version=209, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:25:45,959][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 210, reason: ApplyCommitRequest{term=2, version=210, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T16:26:12,236][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][3740] overhead, spent [485ms] collecting in the last [1s]
[2022-06-24T17:24:34,812][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T17:24:34,834][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 279, reason: becoming candidate: onLeaderFailure
[2022-06-24T17:24:35,260][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 280, reason: ApplyCommitRequest{term=2, version=280, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T17:24:47,410][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T17:24:47,423][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 281, reason: becoming candidate: onLeaderFailure
[2022-06-24T17:24:56,099][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 298, reason: ApplyCommitRequest{term=2, version=298, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T17:25:46,571][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 366, reason: ApplyCommitRequest{term=2, version=366, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T17:25:46,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T17:25:46,735][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T17:25:46,748][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T17:25:46,836][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T17:25:46,846][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T17:25:46,945][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T17:25:46,952][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,044][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,240][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,335][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,437][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,442][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,543][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,553][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,642][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,660][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,736][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,745][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,835][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,839][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T17:25:47,848][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T17:25:48,719][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 367, reason: ApplyCommitRequest{term=2, version=367, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:24:34,693][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T18:24:34,700][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 436, reason: becoming candidate: onLeaderFailure
[2022-06-24T18:24:35,357][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 437, reason: ApplyCommitRequest{term=2, version=437, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:24:51,395][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 439, reason: ApplyCommitRequest{term=2, version=439, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:24:54,188][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 440, reason: ApplyCommitRequest{term=2, version=440, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:24:58,160][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T18:24:58,174][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 440, reason: becoming candidate: onLeaderFailure
[2022-06-24T18:25:00,593][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 444, reason: ApplyCommitRequest{term=2, version=444, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:25:49,111][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 511, reason: ApplyCommitRequest{term=2, version=511, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:25:49,436][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T18:25:49,538][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T18:25:49,639][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T18:25:49,835][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T18:25:49,837][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T18:25:49,939][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T18:25:50,043][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T18:25:50,234][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T18:25:50,435][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T18:25:50,635][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T18:25:50,736][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T18:25:50,838][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,035][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,151][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,341][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,438][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,534][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,646][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,836][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,843][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T18:25:51,934][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T18:25:52,034][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 512, reason: ApplyCommitRequest{term=2, version=512, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:25:52,459][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 513, reason: ApplyCommitRequest{term=2, version=513, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:25:52,970][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 514, reason: ApplyCommitRequest{term=2, version=514, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T18:25:55,345][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][10919] overhead, spent [485ms] collecting in the last [1.4s]
[2022-06-24T19:24:34,504][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T19:24:34,537][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 582, reason: becoming candidate: onLeaderFailure
[2022-06-24T19:24:34,890][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 583, reason: ApplyCommitRequest{term=2, version=583, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T19:24:53,804][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 585, reason: ApplyCommitRequest{term=2, version=585, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T19:24:56,999][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 586, reason: ApplyCommitRequest{term=2, version=586, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T19:25:02,012][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T19:25:02,021][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 586, reason: becoming candidate: onLeaderFailure
[2022-06-24T19:25:07,375][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 592, reason: ApplyCommitRequest{term=2, version=592, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T19:25:53,434][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 661, reason: ApplyCommitRequest{term=2, version=661, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T19:25:53,539][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T19:25:53,636][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T19:25:53,740][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T19:25:53,743][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T19:25:53,937][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T19:25:53,953][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,037][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,136][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,146][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,249][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,340][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,354][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,369][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,455][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,458][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,459][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,545][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,555][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,646][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,834][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T19:25:54,851][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T19:25:55,779][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 662, reason: ApplyCommitRequest{term=2, version=662, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T20:24:34,200][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T20:24:34,234][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 730, reason: becoming candidate: onLeaderFailure
[2022-06-24T20:24:34,834][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 731, reason: ApplyCommitRequest{term=2, version=731, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T20:24:56,392][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 733, reason: ApplyCommitRequest{term=2, version=733, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T20:24:59,577][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 734, reason: ApplyCommitRequest{term=2, version=734, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T20:25:08,860][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T20:25:08,872][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 734, reason: becoming candidate: onLeaderFailure
[2022-06-24T20:25:13,243][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 738, reason: ApplyCommitRequest{term=2, version=738, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T20:25:56,156][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 806, reason: ApplyCommitRequest{term=2, version=806, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T20:25:56,334][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,434][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,440][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,447][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,535][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,554][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,636][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,643][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,744][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,834][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,841][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,854][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,942][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,948][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T20:25:56,962][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,042][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,055][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,134][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,143][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,152][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,236][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,251][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T20:25:57,789][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 807, reason: ApplyCommitRequest{term=2, version=807, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T21:24:34,018][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T21:24:34,041][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 875, reason: becoming candidate: onLeaderFailure
[2022-06-24T21:24:34,854][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 876, reason: ApplyCommitRequest{term=2, version=876, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T21:24:59,090][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 878, reason: ApplyCommitRequest{term=2, version=878, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T21:25:02,220][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 879, reason: ApplyCommitRequest{term=2, version=879, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T21:25:14,992][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T21:25:15,005][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 879, reason: becoming candidate: onLeaderFailure
[2022-06-24T21:25:18,384][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 883, reason: ApplyCommitRequest{term=2, version=883, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T21:25:58,268][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 952, reason: ApplyCommitRequest{term=2, version=952, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T21:25:58,350][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,441][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,536][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,551][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,635][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,638][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,654][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,744][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,755][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,840][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,850][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,949][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T21:25:58,955][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,035][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,041][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,143][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,241][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,247][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,346][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,444][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,535][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T21:25:59,544][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T21:26:00,297][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 953, reason: ApplyCommitRequest{term=2, version=953, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T22:24:34,092][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T22:24:34,136][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1021, reason: becoming candidate: onLeaderFailure
[2022-06-24T22:24:34,440][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1022, reason: ApplyCommitRequest{term=2, version=1022, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T22:25:02,035][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1024, reason: ApplyCommitRequest{term=2, version=1024, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T22:25:04,871][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1025, reason: ApplyCommitRequest{term=2, version=1025, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T22:25:20,613][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T22:25:20,626][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1025, reason: becoming candidate: onLeaderFailure
[2022-06-24T22:25:22,261][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1029, reason: ApplyCommitRequest{term=2, version=1029, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T22:26:00,941][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][25317] overhead, spent [291ms] collecting in the last [1s]
[2022-06-24T22:26:00,946][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1097, reason: ApplyCommitRequest{term=2, version=1097, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T22:26:01,334][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,434][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,450][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,537][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,546][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,640][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,735][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,747][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,850][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,942][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T22:26:01,948][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,034][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,056][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,248][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,336][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,344][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,445][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,456][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,538][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,548][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,634][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,648][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T22:26:02,756][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1098, reason: ApplyCommitRequest{term=2, version=1098, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T23:24:33,894][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-24T23:24:33,934][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1168, reason: becoming candidate: onLeaderFailure
[2022-06-24T23:24:34,642][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1169, reason: ApplyCommitRequest{term=2, version=1169, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T23:25:04,509][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1171, reason: ApplyCommitRequest{term=2, version=1171, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T23:25:07,680][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1172, reason: ApplyCommitRequest{term=2, version=1172, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T23:25:23,786][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-24T23:25:23,799][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1172, reason: becoming candidate: onLeaderFailure
[2022-06-24T23:25:26,096][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1176, reason: ApplyCommitRequest{term=2, version=1176, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T23:26:02,812][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1244, reason: ApplyCommitRequest{term=2, version=1244, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-24T23:26:02,855][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T23:26:02,862][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,036][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,050][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,144][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,337][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,351][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,441][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,443][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,544][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,553][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,634][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,644][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,653][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,736][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,755][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,844][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,938][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T23:26:03,946][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-24T23:26:04,035][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-24T23:26:05,548][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1245, reason: ApplyCommitRequest{term=2, version=1245, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T00:24:33,693][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T00:24:33,934][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1314, reason: becoming candidate: onLeaderFailure
[2022-06-25T00:24:34,345][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1315, reason: ApplyCommitRequest{term=2, version=1315, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T00:25:07,832][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1317, reason: ApplyCommitRequest{term=2, version=1317, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T00:25:10,411][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1318, reason: ApplyCommitRequest{term=2, version=1318, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T00:25:27,598][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T00:25:27,610][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1318, reason: becoming candidate: onLeaderFailure
[2022-06-25T00:25:28,944][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1322, reason: ApplyCommitRequest{term=2, version=1322, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T00:26:05,722][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1393, reason: ApplyCommitRequest{term=2, version=1393, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T00:26:05,765][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T00:26:05,839][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T00:26:05,934][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T00:26:05,935][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T00:26:05,943][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,040][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,136][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,144][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,234][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,244][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,260][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,338][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,343][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,360][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,439][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,449][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,543][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,640][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,751][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,834][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T00:26:06,846][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T00:26:08,270][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1394, reason: ApplyCommitRequest{term=2, version=1394, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T01:24:33,752][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T01:24:33,835][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1464, reason: becoming candidate: onLeaderFailure
[2022-06-25T01:24:34,244][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1465, reason: ApplyCommitRequest{term=2, version=1465, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T01:25:10,279][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1467, reason: ApplyCommitRequest{term=2, version=1467, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T01:25:13,107][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1468, reason: ApplyCommitRequest{term=2, version=1468, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T01:25:30,475][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T01:25:30,486][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1468, reason: becoming candidate: onLeaderFailure
[2022-06-25T01:25:32,788][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1472, reason: ApplyCommitRequest{term=2, version=1472, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T01:26:08,713][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1540, reason: ApplyCommitRequest{term=2, version=1540, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T01:26:08,742][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T01:26:08,751][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T01:26:08,763][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T01:26:08,844][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T01:26:08,936][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T01:26:08,947][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,047][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,049][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,140][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,155][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,240][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,249][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,254][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,355][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,539][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,634][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,643][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,656][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,738][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T01:26:09,845][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T01:26:10,971][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1541, reason: ApplyCommitRequest{term=2, version=1541, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T02:24:33,623][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T02:24:33,630][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1612, reason: becoming candidate: onLeaderFailure
[2022-06-25T02:24:34,056][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1613, reason: ApplyCommitRequest{term=2, version=1613, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T02:25:12,735][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1614, reason: ApplyCommitRequest{term=2, version=1614, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T02:25:14,835][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1615, reason: ApplyCommitRequest{term=2, version=1615, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T02:25:35,338][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T02:25:35,353][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1615, reason: becoming candidate: onLeaderFailure
[2022-06-25T02:25:36,025][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1618, reason: ApplyCommitRequest{term=2, version=1618, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T02:26:11,057][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1686, reason: ApplyCommitRequest{term=2, version=1686, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T02:26:11,140][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,150][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,237][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,247][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,337][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,348][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,534][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,546][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,635][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,643][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,654][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,743][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,748][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,840][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,845][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,951][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T02:26:11,959][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T02:26:12,041][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T02:26:12,142][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T02:26:12,144][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T02:26:12,242][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T02:26:12,334][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T02:26:13,215][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1687, reason: ApplyCommitRequest{term=2, version=1687, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T03:24:33,331][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T03:24:33,341][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1756, reason: becoming candidate: onLeaderFailure
[2022-06-25T03:24:33,845][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1757, reason: ApplyCommitRequest{term=2, version=1757, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T03:25:14,468][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1758, reason: ApplyCommitRequest{term=2, version=1758, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T03:25:16,543][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1759, reason: ApplyCommitRequest{term=2, version=1759, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T03:25:38,068][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T03:25:38,079][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1760, reason: becoming candidate: onLeaderFailure
[2022-06-25T03:25:38,641][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1762, reason: ApplyCommitRequest{term=2, version=1762, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T03:26:13,398][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1833, reason: ApplyCommitRequest{term=2, version=1833, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T03:26:13,434][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,450][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,538][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,542][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,635][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,649][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,735][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,748][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,848][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,940][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T03:26:13,951][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,042][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,135][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,350][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,419][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,431][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,511][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,539][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,547][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,551][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T03:26:14,560][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T03:26:15,572][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1834, reason: ApplyCommitRequest{term=2, version=1834, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T04:24:33,087][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T04:24:33,094][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1903, reason: becoming candidate: onLeaderFailure
[2022-06-25T04:24:33,549][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1905, reason: ApplyCommitRequest{term=2, version=1905, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T04:25:16,205][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1906, reason: ApplyCommitRequest{term=2, version=1906, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T04:25:18,621][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 1907, reason: ApplyCommitRequest{term=2, version=1907, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T04:25:40,770][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T04:25:40,781][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 1908, reason: becoming candidate: onLeaderFailure
[2022-06-25T04:25:41,113][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 1910, reason: ApplyCommitRequest{term=2, version=1910, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T04:26:09,142][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][46914] overhead, spent [275ms] collecting in the last [1s]
[2022-06-25T04:26:15,941][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1977, reason: ApplyCommitRequest{term=2, version=1977, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T04:26:16,036][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,050][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,134][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,145][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,233][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,243][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,252][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,340][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,349][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,440][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,538][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,552][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,642][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,650][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,735][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,749][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,752][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,845][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,852][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,943][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,954][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T04:26:16,964][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T04:26:17,934][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 1978, reason: ApplyCommitRequest{term=2, version=1978, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T05:24:32,858][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T05:24:32,866][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2047, reason: becoming candidate: onLeaderFailure
[2022-06-25T05:24:33,096][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2048, reason: ApplyCommitRequest{term=2, version=2048, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T05:25:17,955][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2049, reason: ApplyCommitRequest{term=2, version=2049, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T05:25:20,941][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2050, reason: ApplyCommitRequest{term=2, version=2050, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T05:25:43,303][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T05:25:43,313][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2051, reason: becoming candidate: onLeaderFailure
[2022-06-25T05:25:46,684][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2055, reason: ApplyCommitRequest{term=2, version=2055, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T05:26:17,992][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2119, reason: ApplyCommitRequest{term=2, version=2119, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T05:26:18,063][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,134][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,247][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,334][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,341][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,350][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,434][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,445][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,538][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,638][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,741][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,839][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T05:26:18,944][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,035][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,139][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,243][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,335][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,448][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,534][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,637][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,841][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T05:26:19,935][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T05:26:20,552][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2121, reason: ApplyCommitRequest{term=2, version=2121, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:24:32,610][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T06:24:32,616][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2190, reason: becoming candidate: onLeaderFailure
[2022-06-25T06:24:32,963][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2191, reason: ApplyCommitRequest{term=2, version=2191, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:25:20,808][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2192, reason: ApplyCommitRequest{term=2, version=2192, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:25:22,860][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2193, reason: ApplyCommitRequest{term=2, version=2193, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:25:48,211][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T06:25:48,221][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2194, reason: becoming candidate: onLeaderFailure
[2022-06-25T06:25:53,492][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2198, reason: ApplyCommitRequest{term=2, version=2198, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:26:21,341][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2229, reason: ApplyCommitRequest{term=2, version=2229, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:26:21,448][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,546][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,557][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,735][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,744][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,754][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,850][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,939][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,948][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T06:26:21,957][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T06:26:22,045][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T06:26:22,053][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T06:26:23,425][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2232, reason: ApplyCommitRequest{term=2, version=2232, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T06:26:41,734][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][54143] overhead, spent [296ms] collecting in the last [1s]
[2022-06-25T07:24:32,301][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T07:24:32,308][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2304, reason: becoming candidate: onLeaderFailure
[2022-06-25T07:24:32,679][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2305, reason: ApplyCommitRequest{term=2, version=2305, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T07:25:22,485][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2306, reason: ApplyCommitRequest{term=2, version=2306, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T07:25:25,392][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2307, reason: ApplyCommitRequest{term=2, version=2307, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T07:25:55,951][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T07:25:55,968][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2308, reason: becoming candidate: onLeaderFailure
[2022-06-25T07:25:57,369][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2312, reason: ApplyCommitRequest{term=2, version=2312, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T07:26:24,037][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2365, reason: ApplyCommitRequest{term=2, version=2365, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T07:26:24,234][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,335][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,345][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,355][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,435][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,453][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,454][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,539][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,638][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,735][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,738][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,836][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,837][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T07:26:24,945][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T07:26:25,034][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T07:26:25,043][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T07:26:25,138][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T07:26:26,475][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2369, reason: ApplyCommitRequest{term=2, version=2369, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T08:24:32,003][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T08:24:32,011][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2442, reason: becoming candidate: onLeaderFailure
[2022-06-25T08:24:32,355][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2443, reason: ApplyCommitRequest{term=2, version=2443, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T08:25:25,017][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2444, reason: ApplyCommitRequest{term=2, version=2444, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T08:25:28,105][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2445, reason: ApplyCommitRequest{term=2, version=2445, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T08:25:59,610][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T08:25:59,622][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2446, reason: becoming candidate: onLeaderFailure
[2022-06-25T08:26:01,042][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2450, reason: ApplyCommitRequest{term=2, version=2450, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T08:26:25,967][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2506, reason: ApplyCommitRequest{term=2, version=2506, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T08:26:26,050][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,134][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,142][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,247][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,338][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,434][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,446][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,543][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,640][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,659][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,749][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,841][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,862][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,936][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T08:26:26,946][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T08:26:27,035][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T08:26:29,446][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2510, reason: ApplyCommitRequest{term=2, version=2510, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T09:24:31,724][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T09:24:31,751][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2581, reason: becoming candidate: onLeaderFailure
[2022-06-25T09:24:32,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2583, reason: ApplyCommitRequest{term=2, version=2583, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T09:25:27,695][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2584, reason: ApplyCommitRequest{term=2, version=2584, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T09:25:29,946][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2585, reason: ApplyCommitRequest{term=2, version=2585, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T09:26:02,640][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T09:26:02,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2586, reason: becoming candidate: onLeaderFailure
[2022-06-25T09:26:03,887][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2590, reason: ApplyCommitRequest{term=2, version=2590, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T09:26:30,031][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2655, reason: ApplyCommitRequest{term=2, version=2655, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T09:26:30,144][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,146][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,237][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,248][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,334][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,445][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,541][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,563][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,637][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,660][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,741][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,746][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,763][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,855][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,934][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,944][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,954][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T09:26:30,964][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T09:26:31,135][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T09:26:31,146][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T09:26:32,660][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2658, reason: ApplyCommitRequest{term=2, version=2658, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:24:31,493][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T10:24:31,501][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2727, reason: becoming candidate: onLeaderFailure
[2022-06-25T10:24:31,861][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2728, reason: ApplyCommitRequest{term=2, version=2728, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:25:29,574][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2729, reason: ApplyCommitRequest{term=2, version=2729, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:25:32,439][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2731, reason: ApplyCommitRequest{term=2, version=2731, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:06,204][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T10:26:06,218][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2731, reason: becoming candidate: onLeaderFailure
[2022-06-25T10:26:08,498][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2735, reason: ApplyCommitRequest{term=2, version=2735, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:32,242][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2782, reason: ApplyCommitRequest{term=2, version=2782, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:32,445][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T10:26:32,641][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T10:26:32,841][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,036][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,043][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,241][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,435][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,537][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,641][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T10:26:33,837][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,137][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,442][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,542][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,648][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,834][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,842][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T10:26:34,974][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2783, reason: ApplyCommitRequest{term=2, version=2783, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:36,072][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2784, reason: ApplyCommitRequest{term=2, version=2784, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:37,946][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2786, reason: ApplyCommitRequest{term=2, version=2786, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:38,166][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2787, reason: ApplyCommitRequest{term=2, version=2787, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T10:26:39,251][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2790, reason: ApplyCommitRequest{term=2, version=2790, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T11:24:31,225][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T11:24:31,230][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2865, reason: becoming candidate: onLeaderFailure
[2022-06-25T11:24:31,564][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2866, reason: ApplyCommitRequest{term=2, version=2866, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T11:25:32,214][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2868, reason: ApplyCommitRequest{term=2, version=2868, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T11:25:35,188][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2869, reason: ApplyCommitRequest{term=2, version=2869, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T11:26:10,865][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T11:26:10,879][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2869, reason: becoming candidate: onLeaderFailure
[2022-06-25T11:26:11,456][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2871, reason: ApplyCommitRequest{term=2, version=2871, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T11:26:39,416][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2915, reason: ApplyCommitRequest{term=2, version=2915, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T11:26:39,448][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,452][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,537][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,547][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,555][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,643][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,738][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,748][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,758][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,834][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,846][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,854][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,863][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,872][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,947][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T11:26:39,954][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T11:26:42,139][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 2922, reason: ApplyCommitRequest{term=2, version=2922, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T12:24:30,931][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T12:24:30,938][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2993, reason: becoming candidate: onLeaderFailure
[2022-06-25T12:24:31,354][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 2994, reason: ApplyCommitRequest{term=2, version=2994, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T12:25:34,863][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2996, reason: ApplyCommitRequest{term=2, version=2996, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T12:25:36,998][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 2997, reason: ApplyCommitRequest{term=2, version=2997, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T12:26:12,688][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T12:26:12,698][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 2997, reason: becoming candidate: onLeaderFailure
[2022-06-25T12:26:14,989][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3001, reason: ApplyCommitRequest{term=2, version=3001, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T12:26:42,689][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3070, reason: ApplyCommitRequest{term=2, version=3070, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T12:26:42,719][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T12:26:42,742][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T12:26:42,751][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T12:26:42,835][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T12:26:42,854][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T12:26:42,945][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T12:26:42,953][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,054][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,135][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,145][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,153][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,243][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,245][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,262][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,335][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,349][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,356][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,435][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,454][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,463][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,537][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T12:26:43,778][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3071, reason: ApplyCommitRequest{term=2, version=3071, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T13:24:30,630][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T13:24:30,640][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3141, reason: becoming candidate: onLeaderFailure
[2022-06-25T13:24:31,128][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3142, reason: ApplyCommitRequest{term=2, version=3142, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T13:25:36,546][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3144, reason: ApplyCommitRequest{term=2, version=3144, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T13:25:39,619][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3145, reason: ApplyCommitRequest{term=2, version=3145, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T13:26:16,427][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T13:26:16,437][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3145, reason: becoming candidate: onLeaderFailure
[2022-06-25T13:26:18,736][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3149, reason: ApplyCommitRequest{term=2, version=3149, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T13:26:44,132][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3217, reason: ApplyCommitRequest{term=2, version=3217, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T13:26:44,237][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,243][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,251][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,344][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,352][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,447][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,457][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,541][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,552][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,641][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,653][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,662][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,740][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,757][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,763][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,836][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,845][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,857][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,864][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,934][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,945][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T13:26:44,955][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T13:26:45,913][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3218, reason: ApplyCommitRequest{term=2, version=3218, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T14:24:30,389][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T14:24:30,394][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3286, reason: becoming candidate: onLeaderFailure
[2022-06-25T14:24:30,506][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3287, reason: ApplyCommitRequest{term=2, version=3287, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T14:25:39,266][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3289, reason: ApplyCommitRequest{term=2, version=3289, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T14:25:41,359][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3290, reason: ApplyCommitRequest{term=2, version=3290, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T14:26:20,881][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T14:26:20,934][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3290, reason: becoming candidate: onLeaderFailure
[2022-06-25T14:26:25,240][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3294, reason: ApplyCommitRequest{term=2, version=3294, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T14:26:46,451][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3338, reason: ApplyCommitRequest{term=2, version=3338, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T14:26:46,739][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T14:26:46,934][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T14:26:46,940][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,147][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,243][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,256][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,337][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,351][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,356][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,440][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,442][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,455][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,534][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,544][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T14:26:47,559][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T14:26:49,874][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3341, reason: ApplyCommitRequest{term=2, version=3341, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T15:24:30,079][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T15:24:30,136][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3413, reason: becoming candidate: onLeaderFailure
[2022-06-25T15:24:30,634][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3414, reason: ApplyCommitRequest{term=2, version=3414, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T15:25:41,017][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3416, reason: ApplyCommitRequest{term=2, version=3416, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T15:25:44,062][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3417, reason: ApplyCommitRequest{term=2, version=3417, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T15:26:27,011][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T15:26:27,034][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3417, reason: becoming candidate: onLeaderFailure
[2022-06-25T15:26:29,371][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3423, reason: ApplyCommitRequest{term=2, version=3423, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T15:26:49,094][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3467, reason: ApplyCommitRequest{term=2, version=3467, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T15:26:49,130][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,143][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,236][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,242][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,340][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,347][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,438][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,446][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,457][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,546][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,553][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,636][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,750][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,756][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,840][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T15:26:49,849][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T15:26:52,048][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3472, reason: ApplyCommitRequest{term=2, version=3472, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T16:24:29,928][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T16:24:29,939][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3544, reason: becoming candidate: onLeaderFailure
[2022-06-25T16:24:30,372][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3545, reason: ApplyCommitRequest{term=2, version=3545, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T16:25:43,631][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3547, reason: ApplyCommitRequest{term=2, version=3547, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T16:25:45,766][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3548, reason: ApplyCommitRequest{term=2, version=3548, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T16:26:30,860][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T16:26:30,870][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3548, reason: becoming candidate: onLeaderFailure
[2022-06-25T16:26:34,201][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3552, reason: ApplyCommitRequest{term=2, version=3552, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T16:26:51,439][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3595, reason: ApplyCommitRequest{term=2, version=3595, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T16:26:51,546][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,556][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,640][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,648][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,741][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,936][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,953][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T16:26:51,958][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,042][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,146][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,157][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,171][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,246][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,255][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T16:26:52,264][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T16:26:54,237][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3600, reason: ApplyCommitRequest{term=2, version=3600, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T17:24:29,697][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T17:24:29,703][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3672, reason: becoming candidate: onLeaderFailure
[2022-06-25T17:24:29,817][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3673, reason: ApplyCommitRequest{term=2, version=3673, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T17:25:45,403][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3675, reason: ApplyCommitRequest{term=2, version=3675, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T17:25:48,432][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3676, reason: ApplyCommitRequest{term=2, version=3676, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T17:26:36,205][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T17:26:36,215][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3676, reason: becoming candidate: onLeaderFailure
[2022-06-25T17:26:38,496][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3682, reason: ApplyCommitRequest{term=2, version=3682, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T17:26:54,972][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3726, reason: ApplyCommitRequest{term=2, version=3726, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T17:26:55,041][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,049][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,141][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,241][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,254][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,336][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,435][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,441][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,538][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,657][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,662][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,738][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,756][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,839][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,849][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,859][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T17:26:55,872][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T17:26:57,200][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3730, reason: ApplyCommitRequest{term=2, version=3730, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:24:29,395][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T18:24:29,430][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3802, reason: becoming candidate: onLeaderFailure
[2022-06-25T18:24:29,972][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3803, reason: ApplyCommitRequest{term=2, version=3803, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:25:48,050][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3805, reason: ApplyCommitRequest{term=2, version=3805, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:25:50,196][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3806, reason: ApplyCommitRequest{term=2, version=3806, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:26:40,437][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T18:26:40,538][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3806, reason: becoming candidate: onLeaderFailure
[2022-06-25T18:26:41,834][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3810, reason: ApplyCommitRequest{term=2, version=3810, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:26:57,058][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3856, reason: ApplyCommitRequest{term=2, version=3856, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:26:57,134][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,240][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,251][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,337][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,348][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,353][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,440][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,450][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,551][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,634][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,644][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,654][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,662][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,741][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,754][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T18:26:57,762][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T18:27:00,055][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3861, reason: ApplyCommitRequest{term=2, version=3861, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T18:27:03,445][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][97343] overhead, spent [280ms] collecting in the last [1.1s]
[2022-06-25T19:24:29,200][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T19:24:29,207][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3934, reason: becoming candidate: onLeaderFailure
[2022-06-25T19:24:29,371][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3935, reason: ApplyCommitRequest{term=2, version=3935, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T19:25:49,942][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3937, reason: ApplyCommitRequest{term=2, version=3937, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T19:25:52,828][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 3938, reason: ApplyCommitRequest{term=2, version=3938, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T19:26:43,761][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T19:26:43,771][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 3938, reason: becoming candidate: onLeaderFailure
[2022-06-25T19:26:44,259][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 3940, reason: ApplyCommitRequest{term=2, version=3940, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T19:26:59,679][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3989, reason: ApplyCommitRequest{term=2, version=3989, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T19:26:59,734][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T19:26:59,744][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T19:26:59,835][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T19:26:59,844][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,151][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,236][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,245][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,254][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,262][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,335][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,345][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,358][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,439][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,445][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,455][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,478][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T19:27:00,541][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T19:27:01,834][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 3993, reason: ApplyCommitRequest{term=2, version=3993, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T20:24:28,909][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T20:24:28,919][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4064, reason: becoming candidate: onLeaderFailure
[2022-06-25T20:24:29,457][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4065, reason: ApplyCommitRequest{term=2, version=4065, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T20:25:52,566][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4067, reason: ApplyCommitRequest{term=2, version=4067, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T20:25:55,537][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4068, reason: ApplyCommitRequest{term=2, version=4068, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T20:26:45,843][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T20:26:45,854][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4068, reason: becoming candidate: onLeaderFailure
[2022-06-25T20:26:46,312][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4070, reason: ApplyCommitRequest{term=2, version=4070, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T20:27:01,884][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4110, reason: ApplyCommitRequest{term=2, version=4110, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T20:27:01,942][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T20:27:01,957][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,046][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,142][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,234][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,243][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,353][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,363][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,372][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,442][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,449][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,547][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,563][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,642][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T20:27:02,647][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T20:27:04,739][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4116, reason: ApplyCommitRequest{term=2, version=4116, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T21:24:28,600][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T21:24:28,604][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4191, reason: becoming candidate: onLeaderFailure
[2022-06-25T21:24:28,763][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4192, reason: ApplyCommitRequest{term=2, version=4192, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T21:25:55,163][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4194, reason: ApplyCommitRequest{term=2, version=4194, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T21:25:58,296][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4195, reason: ApplyCommitRequest{term=2, version=4195, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T21:26:48,164][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T21:26:48,178][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4195, reason: becoming candidate: onLeaderFailure
[2022-06-25T21:26:50,441][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4199, reason: ApplyCommitRequest{term=2, version=4199, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T21:27:04,690][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4238, reason: ApplyCommitRequest{term=2, version=4238, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T21:27:04,739][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T21:27:04,748][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T21:27:04,836][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T21:27:04,848][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T21:27:04,938][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T21:27:04,943][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,041][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,049][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,149][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,155][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,261][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,336][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,352][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,440][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T21:27:05,449][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T21:27:08,283][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4241, reason: ApplyCommitRequest{term=2, version=4241, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T22:24:28,319][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T22:24:28,331][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4315, reason: becoming candidate: onLeaderFailure
[2022-06-25T22:24:28,658][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4316, reason: ApplyCommitRequest{term=2, version=4316, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T22:25:58,231][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4318, reason: ApplyCommitRequest{term=2, version=4318, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T22:26:00,991][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4319, reason: ApplyCommitRequest{term=2, version=4319, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T22:26:52,253][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T22:26:52,261][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4319, reason: becoming candidate: onLeaderFailure
[2022-06-25T22:26:53,541][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4323, reason: ApplyCommitRequest{term=2, version=4323, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T22:27:07,089][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4354, reason: ApplyCommitRequest{term=2, version=4354, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T22:27:07,336][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,340][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,352][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,443][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,534][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,648][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,701][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,703][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,744][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,754][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T22:27:07,843][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T22:27:09,151][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4358, reason: ApplyCommitRequest{term=2, version=4358, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T23:24:28,036][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.67.247:9300][disconnected] disconnected
[2022-06-25T23:24:28,042][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4431, reason: becoming candidate: onLeaderFailure
[2022-06-25T23:24:28,446][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4432, reason: ApplyCommitRequest{term=2, version=4432, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T23:26:00,611][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4434, reason: ApplyCommitRequest{term=2, version=4434, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T23:26:03,673][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-99f8cd997-nftr4}{igbnV7qcScCRkekKvjF7_w}{bN3GVq68QFi7DV6SUQRIVA}{10.233.65.153}{10.233.65.153:9300}{r}}, term: 2, version: 4435, reason: ApplyCommitRequest{term=2, version=4435, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T23:26:55,934][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] master node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed, restarting discovery
org.elasticsearch.ElasticsearchException: node [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}] failed [3] consecutive checks
at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.67.247:9300][internal:coordination/fault_detection/leader_check]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{onap-sdnrdb-master-1}{GGccGa1dSbO4lH8NH0whtw}{CxdHGEiFTduhVpCryZtAoA}{10.233.65.72}{10.233.65.72:9300}{dmr}] has been removed from the cluster
at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:192) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
[2022-06-25T23:26:56,034][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}], current []}, term: 2, version: 4435, reason: becoming candidate: onLeaderFailure
[2022-06-25T23:26:56,942][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}]}, term: 2, version: 4437, reason: ApplyCommitRequest{term=2, version=4437, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T23:27:09,468][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4474, reason: ApplyCommitRequest{term=2, version=4474, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-25T23:27:09,544][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,556][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,642][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,650][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,667][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,743][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,750][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,759][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,845][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,858][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][2] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,866][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,945][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][0] primary-replica resync completed with 0 operations
[2022-06-25T23:27:09,954][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][1] primary-replica resync completed with 0 operations
[2022-06-25T23:27:10,035][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations
[2022-06-25T23:27:12,237][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{1lEmZG9WRqOxtUS8cghAtA}{Jx9zUJPpQX-0W-Cfz5TnDw}{10.233.66.124}{10.233.66.124:9300}{dmr}}, term: 2, version: 4479, reason: ApplyCommitRequest{term=2, version=4479, sourceNode={onap-sdnrdb-master-0}{PuWbKEJXS7i-1HsSfeiHCg}{660dxkwCRYaujnXBwP8TpA}{10.233.67.247}{10.233.67.247:9300}{dmr}}
[2022-06-26T00:24:27,737][INFO ][o.