03:10:42.10   03:10:42.10 Welcome to the Bitnami elasticsearch container  03:10:42.18 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch  03:10:42.19 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues  03:10:42.28   03:10:42.29 INFO  ==> ** Starting Elasticsearch setup **  03:10:42.70 INFO  ==> Configuring/Initializing Elasticsearch...  03:10:43.28 INFO  ==> Setting default configuration  03:10:43.48 INFO  ==> Configuring Elasticsearch cluster settings... OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  03:11:11.60 INFO  ==> ** Elasticsearch setup finished! **  03:11:11.89 INFO  ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2023-04-22T03:12:15,582][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2023-04-22T03:12:15,585][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] JVM home [/opt/bitnami/java] [2023-04-22T03:12:15,681][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-7218273861772734240, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2023-04-22T03:12:41,191][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [aggs-matrix-stats] [2023-04-22T03:12:41,192][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [analysis-common] [2023-04-22T03:12:41,193][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [geo] [2023-04-22T03:12:41,282][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-common] [2023-04-22T03:12:41,283][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-geoip] [2023-04-22T03:12:41,284][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-user-agent] [2023-04-22T03:12:41,285][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [kibana] [2023-04-22T03:12:41,285][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-expression] [2023-04-22T03:12:41,286][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-mustache] [2023-04-22T03:12:41,287][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-painless] [2023-04-22T03:12:41,288][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [mapper-extras] [2023-04-22T03:12:41,288][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [parent-join] [2023-04-22T03:12:41,289][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [percolator] [2023-04-22T03:12:41,382][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [rank-eval] [2023-04-22T03:12:41,383][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [reindex] [2023-04-22T03:12:41,384][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [repository-url] [2023-04-22T03:12:41,385][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [tasks] [2023-04-22T03:12:41,386][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [transport-netty4] [2023-04-22T03:12:41,388][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded plugin [repository-s3] [2023-04-22T03:12:42,792][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-1] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.5.20:/dockerdata-nfs/onap/elastic-master-2)]], net usable_space [95.2gb], net total_space [99.9gb], types [nfs4] [2023-04-22T03:12:42,793][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-1] heap size [123.7mb], compressed ordinary object pointers [true] [2023-04-22T03:12:44,188][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] node name [onap-sdnrdb-master-1], node ID [zM3UkZ0iRhiHuzrREjcnvA], cluster name [sdnrdb-cluster] [2023-04-22T03:13:53,390][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-master-1] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2023-04-22T03:13:54,883][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-master-1] using discovery type [zen] and seed hosts providers [settings] [2023-04-22T03:14:02,082][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-04-22T03:14:05,887][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] initialized [2023-04-22T03:14:05,888][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] starting ... [2023-04-22T03:14:08,985][INFO ][o.e.t.TransportService ] [onap-sdnrdb-master-1] publish_address {10.233.70.235:9300}, bound_addresses {0.0.0.0:9300} [2023-04-22T03:14:10,585][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:57287}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:10,889][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:46085}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:11,083][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:39217}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:11,383][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:33429}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:12,082][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:43217}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:12,382][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:44153}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:13,083][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:57189}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:13,394][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:56281}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:14,083][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:43551}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:14,391][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:33899}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:15,082][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:47729}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:15,389][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-04-22T03:14:15,387][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.70.235:9300, remoteAddress=/127.0.0.6:37983}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:19,388][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] setting initial configuration to VotingConfiguration{CEQnV2LWTtqPtTRvSCD88Q,zM3UkZ0iRhiHuzrREjcnvA,{bootstrap-placeholder}-onap-sdnrdb-master-2} [2023-04-22T03:14:23,687][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}, targetNode={onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-0][10.233.65.108:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 2 while handling publication at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:25,390][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}, minimumTerm=3, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}, targetNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-1][10.233.70.235:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: received a newer join from {onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr} at org.elasticsearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.handleJoinRequest(JoinHelper.java:459) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:533) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:496) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:375) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:362) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:483) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:136) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$8.doRun(TransportService.java:800) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T03:14:25,685][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][19] overhead, spent [302ms] collecting in the last [1s] [2023-04-22T03:14:27,184][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 6, version: 1, delta: master node changed {previous [], current [{onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T03:14:29,283][INFO ][o.e.c.c.CoordinationState] [onap-sdnrdb-master-1] cluster UUID set to [i0R_Rf-_Rpe6uZeNeq391g] [2023-04-22T03:14:30,118][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 1, reason: Publication{term=6, version=1} [2023-04-22T03:14:30,583][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader, {onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 2, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T03:14:30,588][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-1] publish_address {10.233.70.235:9200}, bound_addresses {0.0.0.0:9200} [2023-04-22T03:14:30,589][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] started [2023-04-22T03:14:31,282][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 2, reason: Publication{term=6, version=2} [2023-04-22T03:14:31,990][INFO ][o.e.g.GatewayService ] [onap-sdnrdb-master-1] recovered [0] indices into cluster_state [2023-04-22T03:14:40,782][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-master-1] updating [action.auto_create_index] from [true] to [false] [2023-04-22T03:14:45,290][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [historicalperformance24h-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:15:03,484][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [networkelement-connection-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:15:10,183][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 13, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T03:15:12,205][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 13, reason: Publication{term=6, version=13} [2023-04-22T03:15:22,186][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [guicutthrough-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:15:38,787][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [connectionlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:15:49,186][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [userdata-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:15:58,790][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [historicalperformance15min-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:08,284][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [inventoryequipment-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:14,985][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [cmlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:23,883][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [maintenancemode-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:32,987][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [faultlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:39,403][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [mediator-server-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:48,093][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [faultcurrent-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:16:56,613][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [eventlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-04-22T03:17:07,519][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[networkelement-connection-v7][2]]]). [2023-04-22T04:14:18,886][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 97, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T04:14:23,914][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 97, reason: Publication{term=6, version=97} [2023-04-22T04:14:24,684][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][2] primary-replica resync completed with 0 operations [2023-04-22T04:14:24,783][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance24h-v7][0] primary-replica resync completed with 0 operations [2023-04-22T04:14:24,983][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][3] primary-replica resync completed with 0 operations [2023-04-22T04:14:25,183][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][3] primary-replica resync completed with 0 operations [2023-04-22T04:14:25,582][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [cmlog-v7][3] primary-replica resync completed with 0 operations [2023-04-22T04:14:25,691][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [52.7s] (43 delayed shards) [2023-04-22T04:14:25,882][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 98, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T04:14:25,983][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][3] primary-replica resync completed with 0 operations [2023-04-22T04:14:26,482][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][2] primary-replica resync completed with 0 operations [2023-04-22T04:14:26,915][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 98, reason: Publication{term=6, version=98} [2023-04-22T04:14:27,497][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 99, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T04:14:28,082][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 99, reason: Publication{term=6, version=99} [2023-04-22T04:14:28,287][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 100, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T04:14:28,706][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 100, reason: Publication{term=6, version=100} [2023-04-22T04:14:28,709][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 101, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T04:14:29,214][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 101, reason: Publication{term=6, version=101} [2023-04-22T04:14:29,483][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 102, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T04:14:29,714][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 102, reason: Publication{term=6, version=102} [2023-04-22T04:14:29,995][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 103, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T04:14:34,621][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 103, reason: Publication{term=6, version=103} [2023-04-22T04:14:34,624][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 104, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T04:14:35,115][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 104, reason: Publication{term=6, version=104} [2023-04-22T04:15:07,688][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 137, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T04:15:12,331][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 137, reason: Publication{term=6, version=137} [2023-04-22T04:15:12,490][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][3] primary-replica resync completed with 0 operations [2023-04-22T04:15:12,606][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-04-22T04:15:12,786][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][1] primary-replica resync completed with 0 operations [2023-04-22T04:15:12,883][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [guicutthrough-v7][0] primary-replica resync completed with 0 operations [2023-04-22T04:15:12,993][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,195][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][1] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,283][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][4] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,482][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [maintenancemode-v7][1] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,596][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,701][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][3] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,782][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 138, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T04:15:13,890][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][0] primary-replica resync completed with 0 operations [2023-04-22T04:15:13,982][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [connectionlog-v7][2] primary-replica resync completed with 0 operations [2023-04-22T04:15:14,129][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 138, reason: Publication{term=6, version=138} [2023-04-22T04:15:14,285][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 139, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T04:15:16,282][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 139, reason: Publication{term=6, version=139} [2023-04-22T04:15:17,790][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 141, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T04:15:18,219][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 141, reason: Publication{term=6, version=141} [2023-04-22T04:15:18,389][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 142, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T04:15:18,784][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 142, reason: Publication{term=6, version=142} [2023-04-22T04:15:21,529][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 145, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T04:15:24,648][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 145, reason: Publication{term=6, version=145} [2023-04-22T04:15:30,965][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[networkelement-connection-v7][1], [historicalperformance24h-v7][4], [cmlog-v7][0], [inventoryequipment-v7][2]]]). [2023-04-22T04:15:33,283][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [34.2s] (30 delayed shards) [2023-04-22T04:16:43,529][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][4]]]). [2023-04-22T05:14:29,885][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 219, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T05:14:34,566][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 219, reason: Publication{term=6, version=219} [2023-04-22T05:14:34,574][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.2s] (43 delayed shards) [2023-04-22T05:14:34,582][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 220, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T05:14:34,992][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 220, reason: Publication{term=6, version=220} [2023-04-22T05:14:35,084][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 221, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T05:14:35,535][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 221, reason: Publication{term=6, version=221} [2023-04-22T05:14:35,690][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 222, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T05:14:36,098][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 222, reason: Publication{term=6, version=222} [2023-04-22T05:14:36,583][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 223, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T05:14:38,584][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 223, reason: Publication{term=6, version=223} [2023-04-22T05:14:38,589][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 224, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T05:14:38,931][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 224, reason: Publication{term=6, version=224} [2023-04-22T05:15:19,589][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 271, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T05:15:23,630][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 271, reason: Publication{term=6, version=271} [2023-04-22T05:15:23,639][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 272, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T05:15:23,968][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 272, reason: Publication{term=6, version=272} [2023-04-22T05:15:24,183][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 273, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T05:15:26,137][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 273, reason: Publication{term=6, version=273} [2023-04-22T05:15:26,143][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 274, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T05:15:26,533][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 274, reason: Publication{term=6, version=274} [2023-04-22T05:15:26,640][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 275, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T05:15:26,760][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 275, reason: Publication{term=6, version=275} [2023-04-22T05:15:27,333][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 277, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T05:15:28,937][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 277, reason: Publication{term=6, version=277} [2023-04-22T05:15:33,637][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[eventlog-v7][1], [eventlog-v7][4], [historicalperformance24h-v7][4], [networkelement-connection-v7][1]]]). [2023-04-22T05:15:34,764][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [44.7s] (34 delayed shards) [2023-04-22T05:16:30,298][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T06:14:36,413][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 348, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T06:14:39,283][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 348, reason: Publication{term=6, version=348} [2023-04-22T06:14:39,289][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.1s] (43 delayed shards) [2023-04-22T06:14:39,293][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 349, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T06:14:39,565][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 349, reason: Publication{term=6, version=349} [2023-04-22T06:14:39,797][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 350, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T06:14:41,448][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 350, reason: Publication{term=6, version=350} [2023-04-22T06:14:42,367][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 352, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T06:14:42,953][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 352, reason: Publication{term=6, version=352} [2023-04-22T06:15:26,262][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][2]]]). [2023-04-22T06:15:27,070][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 420, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T06:15:30,972][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 420, reason: Publication{term=6, version=420} [2023-04-22T06:15:30,976][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56s] (44 delayed shards) [2023-04-22T06:15:30,978][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 421, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T06:15:31,295][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 421, reason: Publication{term=6, version=421} [2023-04-22T06:15:31,393][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 422, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T06:15:31,782][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 422, reason: Publication{term=6, version=422} [2023-04-22T06:15:32,353][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 423, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T06:15:35,403][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 423, reason: Publication{term=6, version=423} [2023-04-22T06:16:22,795][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T07:04:52,507][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:33222}], closing connection [2023-04-22T07:11:09,982][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][14212] overhead, spent [294ms] collecting in the last [1s] [2023-04-22T07:14:39,706][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 492, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T07:14:43,476][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 492, reason: Publication{term=6, version=492} [2023-04-22T07:14:43,483][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.2s] (43 delayed shards) [2023-04-22T07:14:43,487][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 493, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T07:14:43,972][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 493, reason: Publication{term=6, version=493} [2023-04-22T07:14:43,978][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 494, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T07:14:44,474][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 494, reason: Publication{term=6, version=494} [2023-04-22T07:14:44,593][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 495, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T07:14:44,983][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 495, reason: Publication{term=6, version=495} [2023-04-22T07:14:44,986][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 496, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T07:14:45,570][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 496, reason: Publication{term=6, version=496} [2023-04-22T07:14:45,636][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 497, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T07:14:46,071][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 497, reason: Publication{term=6, version=497} [2023-04-22T07:14:46,077][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 498, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T07:14:46,524][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 498, reason: Publication{term=6, version=498} [2023-04-22T07:14:46,792][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 499, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T07:14:49,089][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 499, reason: Publication{term=6, version=499} [2023-04-22T07:15:32,284][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 557, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T07:15:35,395][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 557, reason: Publication{term=6, version=557} [2023-04-22T07:15:36,968][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 559, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T07:15:37,418][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 559, reason: Publication{term=6, version=559} [2023-04-22T07:15:37,488][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 560, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T07:15:39,301][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 560, reason: Publication{term=6, version=560} [2023-04-22T07:15:39,939][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 562, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T07:15:40,185][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 562, reason: Publication{term=6, version=562} [2023-04-22T07:15:40,293][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 563, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T07:15:40,505][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 563, reason: Publication{term=6, version=563} [2023-04-22T07:15:40,509][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[networkelement-connection-v7][1], [historicalperformance24h-v7][4]]]). [2023-04-22T07:15:42,572][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 566, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T07:15:46,053][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 566, reason: Publication{term=6, version=566} [2023-04-22T07:15:50,796][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [41.4s] (38 delayed shards) [2023-04-22T07:16:31,353][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T08:04:55,727][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:55420}], closing connection [2023-04-22T08:14:45,813][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 637, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T08:14:45,967][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 637, reason: Publication{term=6, version=637} [2023-04-22T08:14:46,701][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 638, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T08:14:50,791][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 638, reason: Publication{term=6, version=638} [2023-04-22T08:14:50,795][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.9s] (43 delayed shards) [2023-04-22T08:14:50,797][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader, {onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 639, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r},{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T08:14:51,096][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r},{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 639, reason: Publication{term=6, version=639} [2023-04-22T08:14:51,192][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 640, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T08:14:51,453][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 640, reason: Publication{term=6, version=640} [2023-04-22T08:14:51,776][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 641, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T08:14:53,310][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 641, reason: Publication{term=6, version=641} [2023-04-22T08:15:40,991][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 707, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T08:15:44,980][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 707, reason: Publication{term=6, version=707} [2023-04-22T08:15:44,984][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.9s] (44 delayed shards) [2023-04-22T08:15:45,801][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 709, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T08:15:46,286][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 709, reason: Publication{term=6, version=709} [2023-04-22T08:15:46,387][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 710, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T08:15:47,081][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 710, reason: Publication{term=6, version=710} [2023-04-22T08:15:47,503][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 711, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T08:15:49,304][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 711, reason: Publication{term=6, version=711} [2023-04-22T08:16:35,453][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T09:04:25,635][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] invalid internal transport message format, got (4a,44,57,50), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:48998}], closing connection [2023-04-22T09:14:48,812][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 781, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T09:14:48,930][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 781, reason: Publication{term=6, version=781} [2023-04-22T09:14:51,750][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 782, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T09:14:55,118][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 782, reason: Publication{term=6, version=782} [2023-04-22T09:14:55,122][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.6s] (43 delayed shards) [2023-04-22T09:14:55,124][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader, {onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 783, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r},{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T09:14:55,482][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r},{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 783, reason: Publication{term=6, version=783} [2023-04-22T09:14:55,595][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 784, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T09:14:55,995][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 784, reason: Publication{term=6, version=784} [2023-04-22T09:14:56,081][WARN ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] failed to validate incoming join request from node [{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}] org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.65.108:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-22T09:14:56,085][WARN ][o.e.t.TransportService ] [onap-sdnrdb-master-1] Transport response handler not found of id [134535] [2023-04-22T09:14:56,883][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 785, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T09:14:59,046][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 785, reason: Publication{term=6, version=785} [2023-04-22T09:15:35,772][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][4]]]). [2023-04-22T09:15:47,405][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 853, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T09:15:50,909][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 853, reason: Publication{term=6, version=853} [2023-04-22T09:15:50,912][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.4s] (44 delayed shards) [2023-04-22T09:15:50,915][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 854, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T09:15:51,156][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 854, reason: Publication{term=6, version=854} [2023-04-22T09:15:51,287][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 855, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T09:15:51,543][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 855, reason: Publication{term=6, version=855} [2023-04-22T09:15:52,083][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 856, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T09:15:53,668][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 856, reason: Publication{term=6, version=856} [2023-04-22T09:16:36,513][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][4]]]). [2023-04-22T10:14:52,009][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 927, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T10:14:52,112][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 927, reason: Publication{term=6, version=927} [2023-04-22T10:14:54,913][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 928, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T10:14:55,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 928, reason: Publication{term=6, version=928} [2023-04-22T10:14:56,884][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 929, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T10:14:59,830][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 929, reason: Publication{term=6, version=929} [2023-04-22T10:14:59,834][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57s] (43 delayed shards) [2023-04-22T10:14:59,835][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 930, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T10:15:00,039][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 930, reason: Publication{term=6, version=930} [2023-04-22T10:15:00,084][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 931, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T10:15:00,310][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 931, reason: Publication{term=6, version=931} [2023-04-22T10:15:00,784][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 932, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T10:15:02,338][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 932, reason: Publication{term=6, version=932} [2023-04-22T10:15:42,480][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][0]]]). [2023-04-22T10:15:52,028][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 1000, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T10:15:55,145][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1000, reason: Publication{term=6, version=1000} [2023-04-22T10:15:55,150][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.8s] (44 delayed shards) [2023-04-22T10:15:55,182][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 1001, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T10:15:55,574][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1001, reason: Publication{term=6, version=1001} [2023-04-22T10:15:55,688][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 1002, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T10:15:56,110][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1002, reason: Publication{term=6, version=1002} [2023-04-22T10:15:56,182][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 1003, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T10:15:56,482][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1003, reason: Publication{term=6, version=1003} [2023-04-22T10:15:56,592][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 1004, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T10:15:56,758][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1004, reason: Publication{term=6, version=1004} [2023-04-22T10:15:56,760][WARN ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] failed to validate incoming join request from node [{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}] org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-2][10.233.71.179:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-22T10:15:57,782][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 1005, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T10:16:00,356][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1005, reason: Publication{term=6, version=1005} [2023-04-22T10:16:51,893][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T11:14:54,873][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 1075, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T11:14:54,984][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 1075, reason: Publication{term=6, version=1075} [2023-04-22T11:14:57,017][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 1076, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T11:14:57,494][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 1076, reason: Publication{term=6, version=1076} [2023-04-22T11:15:00,746][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 1077, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T11:15:04,150][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 1077, reason: Publication{term=6, version=1077} [2023-04-22T11:15:04,154][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.5s] (43 delayed shards) [2023-04-22T11:15:04,156][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 1078, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T11:15:04,368][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 1078, reason: Publication{term=6, version=1078} [2023-04-22T11:15:04,409][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 1079, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T11:15:04,557][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 1079, reason: Publication{term=6, version=1079} [2023-04-22T11:15:04,798][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 6, version: 1080, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T11:15:06,928][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 1080, reason: Publication{term=6, version=1080} [2023-04-22T11:15:45,313][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][0]]]). [2023-04-22T11:15:57,758][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 6, version: 1148, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T11:15:59,888][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1148, reason: Publication{term=6, version=1148} [2023-04-22T11:15:59,892][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.8s] (44 delayed shards) [2023-04-22T11:16:00,465][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 6, version: 1149, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T11:16:04,442][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 6, version: 1149, reason: Publication{term=6, version=1149} [2023-04-22T11:16:47,739][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T12:14:56,966][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 6, version: 1217, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T12:14:57,104][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 1217, reason: Publication{term=6, version=1217} [2023-04-22T12:15:00,093][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 6, version: 1218, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T12:15:00,350][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 6, version: 1218, reason: Publication{term=6, version=1218} [2023-04-22T12:15:04,747][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 6, version: 1219, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T12:15:07,665][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 6, version: 1219, reason: Publication{term=6, version=1219} [2023-04-22T12:15:07,669][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57s] (43 delayed shards) [2023-04-22T12:15:07,670][WARN ][o.e.c.c.C.CoordinatorPublication] [onap-sdnrdb-master-1] after [3s] publication of cluster state version [1219] is still waiting for {onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} [SENT_APPLY_COMMIT] [2023-04-22T12:15:07,671][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [{onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}], current []}, term: 6, version: 1219, reason: becoming candidate: joinLeaderInTerm [2023-04-22T12:15:08,473][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} elect leader, {onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 8, version: 1220, delta: master node changed {previous [], current [{onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T12:15:10,473][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}]}, added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1220, reason: Publication{term=8, version=1220} [2023-04-22T12:15:10,874][INFO ][o.e.c.c.JoinHelper ] [onap-sdnrdb-master-1] failed to join {onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} with JoinRequest{sourceNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}, minimumTerm=6, optionalJoin=Optional[Join{term=7, lastAcceptedTerm=6, lastAcceptedVersion=1219, sourceNode={onap-sdnrdb-master-1}{zM3UkZ0iRhiHuzrREjcnvA}{HCfOUQzSQTSUZ06uzSYFqA}{10.233.70.235}{10.233.70.235:9300}{dmr}, targetNode={onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [onap-sdnrdb-master-2][10.233.71.179:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 8 while handling publication at org.elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1083) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:268) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-04-22T12:15:49,066][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][1]]]). [2023-04-22T12:16:00,373][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1289, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T12:16:03,861][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1289, reason: Publication{term=8, version=1289} [2023-04-22T12:16:03,865][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.5s] (44 delayed shards) [2023-04-22T12:16:03,866][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1290, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T12:16:04,066][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1290, reason: Publication{term=8, version=1290} [2023-04-22T12:16:04,117][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1291, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T12:16:04,259][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1291, reason: Publication{term=8, version=1291} [2023-04-22T12:16:04,974][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1292, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T12:16:07,300][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1292, reason: Publication{term=8, version=1292} [2023-04-22T12:16:45,981][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T13:15:00,030][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 1361, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T13:15:00,130][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1361, reason: Publication{term=8, version=1361} [2023-04-22T13:15:03,125][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 1362, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T13:15:03,602][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1362, reason: Publication{term=8, version=1362} [2023-04-22T13:15:07,896][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1364, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T13:15:11,290][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1364, reason: Publication{term=8, version=1364} [2023-04-22T13:15:11,292][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.5s] (43 delayed shards) [2023-04-22T13:15:11,294][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1365, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T13:15:11,545][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1365, reason: Publication{term=8, version=1365} [2023-04-22T13:15:11,591][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1366, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T13:15:11,896][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1366, reason: Publication{term=8, version=1366} [2023-04-22T13:15:12,197][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1367, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T13:15:13,747][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1367, reason: Publication{term=8, version=1367} [2023-04-22T13:15:48,495][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][1]]]). [2023-04-22T13:16:04,857][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1434, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T13:16:07,871][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1434, reason: Publication{term=8, version=1434} [2023-04-22T13:16:07,875][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.9s] (44 delayed shards) [2023-04-22T13:16:07,878][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1435, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T13:16:08,073][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1435, reason: Publication{term=8, version=1435} [2023-04-22T13:16:08,115][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1436, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T13:16:08,225][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1436, reason: Publication{term=8, version=1436} [2023-04-22T13:16:08,482][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1437, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T13:16:10,068][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1437, reason: Publication{term=8, version=1437} [2023-04-22T13:16:57,576][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T14:15:03,086][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 1507, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T14:15:03,381][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1507, reason: Publication{term=8, version=1507} [2023-04-22T14:15:06,166][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 1508, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T14:15:06,446][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1508, reason: Publication{term=8, version=1508} [2023-04-22T14:15:12,099][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1510, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T14:15:16,447][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1510, reason: Publication{term=8, version=1510} [2023-04-22T14:15:16,450][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.6s] (43 delayed shards) [2023-04-22T14:15:16,452][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1511, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T14:15:16,646][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1511, reason: Publication{term=8, version=1511} [2023-04-22T14:15:16,694][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1512, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T14:15:16,865][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1512, reason: Publication{term=8, version=1512} [2023-04-22T14:15:16,867][WARN ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] failed to validate incoming join request from node [{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}] org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.65.108:9300][internal:cluster/coordination/join/validate] disconnected [2023-04-22T14:15:16,881][WARN ][o.e.t.TransportService ] [onap-sdnrdb-master-1] Transport response handler not found of id [252806] [2023-04-22T14:15:17,914][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1513, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T14:15:19,369][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1513, reason: Publication{term=8, version=1513} [2023-04-22T14:15:59,984][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][1]]]). [2023-04-22T14:16:08,298][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1580, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T14:16:10,783][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1580, reason: Publication{term=8, version=1580} [2023-04-22T14:16:10,787][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.5s] (44 delayed shards) [2023-04-22T14:16:10,789][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1581, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T14:16:11,104][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1581, reason: Publication{term=8, version=1581} [2023-04-22T14:16:11,185][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1582, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T14:16:11,343][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1582, reason: Publication{term=8, version=1582} [2023-04-22T14:16:11,463][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1583, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T14:16:12,811][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1583, reason: Publication{term=8, version=1583} [2023-04-22T14:16:54,397][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][4]]]). [2023-04-22T15:15:06,124][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 1654, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T15:15:06,349][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1654, reason: Publication{term=8, version=1654} [2023-04-22T15:15:08,412][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 1656, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T15:15:08,892][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1656, reason: Publication{term=8, version=1656} [2023-04-22T15:15:17,889][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1657, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T15:15:21,055][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1657, reason: Publication{term=8, version=1657} [2023-04-22T15:15:21,060][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.8s] (43 delayed shards) [2023-04-22T15:15:21,062][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1658, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T15:15:21,487][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1658, reason: Publication{term=8, version=1658} [2023-04-22T15:15:21,584][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1659, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T15:15:21,750][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1659, reason: Publication{term=8, version=1659} [2023-04-22T15:15:22,064][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1660, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T15:15:23,425][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1660, reason: Publication{term=8, version=1660} [2023-04-22T15:15:59,285][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][1]]]). [2023-04-22T15:16:11,436][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1727, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T15:16:14,119][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1727, reason: Publication{term=8, version=1727} [2023-04-22T15:16:14,122][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.3s] (44 delayed shards) [2023-04-22T15:16:14,123][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1728, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T15:16:14,350][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1728, reason: Publication{term=8, version=1728} [2023-04-22T15:16:14,402][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1729, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T15:16:14,604][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1729, reason: Publication{term=8, version=1729} [2023-04-22T15:16:14,608][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1730, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T15:16:14,824][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1730, reason: Publication{term=8, version=1730} [2023-04-22T15:16:14,881][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1731, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T15:16:15,029][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1731, reason: Publication{term=8, version=1731} [2023-04-22T15:16:15,548][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1732, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T15:16:17,270][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1732, reason: Publication{term=8, version=1732} [2023-04-22T15:16:52,381][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T16:15:08,369][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 1803, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T16:15:08,520][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1803, reason: Publication{term=8, version=1803} [2023-04-22T16:15:11,344][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 1804, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T16:15:11,523][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1804, reason: Publication{term=8, version=1804} [2023-04-22T16:15:22,034][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1805, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T16:15:24,566][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1805, reason: Publication{term=8, version=1805} [2023-04-22T16:15:24,570][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.4s] (43 delayed shards) [2023-04-22T16:15:25,227][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1806, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T16:15:26,763][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1806, reason: Publication{term=8, version=1806} [2023-04-22T16:16:04,687][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][0]]]). [2023-04-22T16:16:15,516][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 1873, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T16:16:18,133][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1873, reason: Publication{term=8, version=1873} [2023-04-22T16:16:18,136][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.3s] (44 delayed shards) [2023-04-22T16:16:18,816][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 1874, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T16:16:19,996][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 1874, reason: Publication{term=8, version=1874} [2023-04-22T16:16:59,459][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T16:22:18,588][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T16:29:18,817][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T16:31:18,873][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T16:37:19,072][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T16:46:49,372][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T16:47:49,406][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T16:50:19,485][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T17:04:19,943][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T17:15:11,296][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 1946, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T17:15:11,419][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1946, reason: Publication{term=8, version=1946} [2023-04-22T17:15:13,415][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 1947, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T17:15:13,603][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 1947, reason: Publication{term=8, version=1947} [2023-04-22T17:15:25,134][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 1948, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T17:15:27,978][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1948, reason: Publication{term=8, version=1948} [2023-04-22T17:15:27,981][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.1s] (43 delayed shards) [2023-04-22T17:15:28,254][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 1949, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T17:15:29,492][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 1949, reason: Publication{term=8, version=1949} [2023-04-22T17:16:04,449][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][0]]]). [2023-04-22T17:16:18,778][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 2017, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T17:16:21,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2017, reason: Publication{term=8, version=2017} [2023-04-22T17:16:21,654][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.1s] (44 delayed shards) [2023-04-22T17:16:21,655][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 2018, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T17:16:21,846][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2018, reason: Publication{term=8, version=2018} [2023-04-22T17:16:21,985][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 2019, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T17:16:22,391][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2019, reason: Publication{term=8, version=2019} [2023-04-22T17:16:22,597][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 2020, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T17:16:23,835][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2020, reason: Publication{term=8, version=2020} [2023-04-22T17:17:07,967][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][4]]]). [2023-04-22T17:43:51,986][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T17:51:52,230][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T17:57:22,400][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T18:03:22,595][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T18:09:22,792][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T18:15:13,384][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 2091, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T18:15:13,487][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 2091, reason: Publication{term=8, version=2091} [2023-04-22T18:15:16,435][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 2092, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T18:15:16,637][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 2092, reason: Publication{term=8, version=2092} [2023-04-22T18:15:28,177][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 2093, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T18:15:33,091][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 2093, reason: Publication{term=8, version=2093} [2023-04-22T18:15:33,094][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55s] (43 delayed shards) [2023-04-22T18:15:33,096][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 2094, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T18:15:33,598][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 2094, reason: Publication{term=8, version=2094} [2023-04-22T18:15:33,658][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 2095, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T18:15:33,999][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 2095, reason: Publication{term=8, version=2095} [2023-04-22T18:15:34,482][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 2096, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T18:15:35,984][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 2096, reason: Publication{term=8, version=2096} [2023-04-22T18:16:24,268][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 2155, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T18:16:29,270][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2155, reason: Publication{term=8, version=2155} [2023-04-22T18:16:30,010][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 2157, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T18:16:30,209][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2157, reason: Publication{term=8, version=2157} [2023-04-22T18:16:30,275][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 2158, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T18:16:30,687][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2158, reason: Publication{term=8, version=2158} [2023-04-22T18:16:31,469][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[historicalperformance24h-v7][4]]]). [2023-04-22T18:16:31,861][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 2161, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T18:16:33,638][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2161, reason: Publication{term=8, version=2161} [2023-04-22T18:16:36,835][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [47.4s] (39 delayed shards) [2023-04-22T18:17:26,528][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][3]]]). [2023-04-22T18:26:24,892][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T18:35:55,181][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T18:37:25,222][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T18:51:55,674][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T19:05:56,106][INFO ][o.e.c.r.a.DiskThresholdMonitor] [onap-sdnrdb-master-1] skipping monitor as a check is already in progress [2023-04-22T19:15:16,398][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} reason: disconnected], term: 8, version: 2232, delta: removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T19:15:16,538][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 2232, reason: Publication{term=8, version=2232} [2023-04-22T19:15:19,497][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r} join existing leader], term: 8, version: 2233, delta: added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}} [2023-04-22T19:15:19,709][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-6c56bdbcb9-rqlmf}{_ZrvxDBHQC-SeyD61ios7g}{k2owJQ6iQ8WWQl-tqF6Dhg}{10.233.66.18}{10.233.66.18:9300}{r}}, term: 8, version: 2233, reason: Publication{term=8, version=2233} [2023-04-22T19:15:34,409][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} reason: disconnected], term: 8, version: 2234, delta: removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T19:15:37,352][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 2234, reason: Publication{term=8, version=2234} [2023-04-22T19:15:37,355][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57s] (43 delayed shards) [2023-04-22T19:15:37,682][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr} join existing leader], term: 8, version: 2235, delta: added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}} [2023-04-22T19:15:38,980][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{CEQnV2LWTtqPtTRvSCD88Q}{kp5qJOBKSvuKz_WUs2GtHg}{10.233.65.108}{10.233.65.108:9300}{dmr}}, term: 8, version: 2235, reason: Publication{term=8, version=2235} [2023-04-22T19:16:15,932][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][1]]]). [2023-04-22T19:16:30,901][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} reason: disconnected], term: 8, version: 2302, delta: removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T19:16:33,309][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2302, reason: Publication{term=8, version=2302} [2023-04-22T19:16:33,311][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.5s] (44 delayed shards) [2023-04-22T19:16:34,112][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr} join existing leader], term: 8, version: 2303, delta: added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}} [2023-04-22T19:16:35,525][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{acckIT0fTx6TsmjaQcaCGA}{ZXaGZmh_T26tMDSHemU-2w}{10.233.71.179}{10.233.71.179:9300}{dmr}}, term: 8, version: 2303, reason: Publication{term=8, version=2303} [2023-04-22T19:17:16,815][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[historicalperformance24h-v7][4]]]).