00:04:08.65   00:04:08.73 Welcome to the Bitnami elasticsearch container  00:04:08.75 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch  00:04:08.75 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues  00:04:08.83   00:04:08.84 INFO  ==> ** Starting Elasticsearch setup **  00:04:09.16 INFO  ==> Configuring/Initializing Elasticsearch...  00:04:09.55 INFO  ==> Setting default configuration  00:04:09.74 INFO  ==> Configuring Elasticsearch cluster settings... OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.  00:04:32.64 INFO  ==> ** Elasticsearch setup finished! **  00:04:32.84 INFO  ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2023-03-06T00:05:24,638][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.0-96-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2023-03-06T00:05:24,737][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] JVM home [/opt/bitnami/java] [2023-03-06T00:05:24,738][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-13224733887098201218, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2023-03-06T00:05:47,337][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [aggs-matrix-stats] [2023-03-06T00:05:47,338][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [analysis-common] [2023-03-06T00:05:47,338][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [geo] [2023-03-06T00:05:47,339][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-common] [2023-03-06T00:05:47,339][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-geoip] [2023-03-06T00:05:47,340][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [ingest-user-agent] [2023-03-06T00:05:47,340][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [kibana] [2023-03-06T00:05:47,340][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-expression] [2023-03-06T00:05:47,341][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-mustache] [2023-03-06T00:05:47,341][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [lang-painless] [2023-03-06T00:05:47,342][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [mapper-extras] [2023-03-06T00:05:47,342][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [parent-join] [2023-03-06T00:05:47,342][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [percolator] [2023-03-06T00:05:47,343][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [rank-eval] [2023-03-06T00:05:47,343][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [reindex] [2023-03-06T00:05:47,344][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [repository-url] [2023-03-06T00:05:47,344][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [tasks] [2023-03-06T00:05:47,345][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded module [transport-netty4] [2023-03-06T00:05:47,346][INFO ][o.e.p.PluginsService ] [onap-sdnrdb-master-1] loaded plugin [repository-s3] [2023-03-06T00:05:48,443][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-1] using [1] data paths, mounts [[/bitnami/elasticsearch/data (192.168.13.53:/dockerdata-nfs/onap/elastic-master-0)]], net usable_space [95.4gb], net total_space [99.9gb], types [nfs4] [2023-03-06T00:05:48,444][INFO ][o.e.e.NodeEnvironment ] [onap-sdnrdb-master-1] heap size [123.7mb], compressed ordinary object pointers [true] [2023-03-06T00:05:49,041][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] node name [onap-sdnrdb-master-1], node ID [L-umM5_xSpy7TzE6qL-O0A], cluster name [sdnrdb-cluster] [2023-03-06T00:06:58,543][INFO ][o.e.t.NettyAllocator ] [onap-sdnrdb-master-1] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2023-03-06T00:06:59,938][INFO ][o.e.d.DiscoveryModule ] [onap-sdnrdb-master-1] using discovery type [zen] and seed hosts providers [settings] [2023-03-06T00:07:04,843][WARN ][o.e.g.DanglingIndicesState] [onap-sdnrdb-master-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-03-06T00:07:08,135][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] initialized [2023-03-06T00:07:08,136][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] starting ... [2023-03-06T00:07:09,251][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][1] overhead, spent [301ms] collecting in the last [1s] [2023-03-06T00:07:10,846][INFO ][o.e.t.TransportService ] [onap-sdnrdb-master-1] publish_address {10.233.67.49:9300}, bound_addresses {0.0.0.0:9300} [2023-03-06T00:07:12,539][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.49:9300, remoteAddress=/127.0.0.6:60489}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-03-06T00:07:13,037][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.49:9300, remoteAddress=/127.0.0.6:39667}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-03-06T00:07:14,043][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.49:9300, remoteAddress=/127.0.0.6:48659}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-03-06T00:07:15,036][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.49:9300, remoteAddress=/127.0.0.6:47779}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-03-06T00:07:16,036][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.49:9300, remoteAddress=/127.0.0.6:35243}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-03-06T00:07:17,025][WARN ][o.e.t.TcpTransport ] [onap-sdnrdb-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.67.49:9300, remoteAddress=/127.0.0.6:45527}], closing connection java.lang.IllegalStateException: transport not ready yet to handle incoming requests at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:943) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:136) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) [transport-netty4-client-7.9.3.jar:7.9.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:834) [?:?] [2023-03-06T00:07:17,142][INFO ][o.e.b.BootstrapChecks ] [onap-sdnrdb-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-03-06T00:07:21,746][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][13] overhead, spent [306ms] collecting in the last [1.2s] [2023-03-06T00:07:27,545][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr}]; discovery will continue using [10.233.67.197:9300, 10.233.71.51:9300, 10.233.65.81:9300] from hosts providers and [{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2023-03-06T00:07:37,549][WARN ][o.e.c.c.ClusterFormationFailureHelper] [onap-sdnrdb-master-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [onap-sdnrdb-master-0, onap-sdnrdb-master-1, onap-sdnrdb-master-2] to bootstrap a cluster: have discovered [{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr}]; discovery will continue using [10.233.67.197:9300, 10.233.71.51:9300, 10.233.65.81:9300] from hosts providers and [{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2023-03-06T00:07:40,943][INFO ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-onap-sdnrdb-master-0,L-umM5_xSpy7TzE6qL-O0A,u8g8HRa2Qkq4RSWsx9v72g} [2023-03-06T00:07:43,147][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] elected-as-master ([2] nodes joined)[{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr} elect leader, {onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T00:07:44,419][INFO ][o.e.c.c.CoordinationState] [onap-sdnrdb-master-1] cluster UUID set to [QWyUoxnjTaSvf1Fwzi1PDQ] [2023-03-06T00:07:45,415][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] master node changed {previous [], current [{onap-sdnrdb-master-1}{L-umM5_xSpy7TzE6qL-O0A}{6fskdG0SQ-Gv9t5gMS4iwQ}{10.233.67.49}{10.233.67.49:9300}{dmr}]}, added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1, reason: Publication{term=1, version=1} [2023-03-06T00:07:45,738][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader, {onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 2, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T00:07:46,037][INFO ][o.e.h.AbstractHttpServerTransport] [onap-sdnrdb-master-1] publish_address {10.233.67.49:9200}, bound_addresses {0.0.0.0:9200} [2023-03-06T00:07:46,038][INFO ][o.e.n.Node ] [onap-sdnrdb-master-1] started [2023-03-06T00:07:46,543][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 2, reason: Publication{term=1, version=2} [2023-03-06T00:07:47,040][INFO ][o.e.g.GatewayService ] [onap-sdnrdb-master-1] recovered [0] indices into cluster_state [2023-03-06T00:07:48,837][INFO ][o.e.c.s.ClusterSettings ] [onap-sdnrdb-master-1] updating [action.auto_create_index] from [true] to [false] [2023-03-06T00:07:52,043][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [faultcurrent-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:06,543][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [faultlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:14,137][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [historicalperformance24h-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:21,646][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [userdata-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:28,635][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [networkelement-connection-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:35,546][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [eventlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:43,940][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [connectionlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:51,449][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [mediator-server-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:08:58,544][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [maintenancemode-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:09:04,440][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [historicalperformance15min-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:09:12,439][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [guicutthrough-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:09:21,337][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [inventoryequipment-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:09:28,138][INFO ][o.e.c.m.MetadataCreateIndexService] [onap-sdnrdb-master-1] [cmlog-v7] creating index, cause [api], templates [], shards [5]/[1] [2023-03-06T00:09:49,619][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][4]]]). [2023-03-06T00:12:12,258][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 98, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T00:12:21,229][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 98, reason: Publication{term=1, version=98} [2023-03-06T01:07:40,547][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 167, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:07:49,237][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 167, reason: Publication{term=1, version=167} [2023-03-06T01:07:49,540][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][2] primary-replica resync completed with 0 operations [2023-03-06T01:07:49,739][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][0] primary-replica resync completed with 0 operations [2023-03-06T01:07:49,946][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][0] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,135][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [networkelement-connection-v7][1] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,235][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][0] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,336][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][0] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,458][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultcurrent-v7][1] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,552][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][0] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,642][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [49.6s] (43 delayed shards) [2023-03-06T01:07:50,945][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][2] primary-replica resync completed with 0 operations [2023-03-06T01:07:50,950][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][0] primary-replica resync completed with 0 operations [2023-03-06T01:07:51,138][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 168, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T01:07:51,453][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 168, reason: Publication{term=1, version=168} [2023-03-06T01:07:51,542][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 169, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:07:53,663][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 169, reason: Publication{term=1, version=169} [2023-03-06T01:07:54,235][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 170, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:07:54,628][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 170, reason: Publication{term=1, version=170} [2023-03-06T01:07:54,643][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader, {onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 171, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr},{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T01:07:55,333][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr},{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 171, reason: Publication{term=1, version=171} [2023-03-06T01:07:55,639][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 172, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:07:55,943][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 172, reason: Publication{term=1, version=172} [2023-03-06T01:07:56,037][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 173, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:07:57,539][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 173, reason: Publication{term=1, version=173} [2023-03-06T01:07:57,740][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 174, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:07:58,333][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 174, reason: Publication{term=1, version=174} [2023-03-06T01:07:58,748][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 175, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T01:08:00,760][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 175, reason: Publication{term=1, version=175} [2023-03-06T01:09:40,064][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][0]]]). [2023-03-06T01:12:10,240][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 241, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T01:12:13,329][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 241, reason: Publication{term=1, version=241} [2023-03-06T01:12:13,536][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [faultlog-v7][4] primary-replica resync completed with 0 operations [2023-03-06T01:12:13,543][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][2] primary-replica resync completed with 0 operations [2023-03-06T01:12:13,634][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [inventoryequipment-v7][4] primary-replica resync completed with 0 operations [2023-03-06T01:12:13,843][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][2] primary-replica resync completed with 0 operations [2023-03-06T01:12:14,138][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [historicalperformance15min-v7][4] primary-replica resync completed with 0 operations [2023-03-06T01:12:14,637][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][4] primary-replica resync completed with 0 operations [2023-03-06T01:12:14,638][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [eventlog-v7][2] primary-replica resync completed with 0 operations [2023-03-06T01:12:14,746][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.1s] (44 delayed shards) [2023-03-06T01:12:14,746][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [mediator-server-v7][4] primary-replica resync completed with 0 operations [2023-03-06T01:12:14,836][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 242, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T01:12:14,942][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][2] primary-replica resync completed with 0 operations [2023-03-06T01:12:14,951][INFO ][o.e.i.s.IndexShard ] [onap-sdnrdb-master-1] [userdata-v7][4] primary-replica resync completed with 0 operations [2023-03-06T01:12:15,145][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 242, reason: Publication{term=1, version=242} [2023-03-06T01:12:15,342][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 243, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T01:12:15,473][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 243, reason: Publication{term=1, version=243} [2023-03-06T01:12:16,426][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 244, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T01:12:20,470][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 244, reason: Publication{term=1, version=244} [2023-03-06T01:13:19,779][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][4]]]). [2023-03-06T02:07:53,665][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 316, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T02:07:53,774][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 316, reason: Publication{term=1, version=316} [2023-03-06T02:07:56,776][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 317, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T02:07:57,100][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 317, reason: Publication{term=1, version=317} [2023-03-06T02:07:58,738][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 318, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T02:08:02,630][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 318, reason: Publication{term=1, version=318} [2023-03-06T02:08:02,637][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.9s] (43 delayed shards) [2023-03-06T02:08:02,639][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 319, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T02:08:03,426][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 319, reason: Publication{term=1, version=319} [2023-03-06T02:08:03,470][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 320, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T02:08:03,741][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 320, reason: Publication{term=1, version=320} [2023-03-06T02:08:04,359][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 321, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T02:08:05,672][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 321, reason: Publication{term=1, version=321} [2023-03-06T02:08:53,799][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][0]]]). [2023-03-06T02:12:16,334][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 387, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T02:12:18,760][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 387, reason: Publication{term=1, version=387} [2023-03-06T02:12:18,765][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.3s] (44 delayed shards) [2023-03-06T02:12:19,774][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 388, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T02:12:22,464][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 388, reason: Publication{term=1, version=388} [2023-03-06T02:12:53,838][INFO ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][7537] overhead, spent [489ms] collecting in the last [1s] [2023-03-06T02:13:10,545][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][3]]]). [2023-03-06T03:07:56,729][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 462, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T03:07:57,139][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 462, reason: Publication{term=1, version=462} [2023-03-06T03:07:59,548][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 463, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T03:07:59,843][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 463, reason: Publication{term=1, version=463} [2023-03-06T03:08:04,446][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 464, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T03:08:08,940][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 464, reason: Publication{term=1, version=464} [2023-03-06T03:08:08,948][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.2s] (43 delayed shards) [2023-03-06T03:08:08,952][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 465, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T03:08:09,454][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 465, reason: Publication{term=1, version=465} [2023-03-06T03:08:09,535][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 466, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T03:08:09,750][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 466, reason: Publication{term=1, version=466} [2023-03-06T03:08:10,102][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 467, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T03:08:11,405][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 467, reason: Publication{term=1, version=467} [2023-03-06T03:08:52,891][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][2]]]). [2023-03-06T03:12:19,543][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 533, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T03:12:22,095][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 533, reason: Publication{term=1, version=533} [2023-03-06T03:12:22,100][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.4s] (44 delayed shards) [2023-03-06T03:12:22,859][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 534, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T03:12:28,048][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 534, reason: Publication{term=1, version=534} [2023-03-06T03:14:30,483][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][4]]]). [2023-03-06T04:07:59,517][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 606, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T04:07:59,921][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 606, reason: Publication{term=1, version=606} [2023-03-06T04:08:01,653][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 607, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T04:08:02,680][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 607, reason: Publication{term=1, version=607} [2023-03-06T04:08:10,034][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 608, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T04:08:20,037][INFO ][o.e.c.c.C.CoordinatorPublication] [onap-sdnrdb-master-1] after [10s] publication of cluster state version [608] is still waiting for {onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} [SENT_APPLY_COMMIT] [2023-03-06T04:08:21,033][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 608, reason: Publication{term=1, version=608} [2023-03-06T04:08:21,041][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [48.9s] (43 delayed shards) [2023-03-06T04:08:21,043][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 609, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T04:08:21,932][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 609, reason: Publication{term=1, version=609} [2023-03-06T04:08:22,135][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 610, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T04:08:22,637][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 610, reason: Publication{term=1, version=610} [2023-03-06T04:08:22,972][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 611, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T04:08:24,618][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 611, reason: Publication{term=1, version=611} [2023-03-06T04:09:05,688][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][3]]]). [2023-03-06T04:12:22,836][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 677, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T04:12:25,723][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 677, reason: Publication{term=1, version=677} [2023-03-06T04:12:25,727][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57s] (44 delayed shards) [2023-03-06T04:12:25,730][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 678, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T04:12:26,208][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 678, reason: Publication{term=1, version=678} [2023-03-06T04:12:26,354][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 679, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T04:12:26,534][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 679, reason: Publication{term=1, version=679} [2023-03-06T04:12:27,738][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 680, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T04:12:30,490][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 680, reason: Publication{term=1, version=680} [2023-03-06T04:13:21,026][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][4]]]). [2023-03-06T05:08:01,647][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 752, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T05:08:02,008][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 752, reason: Publication{term=1, version=752} [2023-03-06T05:08:04,586][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 753, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T05:08:05,336][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 753, reason: Publication{term=1, version=753} [2023-03-06T05:08:22,937][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 754, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T05:08:32,942][INFO ][o.e.c.c.C.CoordinatorPublication] [onap-sdnrdb-master-1] after [9.8s] publication of cluster state version [754] is still waiting for {onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} [SENT_APPLY_COMMIT] [2023-03-06T05:08:33,276][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 754, reason: Publication{term=1, version=754} [2023-03-06T05:08:33,286][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [49.5s] (43 delayed shards) [2023-03-06T05:08:33,288][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 755, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T05:08:34,167][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 755, reason: Publication{term=1, version=755} [2023-03-06T05:08:34,209][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 756, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T05:08:34,574][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 756, reason: Publication{term=1, version=756} [2023-03-06T05:08:34,579][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 757, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T05:08:35,288][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 757, reason: Publication{term=1, version=757} [2023-03-06T05:08:35,332][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 758, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T05:08:35,963][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 758, reason: Publication{term=1, version=758} [2023-03-06T05:08:36,343][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 759, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T05:08:37,869][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 759, reason: Publication{term=1, version=759} [2023-03-06T05:10:00,063][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][1]]]). [2023-03-06T05:12:27,740][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 826, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T05:12:29,761][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 826, reason: Publication{term=1, version=826} [2023-03-06T05:12:29,768][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.8s] (44 delayed shards) [2023-03-06T05:12:30,751][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 827, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T05:12:33,560][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 827, reason: Publication{term=1, version=827} [2023-03-06T05:13:24,282][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][3]]]). [2023-03-06T06:08:04,552][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 900, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T06:08:04,812][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 900, reason: Publication{term=1, version=900} [2023-03-06T06:08:07,046][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 901, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T06:08:07,438][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 901, reason: Publication{term=1, version=901} [2023-03-06T06:08:36,226][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 902, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:43,940][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 902, reason: Publication{term=1, version=902} [2023-03-06T06:08:43,946][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [52.2s] (43 delayed shards) [2023-03-06T06:08:43,950][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 903, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:45,124][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 903, reason: Publication{term=1, version=903} [2023-03-06T06:08:45,236][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 904, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:45,650][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 904, reason: Publication{term=1, version=904} [2023-03-06T06:08:45,657][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 905, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:46,170][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 905, reason: Publication{term=1, version=905} [2023-03-06T06:08:46,214][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 906, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:46,498][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 906, reason: Publication{term=1, version=906} [2023-03-06T06:08:46,503][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 907, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:46,668][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 907, reason: Publication{term=1, version=907} [2023-03-06T06:08:46,736][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 908, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:47,144][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 908, reason: Publication{term=1, version=908} [2023-03-06T06:08:47,315][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 909, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T06:08:48,776][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 909, reason: Publication{term=1, version=909} [2023-03-06T06:10:14,601][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][1]]]). [2023-03-06T06:12:30,668][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 974, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T06:12:33,780][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 974, reason: Publication{term=1, version=974} [2023-03-06T06:12:33,784][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [56.8s] (44 delayed shards) [2023-03-06T06:12:34,335][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 975, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T06:12:40,827][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 975, reason: Publication{term=1, version=975} [2023-03-06T06:13:56,739][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][3]]]). [2023-03-06T07:08:06,947][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 1045, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T07:08:07,162][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 1045, reason: Publication{term=1, version=1045} [2023-03-06T07:08:09,784][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 1046, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T07:08:10,589][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 1046, reason: Publication{term=1, version=1046} [2023-03-06T07:08:47,236][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1047, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T07:08:51,876][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1047, reason: Publication{term=1, version=1047} [2023-03-06T07:08:51,880][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.3s] (43 delayed shards) [2023-03-06T07:08:51,885][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1048, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T07:08:52,742][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1048, reason: Publication{term=1, version=1048} [2023-03-06T07:08:52,785][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1049, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T07:08:52,997][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1049, reason: Publication{term=1, version=1049} [2023-03-06T07:08:53,011][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1050, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T07:08:53,298][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1050, reason: Publication{term=1, version=1050} [2023-03-06T07:08:53,441][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1051, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T07:08:53,616][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1051, reason: Publication{term=1, version=1051} [2023-03-06T07:08:54,046][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1052, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T07:08:55,422][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1052, reason: Publication{term=1, version=1052} [2023-03-06T07:09:49,860][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][0]]]). [2023-03-06T07:12:34,223][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 1118, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T07:12:36,314][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 1118, reason: Publication{term=1, version=1118} [2023-03-06T07:12:36,318][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.8s] (44 delayed shards) [2023-03-06T07:12:37,645][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 1119, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T07:12:44,664][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 1119, reason: Publication{term=1, version=1119} [2023-03-06T07:14:10,217][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][4]]]). [2023-03-06T08:08:09,752][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 1189, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T08:08:09,978][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 1189, reason: Publication{term=1, version=1189} [2023-03-06T08:08:11,894][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 1190, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T08:08:12,464][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 1190, reason: Publication{term=1, version=1190} [2023-03-06T08:08:53,957][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1191, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T08:08:58,376][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1191, reason: Publication{term=1, version=1191} [2023-03-06T08:08:58,380][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [55.5s] (43 delayed shards) [2023-03-06T08:08:58,382][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1192, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T08:08:58,765][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1192, reason: Publication{term=1, version=1192} [2023-03-06T08:08:58,846][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1193, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T08:08:58,990][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1193, reason: Publication{term=1, version=1193} [2023-03-06T08:08:59,422][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1194, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T08:09:00,865][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1194, reason: Publication{term=1, version=1194} [2023-03-06T08:09:31,331][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][1]]]). [2023-03-06T08:12:37,435][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 1261, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T08:12:39,617][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 1261, reason: Publication{term=1, version=1261} [2023-03-06T08:12:39,621][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.7s] (44 delayed shards) [2023-03-06T08:12:40,647][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 1262, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T08:12:43,450][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 1262, reason: Publication{term=1, version=1262} [2023-03-06T08:13:47,549][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][4]]]). [2023-03-06T09:08:11,864][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} reason: disconnected], term: 1, version: 1333, delta: removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T09:08:12,686][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 1333, reason: Publication{term=1, version=1333} [2023-03-06T09:08:14,044][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r} join existing leader], term: 1, version: 1334, delta: added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}} [2023-03-06T09:08:14,813][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-coordinating-only-54c85fc557-gc727}{dGk5QfRmR9SwcAYt8VFlSg}{tfGOa8NrShe6qzOVRu2xBg}{10.233.67.197}{10.233.67.197:9300}{r}}, term: 1, version: 1334, reason: Publication{term=1, version=1334} [2023-03-06T09:08:59,274][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1335, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:09,337][INFO ][o.e.c.c.C.CoordinatorPublication] [onap-sdnrdb-master-1] after [9.8s] publication of cluster state version [1335] is still waiting for {onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} [SENT_APPLY_COMMIT] [2023-03-06T09:09:10,356][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1335, reason: Publication{term=1, version=1335} [2023-03-06T09:09:10,363][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [48.9s] (43 delayed shards) [2023-03-06T09:09:10,366][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1336, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:10,804][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1336, reason: Publication{term=1, version=1336} [2023-03-06T09:09:10,855][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1337, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:11,314][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1337, reason: Publication{term=1, version=1337} [2023-03-06T09:09:11,320][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1338, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:11,864][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1338, reason: Publication{term=1, version=1338} [2023-03-06T09:09:11,902][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1339, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:12,435][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1339, reason: Publication{term=1, version=1339} [2023-03-06T09:09:12,439][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1340, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:13,215][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1340, reason: Publication{term=1, version=1340} [2023-03-06T09:09:13,251][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} reason: disconnected], term: 1, version: 1341, delta: removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:13,531][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1341, reason: Publication{term=1, version=1341} [2023-03-06T09:09:14,267][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr} join existing leader], term: 1, version: 1342, delta: added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}} [2023-03-06T09:09:15,804][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-2}{u8g8HRa2Qkq4RSWsx9v72g}{noSzVmKUTx-AKCOCIKtAWA}{10.233.65.81}{10.233.65.81:9300}{dmr}}, term: 1, version: 1342, reason: Publication{term=1, version=1342} [2023-03-06T09:10:03,450][WARN ][o.e.m.j.JvmGcMonitorService] [onap-sdnrdb-master-1] [gc][32550] overhead, spent [656ms] collecting in the last [1.3s] [2023-03-06T09:10:22,764][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][2]]]). [2023-03-06T09:12:40,481][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-left[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} reason: disconnected], term: 1, version: 1409, delta: removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T09:12:43,065][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] removed {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 1409, reason: Publication{term=1, version=1409} [2023-03-06T09:12:43,070][INFO ][o.e.c.r.DelayedAllocationService] [onap-sdnrdb-master-1] scheduling reroute for delayed shards in [57.4s] (44 delayed shards) [2023-03-06T09:12:43,072][WARN ][o.e.c.c.Coordinator ] [onap-sdnrdb-master-1] failed to validate incoming join request from node [{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}] org.elasticsearch.transport.NodeDisconnectedException: [onap-sdnrdb-master-0][10.233.71.51:9300][internal:cluster/coordination/join/validate] disconnected [2023-03-06T09:12:44,113][INFO ][o.e.c.s.MasterService ] [onap-sdnrdb-master-1] node-join[{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr} join existing leader], term: 1, version: 1410, delta: added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}} [2023-03-06T09:12:46,169][INFO ][o.e.c.s.ClusterApplierService] [onap-sdnrdb-master-1] added {{onap-sdnrdb-master-0}{_U3t-IjgSBqjLA2q7ErHjQ}{-iP5R87pTjGRB7o6Ffxr9A}{10.233.71.51}{10.233.71.51:9300}{dmr}}, term: 1, version: 1410, reason: Publication{term=1, version=1410} [2023-03-06T09:13:27,809][INFO ][o.e.c.r.a.AllocationService] [onap-sdnrdb-master-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[faultcurrent-v7][3]]]).