By type
[38;5;6m [38;5;5m22:51:48.50 [0m
[38;5;6m [38;5;5m22:51:48.58 [0m[1mWelcome to the Bitnami elasticsearch container[0m
[38;5;6m [38;5;5m22:51:48.59 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m
[38;5;6m [38;5;5m22:51:48.60 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m
[38;5;6m [38;5;5m22:51:48.68 [0m
[38;5;6m [38;5;5m22:51:48.69 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup **
[38;5;6m [38;5;5m22:51:49.18 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch...
[38;5;6m [38;5;5m22:51:49.68 [0m[38;5;2mINFO [0m ==> Setting default configuration
[38;5;6m [38;5;5m22:51:49.80 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings...
[38;5;6m [38;5;5m22:51:50.08 [0m[38;5;3mWARN [0m ==> Found more than one IP address associated to hostname dev-sdnrdb-master-2: fd00:100::24d9 10.242.36.217, will use fd00:100::24d9
[38;5;6m [38;5;5m22:51:50.38 [0m[38;5;3mWARN [0m ==> Found more than one IP address associated to hostname dev-sdnrdb-master-2: fd00:100::24d9 10.242.36.217, will use fd00:100::24d9
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[38;5;6m [38;5;5m22:52:11.59 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! **
[38;5;6m [38;5;5m22:52:11.80 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2021-04-25T22:52:52,682][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/4.15.0-117-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS]
[2021-04-25T22:52:52,684][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] JVM home [/opt/bitnami/java]
[2021-04-25T22:52:52,685][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-5278481639166648357, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-04-25T22:53:09,688][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [aggs-matrix-stats]
[2021-04-25T22:53:09,689][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [analysis-common]
[2021-04-25T22:53:09,689][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [geo]
[2021-04-25T22:53:09,690][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [ingest-common]
[2021-04-25T22:53:09,690][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [ingest-geoip]
[2021-04-25T22:53:09,691][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [ingest-user-agent]
[2021-04-25T22:53:09,691][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [kibana]
[2021-04-25T22:53:09,691][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [lang-expression]
[2021-04-25T22:53:09,692][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [lang-mustache]
[2021-04-25T22:53:09,692][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [lang-painless]
[2021-04-25T22:53:09,692][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [mapper-extras]
[2021-04-25T22:53:09,693][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [parent-join]
[2021-04-25T22:53:09,693][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [percolator]
[2021-04-25T22:53:09,693][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [rank-eval]
[2021-04-25T22:53:09,778][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [reindex]
[2021-04-25T22:53:09,779][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [repository-url]
[2021-04-25T22:53:09,779][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [tasks]
[2021-04-25T22:53:09,779][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded module [transport-netty4]
[2021-04-25T22:53:09,780][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-master-2] loaded plugin [repository-s3]
[2021-04-25T22:53:10,394][INFO ][o.e.e.NodeEnvironment ] [dev-sdnrdb-master-2] using [1] data paths, mounts [[/bitnami/elasticsearch/data (172.16.10.118:/dockerdata-nfs/dev/elastic-master-2)]], net usable_space [178.9gb], net total_space [195.8gb], types [nfs4]
[2021-04-25T22:53:10,478][INFO ][o.e.e.NodeEnvironment ] [dev-sdnrdb-master-2] heap size [123.7mb], compressed ordinary object pointers [true]
[2021-04-25T22:53:11,085][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] node name [dev-sdnrdb-master-2], node ID [b2WbhbUSQN2vhY_7wRUxuA], cluster name [sdnrdb-cluster]
[2021-04-25T22:53:57,887][INFO ][o.e.t.NettyAllocator ] [dev-sdnrdb-master-2] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}]
[2021-04-25T22:53:58,982][INFO ][o.e.d.DiscoveryModule ] [dev-sdnrdb-master-2] using discovery type [zen] and seed hosts providers [settings]
[2021-04-25T22:54:03,185][WARN ][o.e.g.DanglingIndicesState] [dev-sdnrdb-master-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-04-25T22:54:05,690][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] initialized
[2021-04-25T22:54:05,691][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] starting ...
[2021-04-25T22:54:07,484][INFO ][o.e.t.TransportService ] [dev-sdnrdb-master-2] publish_address {[fd00:100::24d9]:9300}, bound_addresses {[::]:9300}
[2021-04-25T22:54:11,187][INFO ][o.e.b.BootstrapChecks ] [dev-sdnrdb-master-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-04-25T22:54:14,580][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-master-2] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-dev-sdnrdb-master-1,Ze-vtMQ9QYeaWYqHv_0q7Q,b2WbhbUSQN2vhY_7wRUxuA}
[2021-04-25T22:54:18,996][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] master node changed {previous [], current [{dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}]}, added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr},{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r},{dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}, term: 4, version: 61, reason: ApplyCommitRequest{term=4, version=61, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T22:54:19,407][INFO ][o.e.c.s.ClusterSettings ] [dev-sdnrdb-master-2] updating [action.auto_create_index] from [true] to [false]
[2021-04-25T22:54:20,296][INFO ][o.e.h.AbstractHttpServerTransport] [dev-sdnrdb-master-2] publish_address {[fd00:100::24d9]:9200}, bound_addresses {[::]:9200}
[2021-04-25T22:54:20,296][INFO ][o.e.n.Node ] [dev-sdnrdb-master-2] started
[2021-04-25T23:07:50,794][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 123, reason: ApplyCommitRequest{term=4, version=123, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:07:51,090][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,180][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,190][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,298][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [maintenancemode-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,379][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [maintenancemode-v5][2] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,403][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][2] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,584][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,589][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,694][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,702][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,878][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][2] primary-replica resync completed with 0 operations
[2021-04-25T23:07:51,983][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:07:52,078][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:09:14,011][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 147, reason: ApplyCommitRequest{term=4, version=147, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:10:14,190][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 149, reason: ApplyCommitRequest{term=4, version=149, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:10:32,611][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 174, reason: ApplyCommitRequest{term=4, version=174, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:11:31,116][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 176, reason: ApplyCommitRequest{term=4, version=176, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:11:31,418][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 177, reason: ApplyCommitRequest{term=4, version=177, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:11:31,810][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 178, reason: ApplyCommitRequest{term=4, version=178, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:11:35,623][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 182, reason: ApplyCommitRequest{term=4, version=182, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:13:07,006][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-cdwcb}{nd_ZdWVISA-ASGCoxLpUYA}{gkNhrWK4Sais9PJMlMNkUg}{fd00:100:0:0:0:0:0:206e}{[fd00:100::206e]:9300}{r}}, term: 4, version: 228, reason: ApplyCommitRequest{term=4, version=228, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:18:53,287][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 247, reason: ApplyCommitRequest{term=4, version=247, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:18:53,490][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][1] primary-replica resync completed with 0 operations
[2021-04-25T23:18:53,587][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:18:53,678][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:18:53,695][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:18:53,879][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:18:53,981][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:20:34,191][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 276, reason: ApplyCommitRequest{term=4, version=276, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:21:04,285][WARN ][o.e.c.NodeConnectionsService] [dev-sdnrdb-master-2] failed to connect to {dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [dev-sdnrdb-master-1][[fd00:100::5daa]:9300] handshake_timeout[30s]
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:73) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-04-25T23:21:04,480][WARN ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] cluster state applier task [ApplyCommitRequest{term=4, version=276, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}] took [30.2s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=4, version=276, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}]] took [0ms], [connecting to new nodes] took [30194ms], [applying settings] took [0ms], [running applier [org.elasticsearch.repositories.RepositoriesService@3b478a6]] took [0ms], [running applier [org.elasticsearch.indices.cluster.IndicesClusterStateService@260e1292]] took [90ms], [running applier [org.elasticsearch.script.ScriptService@5a1c61b]] took [0ms], [running applier [org.elasticsearch.snapshots.RestoreService@4da49111]] took [0ms], [running applier [org.elasticsearch.ingest.IngestService@7b5fcbf2]] took [0ms], [running applier [org.elasticsearch.action.ingest.IngestActionForwarder@37efcdf]] took [0ms], [running applier [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2399/0x0000000100929c40@6a3bd9b8]] took [0ms], [running applier [org.elasticsearch.tasks.TaskManager@6f3e9f56]] took [0ms], [running applier [org.elasticsearch.snapshots.SnapshotsService@44672225]] took [0ms], [notifying listener [org.elasticsearch.cluster.InternalClusterInfoService@629dcbef]] took [0ms], [notifying listener [org.elasticsearch.cluster.metadata.TemplateUpgradeService@fc8d9e8]] took [0ms], [notifying listener [org.elasticsearch.node.ResponseCollectorService@4131209c]] took [0ms], [notifying listener [org.elasticsearch.snapshots.SnapshotShardsService@6bac3bac]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksClusterService@67e25785]] took [0ms], [notifying listener [org.elasticsearch.cluster.routing.DelayedAllocationService@2eacc508]] took [0ms], [notifying listener [org.elasticsearch.indices.store.IndicesStore@66513581]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksNodeService@12bbfc51]] took [0ms], [notifying listener [org.elasticsearch.gateway.GatewayService@2aa58535]] took [0ms], [notifying listener [org.elasticsearch.indices.recovery.PeerRecoverySourceService@6b845fa5]] took [0ms], [notifying listener [org.elasticsearch.cluster.service.ClusterApplierService$LocalNodeMasterListeners@4cd4f9e6]] took [0ms]
[2021-04-25T23:21:34,430][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 278, reason: ApplyCommitRequest{term=4, version=278, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:25:01,813][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 310, reason: ApplyCommitRequest{term=4, version=310, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:25:31,823][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 311, reason: ApplyCommitRequest{term=4, version=311, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:27:04,431][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 312, reason: ApplyCommitRequest{term=4, version=312, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:33:51,506][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 375, reason: ApplyCommitRequest{term=4, version=375, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:33:51,682][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:33:51,691][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:33:51,779][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][1] primary-replica resync completed with 0 operations
[2021-04-25T23:33:51,794][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:33:51,891][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][1] primary-replica resync completed with 0 operations
[2021-04-25T23:33:51,978][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-25T23:33:52,080][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][2] primary-replica resync completed with 0 operations
[2021-04-25T23:33:52,101][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-25T23:33:52,182][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][4] primary-replica resync completed with 0 operations
[2021-04-25T23:33:52,285][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][1] primary-replica resync completed with 0 operations
[2021-04-25T23:33:52,295][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [maintenancemode-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:33:52,392][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [maintenancemode-v5][1] primary-replica resync completed with 0 operations
[2021-04-25T23:37:19,130][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 435, reason: ApplyCommitRequest{term=4, version=435, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:38:19,113][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 437, reason: ApplyCommitRequest{term=4, version=437, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:38:55,059][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 438, reason: ApplyCommitRequest{term=4, version=438, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:39:08,479][INFO ][o.e.m.j.JvmGcMonitorService] [dev-sdnrdb-master-2] [gc][2697] overhead, spent [675ms] collecting in the last [1.4s]
[2021-04-25T23:44:34,770][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 443, reason: ApplyCommitRequest{term=4, version=443, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:44:34,878][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:47:22,193][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 449, reason: ApplyCommitRequest{term=4, version=449, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:48:22,231][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 451, reason: ApplyCommitRequest{term=4, version=451, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:49:33,798][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 452, reason: ApplyCommitRequest{term=4, version=452, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:53:46,060][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 457, reason: ApplyCommitRequest{term=4, version=457, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:53:46,181][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][3] primary-replica resync completed with 0 operations
[2021-04-25T23:58:19,024][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 461, reason: ApplyCommitRequest{term=4, version=461, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-25T23:58:49,127][WARN ][o.e.c.NodeConnectionsService] [dev-sdnrdb-master-2] failed to connect to {dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr} (tried [1] times)
org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-1][[fd00:100::5daa]:9300][internal:transport/handshake] request_id [9658] timed out after [30036ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-04-25T23:58:49,134][WARN ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] cluster state applier task [ApplyCommitRequest{term=4, version=461, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}] took [30.2s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=4, version=461, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}]] took [0ms], [connecting to new nodes] took [30105ms], [applying settings] took [0ms], [running applier [org.elasticsearch.repositories.RepositoriesService@3b478a6]] took [0ms], [running applier [org.elasticsearch.indices.cluster.IndicesClusterStateService@260e1292]] took [1ms], [running applier [org.elasticsearch.script.ScriptService@5a1c61b]] took [0ms], [running applier [org.elasticsearch.snapshots.RestoreService@4da49111]] took [0ms], [running applier [org.elasticsearch.ingest.IngestService@7b5fcbf2]] took [0ms], [running applier [org.elasticsearch.action.ingest.IngestActionForwarder@37efcdf]] took [0ms], [running applier [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2399/0x0000000100929c40@6a3bd9b8]] took [0ms], [running applier [org.elasticsearch.tasks.TaskManager@6f3e9f56]] took [0ms], [running applier [org.elasticsearch.snapshots.SnapshotsService@44672225]] took [0ms], [notifying listener [org.elasticsearch.cluster.InternalClusterInfoService@629dcbef]] took [0ms], [notifying listener [org.elasticsearch.cluster.metadata.TemplateUpgradeService@fc8d9e8]] took [0ms], [notifying listener [org.elasticsearch.node.ResponseCollectorService@4131209c]] took [0ms], [notifying listener [org.elasticsearch.snapshots.SnapshotShardsService@6bac3bac]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksClusterService@67e25785]] took [0ms], [notifying listener [org.elasticsearch.cluster.routing.DelayedAllocationService@2eacc508]] took [0ms], [notifying listener [org.elasticsearch.indices.store.IndicesStore@66513581]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksNodeService@12bbfc51]] took [0ms], [notifying listener [org.elasticsearch.gateway.GatewayService@2aa58535]] took [0ms], [notifying listener [org.elasticsearch.indices.recovery.PeerRecoverySourceService@6b845fa5]] took [0ms], [notifying listener [org.elasticsearch.cluster.service.ClusterApplierService$LocalNodeMasterListeners@4cd4f9e6]] took [0ms]
[2021-04-25T23:59:19,104][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 463, reason: ApplyCommitRequest{term=4, version=463, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:01:07,899][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 464, reason: ApplyCommitRequest{term=4, version=464, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:01:38,500][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 465, reason: ApplyCommitRequest{term=4, version=465, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:02:17,451][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 466, reason: ApplyCommitRequest{term=4, version=466, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:03:27,405][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 468, reason: ApplyCommitRequest{term=4, version=468, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:05:13,192][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 469, reason: ApplyCommitRequest{term=4, version=469, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:10:21,514][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 473, reason: ApplyCommitRequest{term=4, version=473, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:10:21,615][WARN ][o.e.i.s.RetentionLeaseSyncAction] [dev-sdnrdb-master-2] [[inventoryequipment-v5][3]] failed to perform indices:admin/seq_no/retention_lease_sync on replica [inventoryequipment-v5][3], node[IeNHB3AjSESvvbc3slnMCg], [R], s[STARTED], a[id=DEwArrqZRHS7_IFNPwocHg]
org.elasticsearch.client.transport.NoNodeAvailableException: unknown node [IeNHB3AjSESvvbc3slnMCg]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1084) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:244) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:99) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Suppressed: org.elasticsearch.transport.NodeDisconnectedException: [dev-sdnrdb-master-1][[fd00:100::5daa]:9300][indices:admin/seq_no/retention_lease_sync[r]] disconnected
[2021-04-26T00:11:41,467][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 477, reason: ApplyCommitRequest{term=4, version=477, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:12:41,598][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 479, reason: ApplyCommitRequest{term=4, version=479, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:17:47,941][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 480, reason: ApplyCommitRequest{term=4, version=480, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:18:22,961][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 481, reason: ApplyCommitRequest{term=4, version=481, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:18:53,030][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 482, reason: ApplyCommitRequest{term=4, version=482, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:24:48,407][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 487, reason: ApplyCommitRequest{term=4, version=487, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:24:48,507][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T00:25:29,177][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 491, reason: ApplyCommitRequest{term=4, version=491, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:25:42,704][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 492, reason: ApplyCommitRequest{term=4, version=492, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:28:04,729][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 493, reason: ApplyCommitRequest{term=4, version=493, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:29:04,999][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 495, reason: ApplyCommitRequest{term=4, version=495, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:33:38,931][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 496, reason: ApplyCommitRequest{term=4, version=496, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:34:58,400][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 497, reason: ApplyCommitRequest{term=4, version=497, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:36:03,825][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 498, reason: ApplyCommitRequest{term=4, version=498, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:36:47,790][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 499, reason: ApplyCommitRequest{term=4, version=499, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:43:26,554][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 500, reason: ApplyCommitRequest{term=4, version=500, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:45:29,121][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 502, reason: ApplyCommitRequest{term=4, version=502, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T00:56:49,795][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 503, reason: ApplyCommitRequest{term=4, version=503, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:04:30,281][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 504, reason: ApplyCommitRequest{term=4, version=504, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:04:31,872][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 505, reason: ApplyCommitRequest{term=4, version=505, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:09:56,207][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 568, reason: ApplyCommitRequest{term=4, version=568, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:09:56,397][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,503][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,579][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,598][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,691][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,700][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,794][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,798][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,805][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,884][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,981][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:09:56,994][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:11:29,907][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 622, reason: ApplyCommitRequest{term=4, version=622, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:12:29,993][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 624, reason: ApplyCommitRequest{term=4, version=624, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:19:45,082][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 629, reason: ApplyCommitRequest{term=4, version=629, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:20:44,923][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 631, reason: ApplyCommitRequest{term=4, version=631, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:21:15,086][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 632, reason: ApplyCommitRequest{term=4, version=632, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:21:40,018][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 633, reason: ApplyCommitRequest{term=4, version=633, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:26:22,406][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 634, reason: ApplyCommitRequest{term=4, version=634, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:27:33,925][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 635, reason: ApplyCommitRequest{term=4, version=635, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:28:31,026][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 636, reason: ApplyCommitRequest{term=4, version=636, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:29:32,310][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 638, reason: ApplyCommitRequest{term=4, version=638, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:34:39,903][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 639, reason: ApplyCommitRequest{term=4, version=639, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:44:51,610][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 701, reason: ApplyCommitRequest{term=4, version=701, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:44:51,778][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:44:51,785][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][1] primary-replica resync completed with 0 operations
[2021-04-26T01:44:51,802][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:44:51,888][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:44:51,894][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T01:44:51,979][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:44:51,997][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:44:52,011][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:44:52,020][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T01:44:52,089][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T01:44:52,181][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:44:52,190][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [maintenancemode-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T01:47:20,696][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 758, reason: ApplyCommitRequest{term=4, version=758, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:47:38,440][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 761, reason: ApplyCommitRequest{term=4, version=761, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:48:08,487][WARN ][o.e.c.NodeConnectionsService] [dev-sdnrdb-master-2] failed to connect to {dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd][[fd00:100::5da8]:9300] handshake_timeout[30s]
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:73) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-04-26T01:48:08,499][WARN ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] cluster state applier task [ApplyCommitRequest{term=4, version=761, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}] took [30s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=4, version=761, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}]] took [0ms], [connecting to new nodes] took [30055ms], [applying settings] took [0ms], [running applier [org.elasticsearch.repositories.RepositoriesService@3b478a6]] took [0ms], [running applier [org.elasticsearch.indices.cluster.IndicesClusterStateService@260e1292]] took [1ms], [running applier [org.elasticsearch.script.ScriptService@5a1c61b]] took [0ms], [running applier [org.elasticsearch.snapshots.RestoreService@4da49111]] took [0ms], [running applier [org.elasticsearch.ingest.IngestService@7b5fcbf2]] took [0ms], [running applier [org.elasticsearch.action.ingest.IngestActionForwarder@37efcdf]] took [0ms], [running applier [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2399/0x0000000100929c40@6a3bd9b8]] took [0ms], [running applier [org.elasticsearch.tasks.TaskManager@6f3e9f56]] took [0ms], [running applier [org.elasticsearch.snapshots.SnapshotsService@44672225]] took [0ms], [notifying listener [org.elasticsearch.cluster.InternalClusterInfoService@629dcbef]] took [0ms], [notifying listener [org.elasticsearch.cluster.metadata.TemplateUpgradeService@fc8d9e8]] took [0ms], [notifying listener [org.elasticsearch.node.ResponseCollectorService@4131209c]] took [0ms], [notifying listener [org.elasticsearch.snapshots.SnapshotShardsService@6bac3bac]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksClusterService@67e25785]] took [0ms], [notifying listener [org.elasticsearch.cluster.routing.DelayedAllocationService@2eacc508]] took [0ms], [notifying listener [org.elasticsearch.indices.store.IndicesStore@66513581]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksNodeService@12bbfc51]] took [0ms], [notifying listener [org.elasticsearch.gateway.GatewayService@2aa58535]] took [0ms], [notifying listener [org.elasticsearch.indices.recovery.PeerRecoverySourceService@6b845fa5]] took [0ms], [notifying listener [org.elasticsearch.cluster.service.ClusterApplierService$LocalNodeMasterListeners@4cd4f9e6]] took [0ms]
[2021-04-26T01:48:10,434][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 762, reason: ApplyCommitRequest{term=4, version=762, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:48:39,569][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 763, reason: ApplyCommitRequest{term=4, version=763, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:52:15,429][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 764, reason: ApplyCommitRequest{term=4, version=764, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:53:38,170][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 765, reason: ApplyCommitRequest{term=4, version=765, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:55:03,199][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 766, reason: ApplyCommitRequest{term=4, version=766, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T01:56:03,303][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 768, reason: ApplyCommitRequest{term=4, version=768, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:09:18,154][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 769, reason: ApplyCommitRequest{term=4, version=769, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:09:48,184][WARN ][o.e.c.NodeConnectionsService] [dev-sdnrdb-master-2] failed to connect to {dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [dev-sdnrdb-master-1][[fd00:100::5daa]:9300] handshake_timeout[30s]
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:73) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-04-26T02:09:48,192][WARN ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] cluster state applier task [ApplyCommitRequest{term=4, version=769, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}] took [30s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=4, version=769, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}]] took [0ms], [connecting to new nodes] took [30022ms], [applying settings] took [0ms], [running applier [org.elasticsearch.repositories.RepositoriesService@3b478a6]] took [0ms], [running applier [org.elasticsearch.indices.cluster.IndicesClusterStateService@260e1292]] took [1ms], [running applier [org.elasticsearch.script.ScriptService@5a1c61b]] took [0ms], [running applier [org.elasticsearch.snapshots.RestoreService@4da49111]] took [0ms], [running applier [org.elasticsearch.ingest.IngestService@7b5fcbf2]] took [0ms], [running applier [org.elasticsearch.action.ingest.IngestActionForwarder@37efcdf]] took [0ms], [running applier [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2399/0x0000000100929c40@6a3bd9b8]] took [0ms], [running applier [org.elasticsearch.tasks.TaskManager@6f3e9f56]] took [0ms], [running applier [org.elasticsearch.snapshots.SnapshotsService@44672225]] took [0ms], [notifying listener [org.elasticsearch.cluster.InternalClusterInfoService@629dcbef]] took [0ms], [notifying listener [org.elasticsearch.cluster.metadata.TemplateUpgradeService@fc8d9e8]] took [0ms], [notifying listener [org.elasticsearch.node.ResponseCollectorService@4131209c]] took [0ms], [notifying listener [org.elasticsearch.snapshots.SnapshotShardsService@6bac3bac]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksClusterService@67e25785]] took [0ms], [notifying listener [org.elasticsearch.cluster.routing.DelayedAllocationService@2eacc508]] took [0ms], [notifying listener [org.elasticsearch.indices.store.IndicesStore@66513581]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksNodeService@12bbfc51]] took [0ms], [notifying listener [org.elasticsearch.gateway.GatewayService@2aa58535]] took [0ms], [notifying listener [org.elasticsearch.indices.recovery.PeerRecoverySourceService@6b845fa5]] took [0ms], [notifying listener [org.elasticsearch.cluster.service.ClusterApplierService$LocalNodeMasterListeners@4cd4f9e6]] took [0ms]
[2021-04-26T02:10:18,298][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 771, reason: ApplyCommitRequest{term=4, version=771, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:10:51,364][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 772, reason: ApplyCommitRequest{term=4, version=772, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:12:04,596][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 773, reason: ApplyCommitRequest{term=4, version=773, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:20:02,595][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 774, reason: ApplyCommitRequest{term=4, version=774, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:23:02,800][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 836, reason: ApplyCommitRequest{term=4, version=836, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:23:02,888][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T02:23:02,984][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T02:23:02,990][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T02:23:03,091][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T02:23:03,101][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T02:23:03,188][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T02:23:03,282][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][1] primary-replica resync completed with 0 operations
[2021-04-26T02:23:03,293][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T02:24:48,914][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 839, reason: ApplyCommitRequest{term=4, version=839, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:24:52,699][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr},{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 848, reason: ApplyCommitRequest{term=4, version=848, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:28:09,093][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 909, reason: ApplyCommitRequest{term=4, version=909, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:34:14,175][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 966, reason: ApplyCommitRequest{term=4, version=966, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:35:14,224][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 968, reason: ApplyCommitRequest{term=4, version=968, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:43:23,127][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 969, reason: ApplyCommitRequest{term=4, version=969, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:43:53,183][WARN ][o.e.c.NodeConnectionsService] [dev-sdnrdb-master-2] failed to connect to {dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [dev-sdnrdb-master-1][[fd00:100::5daa]:9300] handshake_timeout[30s]
at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:73) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-04-26T02:43:53,287][WARN ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] cluster state applier task [ApplyCommitRequest{term=4, version=969, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}] took [30s] which is above the warn threshold of [30s]: [running task [ApplyCommitRequest{term=4, version=969, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}]] took [0ms], [connecting to new nodes] took [30155ms], [applying settings] took [0ms], [running applier [org.elasticsearch.repositories.RepositoriesService@3b478a6]] took [0ms], [running applier [org.elasticsearch.indices.cluster.IndicesClusterStateService@260e1292]] took [0ms], [running applier [org.elasticsearch.script.ScriptService@5a1c61b]] took [0ms], [running applier [org.elasticsearch.snapshots.RestoreService@4da49111]] took [0ms], [running applier [org.elasticsearch.ingest.IngestService@7b5fcbf2]] took [0ms], [running applier [org.elasticsearch.action.ingest.IngestActionForwarder@37efcdf]] took [0ms], [running applier [org.elasticsearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$$Lambda$2399/0x0000000100929c40@6a3bd9b8]] took [0ms], [running applier [org.elasticsearch.tasks.TaskManager@6f3e9f56]] took [0ms], [running applier [org.elasticsearch.snapshots.SnapshotsService@44672225]] took [0ms], [notifying listener [org.elasticsearch.cluster.InternalClusterInfoService@629dcbef]] took [0ms], [notifying listener [org.elasticsearch.cluster.metadata.TemplateUpgradeService@fc8d9e8]] took [0ms], [notifying listener [org.elasticsearch.node.ResponseCollectorService@4131209c]] took [0ms], [notifying listener [org.elasticsearch.snapshots.SnapshotShardsService@6bac3bac]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksClusterService@67e25785]] took [0ms], [notifying listener [org.elasticsearch.cluster.routing.DelayedAllocationService@2eacc508]] took [0ms], [notifying listener [org.elasticsearch.indices.store.IndicesStore@66513581]] took [0ms], [notifying listener [org.elasticsearch.persistent.PersistentTasksNodeService@12bbfc51]] took [0ms], [notifying listener [org.elasticsearch.gateway.GatewayService@2aa58535]] took [0ms], [notifying listener [org.elasticsearch.indices.recovery.PeerRecoverySourceService@6b845fa5]] took [0ms], [notifying listener [org.elasticsearch.cluster.service.ClusterApplierService$LocalNodeMasterListeners@4cd4f9e6]] took [0ms]
[2021-04-26T02:44:23,120][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 971, reason: ApplyCommitRequest{term=4, version=971, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:45:20,874][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 972, reason: ApplyCommitRequest{term=4, version=972, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:46:38,775][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 973, reason: ApplyCommitRequest{term=4, version=973, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T02:56:27,486][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 974, reason: ApplyCommitRequest{term=4, version=974, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:04:21,102][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 975, reason: ApplyCommitRequest{term=4, version=975, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:04:21,318][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 976, reason: ApplyCommitRequest{term=4, version=976, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:24:18,469][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1039, reason: ApplyCommitRequest{term=4, version=1039, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:24:18,587][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T03:24:18,622][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][1] primary-replica resync completed with 0 operations
[2021-04-26T03:24:18,688][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:24:18,779][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:24:18,881][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T03:24:18,884][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:24:19,094][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:24:19,109][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][1] primary-replica resync completed with 0 operations
[2021-04-26T03:25:30,428][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1060, reason: ApplyCommitRequest{term=4, version=1060, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:26:30,428][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1062, reason: ApplyCommitRequest{term=4, version=1062, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:27:35,795][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1101, reason: ApplyCommitRequest{term=4, version=1101, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:39:52,215][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1165, reason: ApplyCommitRequest{term=4, version=1165, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:39:52,286][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,295][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,385][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,401][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,489][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,579][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,589][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:39:52,606][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:40:43,291][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1166, reason: ApplyCommitRequest{term=4, version=1166, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:43:42,851][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1169, reason: ApplyCommitRequest{term=4, version=1169, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:44:08,682][INFO ][o.e.m.j.JvmGcMonitorService] [dev-sdnrdb-master-2] [gc][17386] overhead, spent [300ms] collecting in the last [1s]
[2021-04-26T03:45:42,484][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1228, reason: ApplyCommitRequest{term=4, version=1228, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:46:42,537][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1230, reason: ApplyCommitRequest{term=4, version=1230, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:47:53,324][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1231, reason: ApplyCommitRequest{term=4, version=1231, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:55:04,264][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1297, reason: ApplyCommitRequest{term=4, version=1297, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T03:55:04,628][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,678][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,681][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][1] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,683][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,686][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,778][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,779][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,780][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,780][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultlog-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,781][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [guicutthrough-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T03:55:04,782][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T04:05:26,496][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1359, reason: ApplyCommitRequest{term=4, version=1359, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:11:55,005][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1360, reason: ApplyCommitRequest{term=4, version=1360, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:15:58,001][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1361, reason: ApplyCommitRequest{term=4, version=1361, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:18:56,948][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1362, reason: ApplyCommitRequest{term=4, version=1362, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:26:52,096][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1363, reason: ApplyCommitRequest{term=4, version=1363, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:31:55,605][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1364, reason: ApplyCommitRequest{term=4, version=1364, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:33:10,299][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1365, reason: ApplyCommitRequest{term=4, version=1365, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:39:03,092][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1366, reason: ApplyCommitRequest{term=4, version=1366, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T04:54:38,903][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1367, reason: ApplyCommitRequest{term=4, version=1367, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:09:58,495][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-coordinating-only-65cbf8cbbd-kdkwd}{W-ycVbsFSIitXRFv1MN0Dw}{3n80UzgWT5q-62u9eaxn8w}{fd00:100:0:0:0:0:0:5da8}{[fd00:100::5da8]:9300}{r}}, term: 4, version: 1368, reason: ApplyCommitRequest{term=4, version=1368, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:09:59,847][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1369, reason: ApplyCommitRequest{term=4, version=1369, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:14:44,290][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1432, reason: ApplyCommitRequest{term=4, version=1432, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:14:44,393][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,478][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [networkelement-connection-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,495][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [mediator-server-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,579][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance24h-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,591][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [connectionlog-v5][4] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,604][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [maintenancemode-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,679][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [inventoryequipment-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,692][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [faultcurrent-v5][2] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,798][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [historicalperformance15min-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T05:14:44,815][INFO ][o.e.i.s.IndexShard ] [dev-sdnrdb-master-2] [eventlog-v5][3] primary-replica resync completed with 0 operations
[2021-04-26T05:16:15,553][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1492, reason: ApplyCommitRequest{term=4, version=1492, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:17:15,586][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1494, reason: ApplyCommitRequest{term=4, version=1494, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:17:21,375][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1495, reason: ApplyCommitRequest{term=4, version=1495, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:19:38,994][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] removed {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1497, reason: ApplyCommitRequest{term=4, version=1497, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}
[2021-04-26T05:23:17,237][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-master-2] added {{dev-sdnrdb-master-1}{IeNHB3AjSESvvbc3slnMCg}{ivncIJVeShmgo2v0UlU9Sg}{fd00:100:0:0:0:0:0:5daa}{[fd00:100::5daa]:9300}{dmr}}, term: 4, version: 1498, reason: ApplyCommitRequest{term=4, version=1498, sourceNode={dev-sdnrdb-master-0}{Ze-vtMQ9QYeaWYqHv_0q7Q}{F4c8Q1WqRqyCd3jpUssufw}{fd00:100:0:0:0:0:0:9161}{[fd00:100::9161]:9300}{dmr}}