By type
[38;5;6m [38;5;5m22:51:56.09 [0m [38;5;6m [38;5;5m22:51:56.10 [0m[1mWelcome to the Bitnami elasticsearch container[0m [38;5;6m [38;5;5m22:51:56.11 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch[0m [38;5;6m [38;5;5m22:51:56.12 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-elasticsearch/issues[0m [38;5;6m [38;5;5m22:51:56.13 [0m [38;5;6m [38;5;5m22:51:56.14 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch setup ** [38;5;6m [38;5;5m22:51:56.24 [0m[38;5;2mINFO [0m ==> Configuring/Initializing Elasticsearch... [38;5;6m [38;5;5m22:51:56.34 [0m[38;5;2mINFO [0m ==> Setting default configuration [38;5;6m [38;5;5m22:51:56.37 [0m[38;5;2mINFO [0m ==> Configuring Elasticsearch cluster settings... [38;5;6m [38;5;5m22:51:56.41 [0m[38;5;3mWARN [0m ==> Found more than one IP address associated to hostname dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5: fd00:100::66de 10.242.102.222, will use fd00:100::66de [38;5;6m [38;5;5m22:51:56.44 [0m[38;5;3mWARN [0m ==> Found more than one IP address associated to hostname dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5: fd00:100::66de 10.242.102.222, will use fd00:100::66de OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [38;5;6m [38;5;5m22:52:00.35 [0m[38;5;2mINFO [0m ==> ** Elasticsearch setup finished! ** [38;5;6m [38;5;5m22:52:00.41 [0m[38;5;2mINFO [0m ==> ** Starting Elasticsearch ** OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2021-05-27T22:52:04,956][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] version[7.9.3], pid[1], build[oss/tar/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/4.15.0-117-generic/amd64], JVM[BellSoft/OpenJDK 64-Bit Server VM/11.0.9/11.0.9+11-LTS] [2021-05-27T22:52:04,960][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] JVM home [/opt/bitnami/java] [2021-05-27T22:52:04,960][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-15023645191535495870, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true] [2021-05-27T22:52:06,447][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [aggs-matrix-stats] [2021-05-27T22:52:06,448][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [analysis-common] [2021-05-27T22:52:06,448][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [geo] [2021-05-27T22:52:06,448][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [ingest-common] [2021-05-27T22:52:06,449][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [ingest-geoip] [2021-05-27T22:52:06,449][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [ingest-user-agent] [2021-05-27T22:52:06,449][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [kibana] [2021-05-27T22:52:06,450][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [lang-expression] [2021-05-27T22:52:06,450][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [lang-mustache] [2021-05-27T22:52:06,450][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [lang-painless] [2021-05-27T22:52:06,451][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [mapper-extras] [2021-05-27T22:52:06,451][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [parent-join] [2021-05-27T22:52:06,451][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [percolator] [2021-05-27T22:52:06,452][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [rank-eval] [2021-05-27T22:52:06,452][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [reindex] [2021-05-27T22:52:06,452][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [repository-url] [2021-05-27T22:52:06,453][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [tasks] [2021-05-27T22:52:06,453][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded module [transport-netty4] [2021-05-27T22:52:06,454][INFO ][o.e.p.PluginsService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] loaded plugin [repository-s3] [2021-05-27T22:52:06,514][INFO ][o.e.e.NodeEnvironment ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/vda1)]], net usable_space [143.7gb], net total_space [154.8gb], types [ext4] [2021-05-27T22:52:06,514][INFO ][o.e.e.NodeEnvironment ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] heap size [123.7mb], compressed ordinary object pointers [true] [2021-05-27T22:52:06,571][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] node name [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5], node ID [UzzaORNbSzSH6blEN681KQ], cluster name [sdnrdb-cluster] [2021-05-27T22:52:11,372][INFO ][o.e.t.NettyAllocator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=false, g1gc_region_size=0b, heap_size=123.7mb}] [2021-05-27T22:52:11,475][INFO ][o.e.d.DiscoveryModule ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] using discovery type [zen] and seed hosts providers [settings] [2021-05-27T22:52:11,898][WARN ][o.e.g.DanglingIndicesState] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2021-05-27T22:52:12,161][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] initialized [2021-05-27T22:52:12,162][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] starting ... [2021-05-27T22:52:12,362][INFO ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] publish_address {[fd00:100::66de]:9300}, bound_addresses {[::]:9300} [2021-05-27T22:52:12,555][INFO ][o.e.b.BootstrapChecks ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] bound or publishing to a non-loopback address, enforcing bootstrap checks [2021-05-27T22:57:12,578][WARN ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] timed out while waiting for initial discovery state - timeout: 5m [2021-05-27T22:57:12,597][INFO ][o.e.h.AbstractHttpServerTransport] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] publish_address {[fd00:100::66de]:9000}, bound_addresses {[::]:9000} [2021-05-27T22:57:12,599][INFO ][o.e.n.Node ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] started [2021-05-27T23:07:34,395][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [27223ms] ago, timed out [24222ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2676] [2021-05-27T23:07:35,535][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [24420ms] ago, timed out [21418ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2685] [2021-05-27T23:07:35,540][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [9407ms] ago, timed out [6406ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2720] [2021-05-27T23:07:35,541][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [18416ms] ago, timed out [15414ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2699] [2021-05-27T23:07:35,544][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3404ms] ago, timed out [400ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2734] [2021-05-27T23:07:35,544][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [21418ms] ago, timed out [18416ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2692] [2021-05-27T23:07:35,545][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12409ms] ago, timed out [9407ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2713] [2021-05-27T23:07:35,546][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [15414ms] ago, timed out [12409ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2706] [2021-05-27T23:07:35,554][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6406ms] ago, timed out [3404ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2727] [2021-05-27T23:07:57,367][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3005ms] ago, timed out [0ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2794] [2021-05-27T23:08:02,097][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3813ms] ago, timed out [800ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2804] [2021-05-27T23:08:11,466][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6205ms] ago, timed out [3202ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2822] [2021-05-27T23:08:11,497][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3202ms] ago, timed out [200ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2829] [2021-05-27T23:08:17,859][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3611ms] ago, timed out [601ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2845] [2021-05-27T23:08:32,574][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6212ms] ago, timed out [3206ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2879] [2021-05-27T23:08:33,110][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3806ms] ago, timed out [801ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2886] [2021-05-27T23:08:47,849][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3412ms] ago, timed out [409ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2926] [2021-05-27T23:08:54,858][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6406ms] ago, timed out [3404ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2936] [2021-05-27T23:09:05,970][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [8810ms] ago, timed out [5808ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2958] [2021-05-27T23:09:06,159][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6009ms] ago, timed out [3008ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [2965] [2021-05-27T23:09:25,920][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3603ms] ago, timed out [601ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3025] [2021-05-27T23:10:10,365][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [4002ms] ago, timed out [1000ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3148] [2021-05-27T23:10:10,367][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [10010ms] ago, timed out [7005ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3134] [2021-05-27T23:10:10,408][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [7205ms] ago, timed out [4202ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3141] [2021-05-27T23:10:30,668][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [15410ms] ago, timed out [12408ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3172] [2021-05-27T23:10:40,658][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [13411ms] ago, timed out [10409ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3200] [2021-05-27T23:10:40,660][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [19415ms] ago, timed out [16413ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3186] [2021-05-27T23:10:40,661][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [10409ms] ago, timed out [7407ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3207] [2021-05-27T23:10:49,728][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [31424ms] ago, timed out [28422ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3179] [2021-05-27T23:10:51,359][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [18015ms] ago, timed out [15011ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3214] [2021-05-27T23:10:51,360][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [27021ms] ago, timed out [24019ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3193] [2021-05-27T23:10:51,362][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [15011ms] ago, timed out [12009ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3223] [2021-05-27T23:10:51,362][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12009ms] ago, timed out [9007ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3233] [2021-05-27T23:11:01,193][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [9807ms] ago, timed out [6805ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3276] [2021-05-27T23:11:01,194][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [18814ms] ago, timed out [15811ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3246] [2021-05-27T23:11:01,195][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12809ms] ago, timed out [9807ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3266] [2021-05-27T23:11:01,195][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [15811ms] ago, timed out [12809ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3256] [2021-05-27T23:11:01,196][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6805ms] ago, timed out [3803ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3286] [2021-05-27T23:11:01,213][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [4003ms] ago, timed out [1001ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3296] [2021-05-27T23:11:06,717][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3403ms] ago, timed out [400ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3323] [2021-05-27T23:11:06,718][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3403ms] ago, timed out [400ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3321] [2021-05-27T23:11:30,191][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]}, added {{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr},{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}}, term: 1, version: 2, reason: ApplyCommitRequest{term=1, version=2, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-27T23:12:10,434][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [64075ms] ago, timed out [61073ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3332] [2021-05-27T23:12:10,439][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [46047ms] ago, timed out [43045ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3406] [2021-05-27T23:12:15,015][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [68679ms] ago, timed out [65677ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3334] [2021-05-27T23:12:15,016][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [56662ms] ago, timed out [53653ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3376] [2021-05-27T23:12:15,017][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [47649ms] ago, timed out [44647ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3428] [2021-05-27T23:12:42,513][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [84089ms] ago, timed out [81080ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3378] [2021-05-27T23:12:42,514][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [90102ms] ago, timed out [87098ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3354] [2021-05-27T23:12:42,514][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [78078ms] ago, timed out [75076ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3408] [2021-05-27T23:12:42,515][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [90102ms] ago, timed out [87098ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3356] [2021-05-27T23:12:42,515][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [81080ms] ago, timed out [78078ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3390] [2021-05-27T23:12:42,516][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [87298ms] ago, timed out [84289ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3365] [2021-05-27T23:12:42,517][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [81280ms] ago, timed out [78278ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3388] [2021-05-27T23:12:42,517][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [93304ms] ago, timed out [90302ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3345] [2021-05-27T23:12:42,517][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [93304ms] ago, timed out [90302ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3343] [2021-05-27T23:12:42,518][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [87298ms] ago, timed out [84289ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3367] [2021-05-27T23:12:42,518][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [75276ms] ago, timed out [72274ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [3430] [2021-05-27T23:12:44,484][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] added {{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}, term: 1, version: 4, reason: ApplyCommitRequest{term=1, version=4, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-27T23:13:54,908][INFO ][o.e.c.s.ClusterSettings ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] updating [action.auto_create_index] from [true] to [false] [2021-05-27T23:16:00,979][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] removed {{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}, term: 1, version: 11, reason: ApplyCommitRequest{term=1, version=11, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-27T23:24:36,200][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] added {{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}, term: 1, version: 82, reason: ApplyCommitRequest{term=1, version=82, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-27T23:25:36,067][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-27T23:25:36,200][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}], current []}, term: 1, version: 82, reason: becoming candidate: onLeaderFailure [2021-05-27T23:25:46,200][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 1, last-accepted version 83 in term 1 [2021-05-27T23:26:03,542][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]}, term: 2, version: 84, reason: ApplyCommitRequest{term=2, version=84, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-27T23:26:26,562][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] removed {{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}, term: 2, version: 85, reason: ApplyCommitRequest{term=2, version=85, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-27T23:27:53,555][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-27T23:27:53,573][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}], current []}, term: 2, version: 85, reason: becoming candidate: onLeaderFailure [2021-05-27T23:28:03,573][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:28:13,577][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:28:23,580][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:28:33,582][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:28:43,584][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:28:53,587][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:29:03,591][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:29:13,594][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:29:17,980][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /eventlog/_doc, params: {refresh=true, index=eventlog, op_type=create} org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:543) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:408) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:577) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-05-27T23:29:17,987][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /eventlog/_doc, params: {refresh=true, index=eventlog, op_type=create} org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master]; at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:543) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:408) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:577) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-05-27T23:29:23,599][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:29:33,603][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:29:43,604][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:29:53,607][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:30:03,609][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:30:13,611][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:30:23,614][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:30:33,623][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:30:43,625][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:30:53,627][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:31:03,629][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:31:13,634][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:31:23,636][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:31:33,638][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:31:43,640][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:31:53,642][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:32:03,645][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:32:13,648][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:32:23,650][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:32:33,651][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:32:43,653][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:32:53,654][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:33:03,656][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:33:13,658][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:33:23,660][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:33:33,662][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:33:43,667][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:33:53,673][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:34:03,675][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:34:13,677][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:34:23,679][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:34:33,681][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:34:43,683][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:34:53,684][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:35:03,686][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:35:13,689][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:35:23,694][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:35:33,696][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:35:43,698][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:35:53,700][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.170.154:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:36:03,702][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:36:13,708][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:36:23,711][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:36:33,714][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:36:43,717][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:36:53,718][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:37:03,720][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:37:13,722][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:37:23,723][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:37:33,725][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:37:43,727][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:37:53,729][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:38:03,732][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:38:13,735][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:38:23,738][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:38:33,740][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:38:43,742][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:38:53,744][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:03,746][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:13,748][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:23,750][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:33,752][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:43,754][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:53,757][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:39:59,588][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6406ms] ago, timed out [3402ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7670] [2021-05-27T23:40:03,763][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:40:13,765][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:40:23,767][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:40:33,768][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 2, last-accepted version 85 in term 2 [2021-05-27T23:40:34,011][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [15812ms] ago, timed out [12811ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7798] [2021-05-27T23:40:34,012][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [18815ms] ago, timed out [15812ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7778] [2021-05-27T23:40:34,203][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [37027ms] ago, timed out [34025ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7690] [2021-05-27T23:40:34,205][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [22017ms] ago, timed out [19015ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7764] [2021-05-27T23:40:34,315][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [13211ms] ago, timed out [10207ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7813] [2021-05-27T23:40:34,315][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [25219ms] ago, timed out [22217ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7749] [2021-05-27T23:40:34,315][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [4203ms] ago, timed out [1201ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7856] [2021-05-27T23:40:34,316][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [10207ms] ago, timed out [7205ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7827] [2021-05-27T23:40:34,316][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [31223ms] ago, timed out [28222ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7719] [2021-05-27T23:40:34,316][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [7205ms] ago, timed out [4203ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7842] [2021-05-27T23:40:34,316][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [28222ms] ago, timed out [25219ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7734] [2021-05-27T23:40:34,317][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [34225ms] ago, timed out [31223ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7704] [2021-05-27T23:40:40,481][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [5404ms] ago, timed out [2402ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [7881] [2021-05-27T23:41:11,570][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to join {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=3, optionalJoin=Optional[Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=85, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.NotMasterException: Node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] not master for join request [2021-05-27T23:41:13,337][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-27T23:41:23,344][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] last failed join attempt was 11.7s ago, failed to join {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=3, optionalJoin=Optional[Join{term=3, lastAcceptedTerm=2, lastAcceptedVersion=85, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.NotMasterException: Node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] not master for join request [2021-05-27T23:41:23,345][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:41:33,346][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:41:43,349][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:41:53,352][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:42:03,354][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:42:13,356][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:42:23,358][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:42:33,359][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:42:43,361][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:42:53,363][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:43:03,364][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:43:13,366][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:43:23,369][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:43:33,372][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:43:43,374][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:43:53,376][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:44:03,377][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:44:13,379][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.170.154:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:44:23,380][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:44:33,382][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:44:43,384][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:44:53,388][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:45:03,391][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:45:13,394][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300, 10.242.170.154:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:45:23,396][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:45:33,398][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:45:43,400][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.170.154:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] from last-known cluster state; node term 3, last-accepted version 86 in term 3 [2021-05-27T23:45:45,374][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]}, added {{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}, term: 8, version: 87, reason: ApplyCommitRequest{term=8, version=87, sourceNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}} [2021-05-27T23:46:51,195][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] removed {{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}}, term: 8, version: 89, reason: ApplyCommitRequest{term=8, version=89, sourceNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}} [2021-05-27T23:46:51,201][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4016] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,216][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:3893] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,217][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:3770] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,218][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:3399] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,219][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:3647] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,234][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,241][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:3524] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,243][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:5460] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,244][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:5337] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,244][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:5214] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,246][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,249][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,252][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,255][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,258][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,261][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,264][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:5556] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,267][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:5556 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,268][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,278][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,282][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,286][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4016 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,286][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,290][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:5214 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,290][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,293][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:3893 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,293][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,296][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:3770 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,297][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,300][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:3647 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,300][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,302][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:3399 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,304][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:5460 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,305][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,307][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:3524 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,308][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,310][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:5337 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,311][WARN ][r.suppressed ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] path: /mediator-server/_search, params: {index=mediator-server} org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:661) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:703) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:695) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:138) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:79) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:231) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:266) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:400) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:638) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$9.run(TransportService.java:1034) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks at org.elasticsearch.tasks.TaskManager$CancellableTaskHolder.registerChildNode(TaskManager.java:522) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.tasks.TaskManager.registerChildNode(TaskManager.java:213) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:626) ~[elasticsearch-7.9.3.jar:7.9.3] ... 19 more [2021-05-27T23:46:51,312][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4968] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,313][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4968 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,313][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4754] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,313][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4754 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,314][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4630] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,315][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4630 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,316][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4507] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,316][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4507 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,316][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4384] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,317][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4384 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,319][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:5091] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,319][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:5091 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,320][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4261] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,320][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4261 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,320][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4875] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,320][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4875 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:46:51,321][WARN ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Cannot send ban for tasks with the parent [UzzaORNbSzSH6blEN681KQ:4139] to the node [{dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr}] [2021-05-27T23:46:51,321][INFO ][o.e.t.TaskCancellationService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to remove the parent ban for task UzzaORNbSzSH6blEN681KQ:4139 on node {dev-sdnrdb-master-1}{n6lGLXTQQWuKrzbt2tHVuQ}{lIAt6cRnRl-SyYEchLa1UA}{fd00:100:0:0:0:0:0:aa9a}{[fd00:100::aa9a]:9300}{dmr} [2021-05-27T23:50:53,431][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [13614ms] ago, timed out [3604ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9554] [2021-05-27T23:51:06,958][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12412ms] ago, timed out [2401ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9556] [2021-05-27T23:51:28,678][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12610ms] ago, timed out [2603ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9567] [2021-05-27T23:51:48,066][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1073) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:coordination/fault_detection/leader_check] request_id [9569] timed out after [10006ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) ~[elasticsearch-7.9.3.jar:7.9.3] ... 4 more [2021-05-27T23:51:48,076][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], current []}, term: 8, version: 216, reason: becoming candidate: onLeaderFailure [2021-05-27T23:51:58,076][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:52:07,801][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [40632ms] ago, timed out [30621ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9568] [2021-05-27T23:52:07,801][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [29620ms] ago, timed out [19614ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9569] [2021-05-27T23:52:08,077][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:52:18,079][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:52:28,082][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:52:38,083][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:52:42,684][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [5205ms] ago, timed out [2201ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9834] [2021-05-27T23:52:43,877][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [5605ms] ago, timed out [2601ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9837] [2021-05-27T23:52:48,085][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:52:58,088][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:53:08,091][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:53:18,093][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:53:23,480][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [20414ms] ago, timed out [17412ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9959] [2021-05-27T23:53:23,481][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [8406ms] ago, timed out [5404ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10019] [2021-05-27T23:53:23,481][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [11408ms] ago, timed out [8406ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10001] [2021-05-27T23:53:23,481][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [11408ms] ago, timed out [8406ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10000] [2021-05-27T23:53:23,482][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [20414ms] ago, timed out [17412ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9958] [2021-05-27T23:53:23,482][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [29420ms] ago, timed out [26418ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9916] [2021-05-27T23:53:23,482][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [29420ms] ago, timed out [26418ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9917] [2021-05-27T23:53:23,483][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [35424ms] ago, timed out [32421ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9889] [2021-05-27T23:53:23,483][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [32421ms] ago, timed out [29420ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9902] [2021-05-27T23:53:23,483][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [23416ms] ago, timed out [20414ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9944] [2021-05-27T23:53:23,484][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [32421ms] ago, timed out [29420ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9903] [2021-05-27T23:53:23,485][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [14410ms] ago, timed out [11408ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9987] [2021-05-27T23:53:23,485][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [23416ms] ago, timed out [20414ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9945] [2021-05-27T23:53:23,486][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [5404ms] ago, timed out [2402ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10033] [2021-05-27T23:53:23,487][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [17412ms] ago, timed out [14410ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9973] [2021-05-27T23:53:23,487][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [26418ms] ago, timed out [23416ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9931] [2021-05-27T23:53:23,487][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [5404ms] ago, timed out [2402ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10034] [2021-05-27T23:53:23,487][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [14410ms] ago, timed out [11408ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9986] [2021-05-27T23:53:23,488][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [35424ms] ago, timed out [32421ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9888] [2021-05-27T23:53:23,488][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [8406ms] ago, timed out [5404ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10020] [2021-05-27T23:53:23,488][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [26418ms] ago, timed out [23416ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9930] [2021-05-27T23:53:23,488][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [17412ms] ago, timed out [14410ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9972] [2021-05-27T23:53:28,098][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:53:37,463][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to join {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=8, optionalJoin=Optional[Join{term=8, lastAcceptedTerm=3, lastAcceptedVersion=86, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}]} org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:cluster/coordination/join] request_id [9832] timed out after [60044ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-05-27T23:53:37,465][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to join {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=8, optionalJoin=Optional[Join{term=8, lastAcceptedTerm=3, lastAcceptedVersion=86, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}]} org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:cluster/coordination/join] request_id [9832] timed out after [60044ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-05-27T23:53:38,100][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] last failed join attempt was 634ms ago, failed to join {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=8, optionalJoin=Optional[Join{term=8, lastAcceptedTerm=3, lastAcceptedVersion=86, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}}]} org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:cluster/coordination/join] request_id [9832] timed out after [60044ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) ~[elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-05-27T23:53:38,103][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 8, last-accepted version 216 in term 8 [2021-05-27T23:53:55,660][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [23630ms] ago, timed out [20628ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10101] [2021-05-27T23:53:55,662][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [20628ms] ago, timed out [17626ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10116] [2021-05-27T23:53:55,666][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [26632ms] ago, timed out [23630ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10088] [2021-05-27T23:53:55,666][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [17626ms] ago, timed out [14622ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10130] [2021-05-27T23:53:55,667][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [26632ms] ago, timed out [23630ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10087] [2021-05-27T23:53:55,667][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [17626ms] ago, timed out [14622ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10129] [2021-05-27T23:53:55,667][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [20628ms] ago, timed out [17626ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10115] [2021-05-27T23:53:55,668][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [29635ms] ago, timed out [26632ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10073] [2021-05-27T23:53:55,668][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [29635ms] ago, timed out [26632ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10074] [2021-05-27T23:53:55,964][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [24030ms] ago, timed out [21028ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [10102] [2021-05-27T23:54:11,806][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-27T23:54:13,135][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [95684ms] ago, timed out [35640ms] ago, action [internal:cluster/coordination/join], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [9832] [2021-05-27T23:54:21,816][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:54:31,819][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:54:41,821][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:54:51,823][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:55:01,825][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:55:11,827][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:55:21,829][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:55:31,830][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:55:41,832][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:55:51,833][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:56:01,835][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:56:11,837][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:56:21,840][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:56:31,842][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:56:41,844][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:56:51,845][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:57:01,847][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:57:11,849][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:57:19,064][WARN ][o.e.d.HandshakingTransportAddressConnector] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] handshake failed for [connectToRemoteMasterNode[10.242.140.27:9300]] org.elasticsearch.transport.ReceiveTimeoutTransportException: [][10.242.140.27:9300][internal:transport/handshake] request_id [11087] timed out after [1001ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) [elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-05-27T23:57:21,850][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:57:27,032][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [9009ms] ago, timed out [6004ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11088] [2021-05-27T23:57:31,852][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:57:37,882][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [13009ms] ago, timed out [10007ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11119] [2021-05-27T23:57:37,883][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [16012ms] ago, timed out [13009ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11104] [2021-05-27T23:57:37,974][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [7205ms] ago, timed out [4203ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11148] [2021-05-27T23:57:37,976][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [4203ms] ago, timed out [1201ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11162] [2021-05-27T23:57:37,976][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [10207ms] ago, timed out [7205ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11133] [2021-05-27T23:57:41,854][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 10, last-accepted version 217 in term 10 [2021-05-27T23:57:48,171][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6204ms] ago, timed out [3202ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11201] [2021-05-27T23:57:48,258][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3402ms] ago, timed out [400ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11216] [2021-05-27T23:57:48,258][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [9406ms] ago, timed out [6404ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [11187] [2021-05-27T23:58:19,631][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-27T23:58:29,641][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 11, last-accepted version 218 in term 11 [2021-05-27T23:59:04,223][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-27T23:59:14,235][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-27T23:59:24,237][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-27T23:59:34,240][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-27T23:59:44,242][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-27T23:59:54,243][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:00:04,245][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:00:14,807][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:00:28,493][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:00:34,152][WARN ][o.e.m.f.FsHealthService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] health check of [/bitnami/elasticsearch/data/nodes/0] took [17212ms] which is above the warn threshold of [5s] [2021-05-28T00:00:38,496][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:00:48,498][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:01:05,400][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:01:15,429][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:01:25,431][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:01:35,432][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:01:45,434][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:01:55,435][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:02:05,437][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:02:15,439][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:02:25,440][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:02:35,441][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:02:45,443][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:02:55,446][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:03:05,449][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:03:15,451][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:03:25,453][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:03:35,455][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:03:45,457][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:03:55,458][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:04:05,460][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:04:15,461][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:04:25,464][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:04:35,467][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:04:45,469][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:04:55,513][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:05:05,515][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:05:15,516][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 12, last-accepted version 219 in term 12 [2021-05-28T00:05:16,778][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] failed to join {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=13, optionalJoin=Optional[Join{term=13, lastAcceptedTerm=12, lastAcceptedVersion=219, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.NotMasterException: Node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] not master for join request [2021-05-28T00:05:18,319][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]}, term: 17, version: 221, reason: ApplyCommitRequest{term=17, version=221, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-28T00:07:54,158][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-28T00:07:54,177][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}], current []}, term: 17, version: 233, reason: becoming candidate: onLeaderFailure [2021-05-28T00:08:04,173][INFO ][o.e.c.c.JoinHelper ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] last failed join attempt was 2.7m ago, failed to join {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr} with JoinRequest{sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, minimumTerm=13, optionalJoin=Optional[Join{term=13, lastAcceptedTerm=12, lastAcceptedVersion=219, sourceNode={dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, targetNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}}]} org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:cluster/coordination/join] Caused by: org.elasticsearch.cluster.NotMasterException: Node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] not master for join request [2021-05-28T00:08:04,175][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:08:14,177][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:08:24,179][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:08:34,180][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:08:44,182][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:08:54,183][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:09:04,186][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 17, last-accepted version 233 in term 17 [2021-05-28T00:09:07,135][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]}, term: 18, version: 234, reason: ApplyCommitRequest{term=18, version=234, sourceNode={dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}} [2021-05-28T00:11:12,779][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:115) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.RemoteTransportException: [dev-sdnrdb-master-2][[fd00:100::8ad6]:9300][internal:coordination/fault_detection/leader_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check from [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}] sent to a node that is no longer the master at org.elasticsearch.cluster.coordination.LeaderChecker.handleLeaderCheck(LeaderChecker.java:188) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.cluster.coordination.LeaderChecker.lambda$new$0(LeaderChecker.java:113) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:176) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:93) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:78) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:692) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:76) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [2021-05-28T00:11:12,795][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}], current []}, term: 18, version: 234, reason: becoming candidate: onLeaderFailure [2021-05-28T00:11:22,794][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 18, last-accepted version 234 in term 18 [2021-05-28T00:11:32,796][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 18, last-accepted version 234 in term 18 [2021-05-28T00:11:42,798][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 18, last-accepted version 234 in term 18 [2021-05-28T00:11:52,802][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 18, last-accepted version 234 in term 18 [2021-05-28T00:12:02,804][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 18, last-accepted version 234 in term 18 [2021-05-28T00:12:10,380][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]}, term: 19, version: 235, reason: ApplyCommitRequest{term=19, version=235, sourceNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}} [2021-05-28T00:15:05,361][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [13611ms] ago, timed out [3603ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14111] [2021-05-28T00:17:36,765][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12208ms] ago, timed out [2201ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14242] [2021-05-28T00:21:43,759][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [10207ms] ago, timed out [201ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14439] [2021-05-28T00:22:28,309][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1073) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:coordination/fault_detection/leader_check] request_id [14451] timed out after [10012ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) ~[elasticsearch-7.9.3.jar:7.9.3] ... 4 more [2021-05-28T00:22:28,323][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], current []}, term: 19, version: 236, reason: becoming candidate: onLeaderFailure [2021-05-28T00:22:38,321][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:22:48,323][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:22:58,325][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:23:08,327][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:23:18,328][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:23:28,330][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:23:38,115][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [101696ms] ago, timed out [91687ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14444] [2021-05-28T00:23:38,116][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [90686ms] ago, timed out [80678ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14445] [2021-05-28T00:23:38,116][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [79678ms] ago, timed out [69666ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14451] [2021-05-28T00:23:38,331][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 236 in term 19 [2021-05-28T00:23:39,183][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]}, term: 19, version: 237, reason: ApplyCommitRequest{term=19, version=237, sourceNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}} [2021-05-28T00:26:18,766][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [11810ms] ago, timed out [1801ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [14963] [2021-05-28T00:29:23,408][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1073) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:coordination/fault_detection/leader_check] request_id [15057] timed out after [10007ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) ~[elasticsearch-7.9.3.jar:7.9.3] ... 4 more [2021-05-28T00:29:23,419][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], current []}, term: 19, version: 237, reason: becoming candidate: onLeaderFailure [2021-05-28T00:29:33,419][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:29:43,421][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:29:53,423][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:30:03,424][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:30:13,425][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:30:23,428][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:30:33,431][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:30:43,434][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:30:53,435][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:31:03,437][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:31:13,438][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:31:23,440][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:31:33,441][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:31:43,443][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:31:53,444][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:32:03,446][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:32:13,448][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:32:23,451][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:32:33,454][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:32:43,455][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:32:53,457][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:33:03,459][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:33:13,460][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:33:23,462][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:33:33,464][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:33:43,465][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:33:53,468][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:34:03,472][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:34:13,473][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:34:23,475][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:34:33,477][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:34:36,386][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [345087ms] ago, timed out [335080ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [15055] [2021-05-28T00:34:36,387][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [334079ms] ago, timed out [324073ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [15056] [2021-05-28T00:34:36,387][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [323072ms] ago, timed out [313065ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [15057] [2021-05-28T00:34:43,479][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:34:53,480][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:35:03,482][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:35:13,484][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:35:23,486][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:35:33,489][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:35:43,491][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:35:53,493][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:36:03,495][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:36:13,496][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:36:23,498][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:36:33,500][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:36:43,502][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:36:53,503][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:37:03,506][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:37:13,509][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:37:14,582][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [8406ms] ago, timed out [5404ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17051] [2021-05-28T00:37:23,512][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:37:33,515][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:37:43,517][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:37:53,519][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:38:03,520][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:38:13,054][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [63456ms] ago, timed out [60454ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17068] [2021-05-28T00:38:13,056][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [57452ms] ago, timed out [54450ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17102] [2021-05-28T00:38:13,056][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [60454ms] ago, timed out [57452ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17082] [2021-05-28T00:38:13,269][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [24626ms] ago, timed out [21624ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17261] [2021-05-28T00:38:13,269][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [42642ms] ago, timed out [39640ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17174] [2021-05-28T00:38:13,269][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [6606ms] ago, timed out [3602ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17348] [2021-05-28T00:38:13,270][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [18623ms] ago, timed out [15618ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17290] [2021-05-28T00:38:13,270][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [36638ms] ago, timed out [33636ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17203] [2021-05-28T00:38:13,270][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [45644ms] ago, timed out [42642ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17160] [2021-05-28T00:38:13,270][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [9608ms] ago, timed out [6606ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17334] [2021-05-28T00:38:13,271][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [51649ms] ago, timed out [48646ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17131] [2021-05-28T00:38:13,271][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [33636ms] ago, timed out [30633ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17218] [2021-05-28T00:38:13,271][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [15618ms] ago, timed out [12616ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17305] [2021-05-28T00:38:13,278][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [48646ms] ago, timed out [45644ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17145] [2021-05-28T00:38:13,278][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [54650ms] ago, timed out [51649ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17116] [2021-05-28T00:38:13,278][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [27629ms] ago, timed out [24626ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17247] [2021-05-28T00:38:13,283][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [30633ms] ago, timed out [27629ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17232] [2021-05-28T00:38:13,283][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [12616ms] ago, timed out [9608ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17319] [2021-05-28T00:38:13,284][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [21624ms] ago, timed out [18623ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17276] [2021-05-28T00:38:13,284][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [39640ms] ago, timed out [36638ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17189] [2021-05-28T00:38:13,284][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [3602ms] ago, timed out [600ms] ago, action [internal:discovery/request_peers], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17363] [2021-05-28T00:38:13,522][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}, {dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 19, last-accepted version 237 in term 19 [2021-05-28T00:38:14,880][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]}, term: 20, version: 238, reason: ApplyCommitRequest{term=20, version=238, sourceNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}} [2021-05-28T00:40:58,781][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1073) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:coordination/fault_detection/leader_check] request_id [17522] timed out after [10007ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) ~[elasticsearch-7.9.3.jar:7.9.3] ... 4 more [2021-05-28T00:40:58,791][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], current []}, term: 20, version: 238, reason: becoming candidate: onLeaderFailure [2021-05-28T00:41:08,790][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 238 in term 20 [2021-05-28T00:41:18,792][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 238 in term 20 [2021-05-28T00:41:28,794][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 238 in term 20 [2021-05-28T00:41:38,796][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 238 in term 20 [2021-05-28T00:41:48,797][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 238 in term 20 [2021-05-28T00:41:58,572][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [91881ms] ago, timed out [81874ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17520] [2021-05-28T00:41:58,574][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [80873ms] ago, timed out [70864ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17521] [2021-05-28T00:41:58,575][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [69864ms] ago, timed out [59857ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [17522] [2021-05-28T00:41:58,799][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.138.214:9300, 10.242.140.27:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 238 in term 20 [2021-05-28T00:41:59,824][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [], current [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}]}, term: 20, version: 239, reason: ApplyCommitRequest{term=20, version=239, sourceNode={dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}} [2021-05-28T00:45:58,566][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1172) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1073) ~[elasticsearch-7.9.3.jar:7.9.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.3.jar:7.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [dev-sdnrdb-master-0][[fd00:100::8c1b]:9300][internal:coordination/fault_detection/leader_check] request_id [18034] timed out after [10009ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1074) ~[elasticsearch-7.9.3.jar:7.9.3] ... 4 more [2021-05-28T00:45:58,577][INFO ][o.e.c.s.ClusterApplierService] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node changed {previous [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], current []}, term: 20, version: 239, reason: becoming candidate: onLeaderFailure [2021-05-28T00:46:08,575][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:46:15,610][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [49037ms] ago, timed out [39030ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [18032] [2021-05-28T00:46:15,610][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [38030ms] ago, timed out [28023ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [18033] [2021-05-28T00:46:15,611][WARN ][o.e.t.TransportService ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] Received response for a request that has timed out, sent [27023ms] ago, timed out [17014ms] ago, action [internal:coordination/fault_detection/leader_check], node [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}], id [18034] [2021-05-28T00:46:18,577][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:46:28,579][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:46:38,580][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:46:48,583][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:46:58,585][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:47:08,588][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:47:18,590][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:47:28,591][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:47:38,593][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:47:48,594][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:47:58,596][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:48:08,597][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:48:18,598][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master not discovered yet: have discovered [{dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5}{UzzaORNbSzSH6blEN681KQ}{jNjsYQ1IS4-cDRzfC-KDOw}{fd00:100:0:0:0:0:0:66de}{[fd00:100::66de]:9300}{r}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}]; discovery will continue using [10.242.102.222:9300, 10.242.140.27:9300, 10.242.138.214:9300] from hosts providers and [{dev-sdnrdb-master-0}{z9rIOuS-TiK-H0hUqx7AbQ}{M6etPCkeShanKrtwg1s2tA}{fd00:100:0:0:0:0:0:8c1b}{[fd00:100::8c1b]:9300}{dmr}, {dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] from last-known cluster state; node term 20, last-accepted version 239 in term 20 [2021-05-28T00:48:27,635][INFO ][o.e.c.c.Coordinator ] [dev-sdnrdb-coordinating-only-65cbf8cbbd-2jlw5] master node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{dev-sdnrdb-master-2}{FR4Avx6AR6-MuQ2QIxZH_Q}{TRct9nuLQIeFqT24HErW4A}{fd00:100:0:0:0:0:0:8ad6}{[fd00:100::8ad6]:9300}{dmr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java