+ CONSUL_BIND= + [ -n ] + CONSUL_CLIENT= + [ -n ] + CONSUL_DATA_DIR=/consul/data + CONSUL_CONFIG_DIR=/consul/config + [ -n ] + [ a = - ] + [ agent = agent ] + shift + set -- consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui + [ consul = consul ] + stat -c %u /consul/data + id -u consul + [ 100 != 100 ] + stat -c %u /consul/config + id -u consul + [ 100 != 100 ] + [ ! -z ] + exec consul agent -data-dir=/consul/data -config-dir=/consul/config -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-server-0.consul-server -retry-join=dev-consul-server-1.consul-server -retry-join=dev-consul-server-2.consul-server -client=0.0.0.0 -server -ui bootstrap_expect > 0: expecting 3 servers ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.0.6' Node ID: '93531c0c-ceba-3d15-26f7-9a293ac44caa' Node name: 'dev-consul-server-0' Datacenter: 'dc1' (Segment: '') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 10.242.214.7 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2021/06/06 22:42:15 [INFO] raft: Initial configuration (index=0): [] 2021/06/06 22:42:15 [INFO] raft: Node at 10.242.214.7:8300 [Follower] entering Follower state (Leader: "") 2021/06/06 22:42:15 [INFO] serf: EventMemberJoin: dev-consul-server-0.dc1 10.242.214.7 2021/06/06 22:42:15 [INFO] serf: EventMemberJoin: dev-consul-server-0 10.242.214.7 2021/06/06 22:42:15 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2021/06/06 22:42:15 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/06 22:42:15 [INFO] consul: Adding LAN server dev-consul-server-0 (Addr: tcp/10.242.214.7:8300) (DC: dc1) 2021/06/06 22:42:15 [INFO] consul: Handled member-join event for server "dev-consul-server-0.dc1" in area "wan" 2021/06/06 22:42:15 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2021/06/06 22:42:15 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2021/06/06 22:42:15 [INFO] agent: started state syncer 2021/06/06 22:42:15 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os scaleway softlayer 2021/06/06 22:42:15 [INFO] agent: Joining LAN cluster... 2021/06/06 22:42:15 [WARN] memberlist: Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host 2021/06/06 22:42:15 [WARN] memberlist: Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host 2021/06/06 22:42:15 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/06 22:42:15 [INFO] agent: (LAN) joined: 0 Err: 3 error(s) occurred: * Failed to resolve dev-consul-server-0.consul-server: lookup dev-consul-server-0.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-1.consul-server: lookup dev-consul-server-1.consul-server on 10.96.0.10:53: no such host * Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/06 22:42:15 [WARN] agent: Join LAN failed: , retrying in 30s 2021/06/06 22:42:22 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/06 22:42:24 [WARN] raft: no known peers, aborting election 2021/06/06 22:42:29 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 22:42:29 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 22:42:29 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 22:42:29 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 22:42:45 [INFO] agent: (LAN) joining: [dev-consul-server-0.consul-server dev-consul-server-1.consul-server dev-consul-server-2.consul-server] 2021/06/06 22:42:45 [WARN] memberlist: Failed to resolve dev-consul-server-2.consul-server: lookup dev-consul-server-2.consul-server on 10.96.0.10:53: no such host 2021/06/06 22:42:45 [INFO] agent: (LAN) joined: 2 Err: 2021/06/06 22:42:45 [INFO] agent: Join LAN completed. Synced with 2 initial agents 2021/06/06 22:42:46 [ERR] agent: Coordinate update error: No cluster leader ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=3f198c4c-9843-c2e7-6457-6a7c79fc3729&version=1.0.6: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2021/06/06 22:42:54 [ERR] agent: failed to sync remote state: No cluster leader 2021/06/06 22:42:55 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.199.201 2021/06/06 22:42:55 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.199.201:8300) (DC: dc1) 2021/06/06 22:42:55 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.242.214.7:8300,10.242.206.205:8300,10.242.199.201:8300 2021/06/06 22:42:55 [INFO] serf: EventMemberJoin: dev-consul-server-2.dc1 10.242.199.201 2021/06/06 22:42:55 [INFO] consul: Handled member-join event for server "dev-consul-server-2.dc1" in area "wan" 2021/06/06 22:42:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 22:42:55 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/06 22:42:55 [INFO] raft: Node at 10.242.214.7:8300 [Candidate] entering Candidate state in term 2 2021/06/06 22:42:55 [INFO] raft: Election won. Tally: 2 2021/06/06 22:42:55 [INFO] raft: Node at 10.242.214.7:8300 [Leader] entering Leader state 2021/06/06 22:42:55 [INFO] raft: Added peer 73baf299-7352-1f00-9c60-ca943e6125a5, starting replication 2021/06/06 22:42:55 [INFO] raft: Added peer 6ca43584-e629-15ec-4c7c-bfaff3bd0044, starting replication 2021/06/06 22:42:55 [INFO] consul: cluster leadership acquired 2021/06/06 22:42:55 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/06 22:42:55 [WARN] raft: AppendEntries to {Voter 6ca43584-e629-15ec-4c7c-bfaff3bd0044 10.242.199.201:8300} rejected, sending older logs (next: 1) 2021/06/06 22:42:55 [WARN] raft: AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} rejected, sending older logs (next: 1) 2021/06/06 22:42:55 [INFO] raft: pipelining replication to peer {Voter 6ca43584-e629-15ec-4c7c-bfaff3bd0044 10.242.199.201:8300} 2021/06/06 22:42:55 [INFO] raft: pipelining replication to peer {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} 2021/06/06 22:42:55 [INFO] consul: member 'dev-consul-server-2' joined, marking health alive 2021/06/06 22:42:55 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' joined, marking health alive 2021/06/06 22:42:55 [INFO] consul: member 'dev-consul-server-0' joined, marking health alive 2021/06/06 22:42:55 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/06 22:42:57 [INFO] agent: Synced node info 2021/06/06 23:03:36 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:53898->10.242.206.200:8301: i/o timeout 2021/06/06 23:03:36 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:03:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:53926->10.242.206.200:8301: i/o timeout 2021/06/06 23:03:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:03:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:53996->10.242.206.200:8301: i/o timeout 2021/06/06 23:03:49 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:03:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54020->10.242.206.200:8301: i/o timeout 2021/06/06 23:03:53 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:04:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54090->10.242.206.200:8301: i/o timeout 2021/06/06 23:04:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:04:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54304->10.242.206.200:8301: i/o timeout 2021/06/06 23:04:37 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:04:38 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/06 23:04:38 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/06 23:04:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54320->10.242.206.200:8301: i/o timeout 2021/06/06 23:04:40 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:04:40 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500228493s 2021/06/06 23:04:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:44 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500223056s 2021/06/06 23:04:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54354->10.242.206.200:8301: i/o timeout 2021/06/06 23:04:45 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:04:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:60788->10.242.206.205:8301: i/o timeout 2021/06/06 23:04:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:04:47 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 4.997693073s 2021/06/06 23:04:47 [INFO] raft: aborting pipeline replication to peer {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} 2021/06/06 23:04:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:48 [INFO] memberlist: Marking dev-consul-7467bf7b46-wmc2v as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:04:48 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:04:48 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' failed, marking health critical 2021/06/06 23:04:49 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:60806->10.242.206.205:8301: i/o timeout 2021/06/06 23:04:49 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:04:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:49 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500197958s 2021/06/06 23:04:51 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:60820->10.242.206.205:8301: i/o timeout 2021/06/06 23:04:51 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:04:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:52 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 4.99491588s 2021/06/06 23:04:53 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:59872->10.242.206.205:8300: i/o timeout 2021/06/06 23:04:53 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:04:53 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:04:53 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:04:53 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/06 23:04:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:53 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:04:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:60846->10.242.206.205:8301: i/o timeout 2021/06/06 23:04:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:04:54 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 7.485480561s 2021/06/06 23:04:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:44328->10.242.206.205:8302: i/o timeout 2021/06/06 23:04:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:04:56 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/06 23:04:56 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/06 23:04:57 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:39738->10.242.206.205:8300: i/o timeout 2021/06/06 23:04:57 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:04:57 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:04:57 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:04:57 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/06 23:04:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:04:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:60878->10.242.206.205:8301: i/o timeout 2021/06/06 23:04:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:04:59 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:04:59 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' joined, marking health alive 2021/06/06 23:04:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:00 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.50024844s 2021/06/06 23:05:01 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/06 23:05:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:02 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.214.7:8300 2021/06/06 23:05:02 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.206.205:59134 2021/06/06 23:05:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54478->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:03 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54486->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:06 [INFO] raft: pipelining replication to peer {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} 2021/06/06 23:05:16 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54568->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:16 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:24 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/06 23:05:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:32814->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:26 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:26 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500238068s 2021/06/06 23:05:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:29 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 4.970118448s 2021/06/06 23:05:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54628->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:44552->10.242.206.205:8302: i/o timeout 2021/06/06 23:05:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:05:31 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 7.437998529s 2021/06/06 23:05:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:32 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:32848->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:32 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:34 [INFO] raft: aborting pipeline replication to peer {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} 2021/06/06 23:05:34 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54676->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:34 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:34 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:39784->10.242.206.205:8300: i/o timeout 2021/06/06 23:05:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:36 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500209796s 2021/06/06 23:05:37 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:32900->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:37 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:37 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:05:37 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:05:37 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:05:37 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/06 23:05:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:39 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 4.962552441s 2021/06/06 23:05:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54716->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:39 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:44632->10.242.206.205:8302: i/o timeout 2021/06/06 23:05:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:05:40 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:05:40 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' failed, marking health critical 2021/06/06 23:05:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54750->10.242.206.200:8301: i/o timeout 2021/06/06 23:05:41 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:05:41 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 7.442390452s 2021/06/06 23:05:41 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:43 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:44 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:40048->10.242.206.205:8300: i/o timeout 2021/06/06 23:05:44 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:05:45 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:05:45 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40054->10.242.206.205:8300: i/o timeout 2021/06/06 23:05:45 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:05:45 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:05:45 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/06 23:05:45 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:32992->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:47 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:49 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33008->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:51 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:53 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33034->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:53 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:54 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:40136->10.242.206.205:8300: i/o timeout 2021/06/06 23:05:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:44756->10.242.206.205:8302: i/o timeout 2021/06/06 23:05:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:05:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33048->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:56 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40154->10.242.206.205:8300: i/o timeout 2021/06/06 23:05:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33054->10.242.206.205:8301: i/o timeout 2021/06/06 23:05:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:05:57 [WARN] memberlist: Refuting a suspect message (from: dev-consul-7467bf7b46-wmc2v) 2021/06/06 23:05:57 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:05:57 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' joined, marking health alive 2021/06/06 23:05:57 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:05:58 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:05:58 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:05:58 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:05:58 [INFO] consul: member 'dev-consul-server-1' failed, marking health critical 2021/06/06 23:05:59 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33080->10.242.206.205:8301: i/o timeout 2021/06/06 23:06:00 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:06:01 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54898->10.242.206.200:8301: i/o timeout 2021/06/06 23:06:03 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:06:04 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:40212->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:04 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:04 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/06 23:06:04 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/06 23:06:05 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured 2021/06/06 23:06:05 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54934->10.242.206.200:8301: i/o timeout 2021/06/06 23:06:06 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:06:07 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40234->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:07 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:07 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54942->10.242.206.200:8301: i/o timeout 2021/06/06 23:06:08 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:06:08 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:09 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:09 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:11 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:54968->10.242.206.200:8301: i/o timeout 2021/06/06 23:06:11 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:06:11 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500224269s 2021/06/06 23:06:11 [INFO] memberlist: Marking dev-consul-7467bf7b46-wmc2v as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:06:11 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:06:11 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' failed, marking health critical 2021/06/06 23:06:11 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:13 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:13 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:14 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:40286->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:14 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:15 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:15 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:16 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:17 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:17 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:18 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.500186351s 2021/06/06 23:06:19 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/06 23:06:21 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 4.922430305s 2021/06/06 23:06:21 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:23 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 7.375407904s 2021/06/06 23:06:23 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:24 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:40366->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:24 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:44978->10.242.206.205:8302: i/o timeout 2021/06/06 23:06:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:06:25 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:06:25 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:27 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40314->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:27 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:28 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:29 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:31 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:33 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:34 [ERR] raft: Failed to AppendEntries to {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300}: read tcp 10.242.214.7:40436->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:34 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:35 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:37 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:38 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40462->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:38 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:39 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:40 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:06:40 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:06:40 [WARN] consul: error getting server health from "dev-consul-server-1": rpc error making call: EOF 2021/06/06 23:06:40 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.206.205:38440 2021/06/06 23:06:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45052->10.242.206.205:8302: i/o timeout 2021/06/06 23:06:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:06:41 [ERR] raft: peer {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} has newer term, stopping replication 2021/06/06 23:06:41 [INFO] raft: Node at 10.242.214.7:8300 [Follower] entering Follower state (Leader: "") 2021/06/06 23:06:41 [INFO] raft: aborting pipeline replication to peer {Voter 6ca43584-e629-15ec-4c7c-bfaff3bd0044 10.242.199.201:8300} 2021/06/06 23:06:41 [INFO] consul: cluster leadership lost 2021/06/06 23:06:48 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40536->10.242.206.205:8300: i/o timeout 2021/06/06 23:06:49 [WARN] raft: Heartbeat timeout from "" reached, starting election 2021/06/06 23:06:49 [INFO] raft: Node at 10.242.214.7:8300 [Candidate] entering Candidate state in term 3 2021/06/06 23:06:49 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:49 [INFO] raft: Election won. Tally: 2 2021/06/06 23:06:49 [INFO] raft: Node at 10.242.214.7:8300 [Leader] entering Leader state 2021/06/06 23:06:49 [INFO] raft: Added peer 73baf299-7352-1f00-9c60-ca943e6125a5, starting replication 2021/06/06 23:06:49 [INFO] raft: Added peer 6ca43584-e629-15ec-4c7c-bfaff3bd0044, starting replication 2021/06/06 23:06:49 [INFO] consul: cluster leadership acquired 2021/06/06 23:06:49 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:49 [INFO] consul: New leader elected: dev-consul-server-0 2021/06/06 23:06:49 [INFO] raft: pipelining replication to peer {Voter 6ca43584-e629-15ec-4c7c-bfaff3bd0044 10.242.199.201:8300} 2021/06/06 23:06:50 [WARN] Unable to get address for server id 73baf299-7352-1f00-9c60-ca943e6125a5, using fallback address 10.242.206.205:8300: Could not find address for server id 73baf299-7352-1f00-9c60-ca943e6125a5 2021/06/06 23:06:50 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.214.7:8300 2021/06/06 23:06:52 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 2.501420309s 2021/06/06 23:06:52 [WARN] consul: error getting server health from "dev-consul-server-1": context deadline exceeded 2021/06/06 23:06:52 [WARN] consul: error getting server health from "dev-consul-server-2": context deadline exceeded 2021/06/06 23:06:53 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:54 [WARN] raft: Failed to contact 73baf299-7352-1f00-9c60-ca943e6125a5 in 4.991600335s 2021/06/06 23:06:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45148->10.242.206.205:8302: i/o timeout 2021/06/06 23:06:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:06:55 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:06:55 [INFO] consul: member 'dev-consul-7467bf7b46-wmc2v' joined, marking health alive 2021/06/06 23:06:55 [WARN] consul: error getting server health from "dev-consul-server-1": last request still outstanding 2021/06/06 23:06:55 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.214.7:8300 2021/06/06 23:06:55 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:06:55 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:06:55 [INFO] consul: member 'dev-consul-server-1' joined, marking health alive 2021/06/06 23:06:56 [ERR] raft: peer {Voter 73baf299-7352-1f00-9c60-ca943e6125a5 10.242.206.205:8300} has newer term, stopping replication 2021/06/06 23:06:56 [INFO] raft: Node at 10.242.214.7:8300 [Follower] entering Follower state (Leader: "") 2021/06/06 23:06:56 [INFO] raft: aborting pipeline replication to peer {Voter 6ca43584-e629-15ec-4c7c-bfaff3bd0044 10.242.199.201:8300} 2021/06/06 23:06:56 [INFO] consul: cluster leadership lost 2021/06/06 23:06:57 [WARN] memberlist: Was able to connect to dev-consul-server-1 but other probes failed, network may be misconfigured 2021/06/06 23:06:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:55312->10.242.206.200:8301: i/o timeout 2021/06/06 23:06:59 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:06:59 [WARN] consul.coordinate: Batch update failed: node is not the leader 2021/06/06 23:07:00 [ERR] raft: Failed to heartbeat to 10.242.206.205:8300: read tcp 10.242.214.7:40610->10.242.206.205:8300: i/o timeout 2021/06/06 23:07:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33510->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:03 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33532->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:03 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:55348->10.242.206.200:8301: i/o timeout 2021/06/06 23:07:05 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:07:06 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.199.201:47260 2021/06/06 23:07:06 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/06 23:07:07 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.199.201:47288 2021/06/06 23:07:08 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33562->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:08 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:09 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:07:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45286->10.242.206.205:8302: i/o timeout 2021/06/06 23:07:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:07:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:55378->10.242.206.200:8301: i/o timeout 2021/06/06 23:07:10 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:07:12 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33594->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:12 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:12 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:07:12 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:07:12 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:07:14 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33608->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:14 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45362->10.242.206.205:8302: i/o timeout 2021/06/06 23:07:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:07:30 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured 2021/06/06 23:07:35 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.206.205:8301 2021/06/06 23:07:37 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:07:37 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:07:39 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33768->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:39 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:41 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33772->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:41 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33816->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45538->10.242.206.205:8302: i/o timeout 2021/06/06 23:07:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:07:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33836->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:45 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:48 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33858->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:48 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:33868->10.242.206.205:8301: i/o timeout 2021/06/06 23:07:50 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:07:50 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:07:50 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:07:50 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:07:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45614->10.242.206.205:8302: i/o timeout 2021/06/06 23:07:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:08:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45676->10.242.206.205:8302: i/o timeout 2021/06/06 23:08:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:08:06 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:08:15 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/06 23:08:15 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:08:15 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:08:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45712->10.242.206.205:8302: i/o timeout 2021/06/06 23:08:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:08:15 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:08:21 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/06 23:08:28 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:08:35 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:08:35 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:08:36 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:08:36 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.206.200:33816 2021/06/06 23:08:45 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:45956->10.242.206.205:8302: i/o timeout 2021/06/06 23:08:45 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:08:45 [ERR] memberlist: failed to receive: read tcp 10.242.214.7:8301->10.242.206.205:44466: i/o timeout from=10.242.206.205:44466 2021/06/06 23:08:45 [ERR] memberlist: Failed to send error: write tcp 10.242.214.7:8301->10.242.206.205:44466: i/o timeout from=10.242.206.205:44466 2021/06/06 23:08:46 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:08:55 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/06 23:08:56 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:08:56 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:08:57 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:09:01 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:34348->10.242.206.205:8301: i/o timeout 2021/06/06 23:09:01 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:09:04 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:34372->10.242.206.205:8301: i/o timeout 2021/06/06 23:09:04 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:09:05 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/06 23:09:05 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:09:05 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:09:06 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:34382->10.242.206.205:8301: i/o timeout 2021/06/06 23:09:06 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:09:10 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:46120->10.242.206.205:8302: i/o timeout 2021/06/06 23:09:10 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:09:17 [INFO] serf: EventMemberFailed: dev-consul-server-2 10.242.199.201 2021/06/06 23:09:17 [INFO] consul: Removing LAN server dev-consul-server-2 (Addr: tcp/10.242.199.201:8300) (DC: dc1) 2021/06/06 23:09:17 [INFO] serf: EventMemberJoin: dev-consul-server-2 10.242.199.201 2021/06/06 23:09:17 [INFO] consul: Adding LAN server dev-consul-server-2 (Addr: tcp/10.242.199.201:8300) (DC: dc1) 2021/06/06 23:09:18 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:09:19 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-wmc2v but other probes failed, network may be misconfigured 2021/06/06 23:09:20 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:46158->10.242.206.205:8302: i/o timeout 2021/06/06 23:09:20 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:09:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:56292->10.242.206.200:8301: i/o timeout 2021/06/06 23:09:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:09:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:56314->10.242.206.200:8301: i/o timeout 2021/06/06 23:09:24 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:09:26 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:56322->10.242.206.200:8301: i/o timeout 2021/06/06 23:09:26 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:09:26 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.206.205:8301 2021/06/06 23:09:29 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:56350->10.242.206.200:8301: i/o timeout 2021/06/06 23:09:29 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:09:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:46256->10.242.206.205:8302: i/o timeout 2021/06/06 23:09:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:09:31 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:56354->10.242.206.200:8301: i/o timeout 2021/06/06 23:09:31 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:09:32 [INFO] memberlist: Marking dev-consul-7467bf7b46-wmc2v as failed, suspect timeout reached (1 peer confirmations) 2021/06/06 23:09:32 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:09:33 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:56380->10.242.206.200:8301: i/o timeout 2021/06/06 23:09:33 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:09:35 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:09:35 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.206.200:36572 2021/06/06 23:09:35 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/06 23:09:35 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:09:35 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:09:40 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:46326->10.242.206.205:8302: i/o timeout 2021/06/06 23:09:40 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:09:45 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:10:06 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.206.205:8301 2021/06/06 23:10:11 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:10:11 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.206.205:60846 2021/06/06 23:10:12 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:10:17 [ERR] memberlist: failed to receive: read tcp 10.242.214.7:8302->10.242.206.205:40024: i/o timeout from=10.242.206.205:40024 2021/06/06 23:10:17 [ERR] memberlist: Failed to send error: write tcp 10.242.214.7:8302->10.242.206.205:40024: i/o timeout from=10.242.206.205:40024 2021/06/06 23:10:45 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:10:46 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.206.205:8301 2021/06/06 23:10:51 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:10:51 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1.dc1) 2021/06/06 23:10:51 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:10:51 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:10:53 [ERR] memberlist: failed to receive: read tcp 10.242.214.7:8302->10.242.206.205:40148: i/o timeout from=10.242.206.205:40148 2021/06/06 23:10:53 [ERR] memberlist: Failed to send error: write tcp 10.242.214.7:8302->10.242.206.205:40148: i/o timeout from=10.242.206.205:40148 2021/06/06 23:10:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:35092->10.242.206.205:8301: i/o timeout 2021/06/06 23:10:55 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:10:57 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:35104->10.242.206.205:8301: i/o timeout 2021/06/06 23:10:57 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:10:58 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:10:58 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:10:59 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:35114->10.242.206.205:8301: i/o timeout 2021/06/06 23:10:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:11:21 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:11:45 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:11:49 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:11:49 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:11:55 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:47226->10.242.206.205:8302: i/o timeout 2021/06/06 23:11:55 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:11:57 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:12:01 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:12:05 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:47294->10.242.206.205:8302: i/o timeout 2021/06/06 23:12:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:12:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:47358->10.242.206.205:8302: i/o timeout 2021/06/06 23:12:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:12:21 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:12:25 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/06 23:12:25 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:12:25 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:12:30 [WARN] memberlist: Was able to connect to dev-consul-server-1.dc1 but other probes failed, network may be misconfigured 2021/06/06 23:12:41 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.206.205:8301 2021/06/06 23:12:45 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:13:14 [ERR] yamux: keepalive failed: i/o deadline reached 2021/06/06 23:13:14 [ERR] consul.rpc: multiplex conn accept failed: keepalive timeout from=10.242.206.200:38168 2021/06/06 23:13:21 [INFO] serf: attempting reconnect to dev-consul-server-1 10.242.206.205:8301 2021/06/06 23:13:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-8lcxh 10.242.199.229 2021/06/06 23:13:45 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:14:06 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:14:22 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:14:22 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:14:25 [ERR] memberlist: failed to receive: read tcp 10.242.214.7:8301->10.242.206.205:46672: i/o timeout from=10.242.206.205:46672 2021/06/06 23:14:25 [ERR] memberlist: Failed to send error: write tcp 10.242.214.7:8301->10.242.206.205:46672: i/o timeout from=10.242.206.205:46672 2021/06/06 23:14:30 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:48360->10.242.206.205:8302: i/o timeout 2021/06/06 23:14:30 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:14:31 [INFO] serf: attempting reconnect to dev-consul-7467bf7b46-wmc2v 10.242.206.200:8301 2021/06/06 23:14:35 [ERR] memberlist: failed to receive: read tcp 10.242.214.7:8302->10.242.206.205:41436: i/o timeout from=10.242.206.205:41436 2021/06/06 23:14:35 [ERR] memberlist: Failed to send error: write tcp 10.242.214.7:8302->10.242.206.205:41436: i/o timeout from=10.242.206.205:41436 2021/06/06 23:14:41 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:14:41 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:14:44 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:36806->10.242.206.205:8301: i/o timeout 2021/06/06 23:14:44 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:14:47 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:36812->10.242.206.205:8301: i/o timeout 2021/06/06 23:14:47 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:14:48 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/06 23:14:48 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:14:48 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:14:50 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:48560->10.242.206.205:8302: i/o timeout 2021/06/06 23:14:50 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:14:50 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:15:00 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:48634->10.242.206.205:8302: i/o timeout 2021/06/06 23:15:00 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:15:13 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:15:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:48882->10.242.206.205:8302: i/o timeout 2021/06/06 23:15:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:15:15 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:58986->10.242.206.200:8301: i/o timeout 2021/06/06 23:15:15 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:15:16 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/06 23:15:16 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.199.201:56118 2021/06/06 23:15:18 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:59032->10.242.206.200:8301: i/o timeout 2021/06/06 23:15:18 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:15:18 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:15:18 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:15:19 [INFO] memberlist: Marking dev-consul-7467bf7b46-wmc2v as failed, suspect timeout reached (2 peer confirmations) 2021/06/06 23:15:19 [INFO] serf: EventMemberFailed: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:15:21 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:59052->10.242.206.200:8301: i/o timeout 2021/06/06 23:15:21 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:15:24 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:37270->10.242.206.205:8301: i/o timeout 2021/06/06 23:15:24 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:15:24 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:15:24 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:15:25 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/06 23:15:25 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:49000->10.242.206.205:8302: i/o timeout 2021/06/06 23:15:25 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:15:25 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.199.201:56328 2021/06/06 23:15:29 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:15:30 [WARN] memberlist: Refuting a suspect message (from: dev-consul-server-1) 2021/06/06 23:15:31 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:15:31 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:15:32 [INFO] serf: EventMemberJoin: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:15:38 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:59196->10.242.206.200:8301: i/o timeout 2021/06/06 23:15:38 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:15:38 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/06 23:15:39 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.199.201:56642 2021/06/06 23:15:40 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:15:43 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:37434->10.242.206.205:8301: i/o timeout 2021/06/06 23:15:43 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:15:46 [WARN] raft: Heartbeat timeout from "10.242.199.201:8300" reached, starting election 2021/06/06 23:15:46 [INFO] raft: Node at 10.242.214.7:8300 [Candidate] entering Candidate state in term 8 2021/06/06 23:15:46 [INFO] raft: Node at 10.242.214.7:8300 [Follower] entering Follower state (Leader: "") 2021/06/06 23:15:49 [INFO] consul: New leader elected: dev-consul-server-2 2021/06/06 23:15:49 [WARN] raft: Rejecting vote request from 10.242.206.205:8300 since we have a leader: 10.242.199.201:8300 2021/06/06 23:15:49 [DEBUG] raft-net: 10.242.214.7:8300 accepted connection from: 10.242.199.201:56862 2021/06/06 23:15:54 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:37528->10.242.206.205:8301: i/o timeout 2021/06/06 23:15:54 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:15:57 [WARN] memberlist: Was able to connect to dev-consul-7467bf7b46-wmc2v but other probes failed, network may be misconfigured 2021/06/06 23:15:58 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:37572->10.242.206.205:8301: read: connection reset by peer 2021/06/06 23:15:59 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:16:00 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/06 23:16:00 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:16:00 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:16:02 [ERR] memberlist: Failed fallback ping: read tcp 10.242.214.7:59406->10.242.206.200:8301: i/o timeout 2021/06/06 23:16:02 [INFO] memberlist: Suspect dev-consul-7467bf7b46-wmc2v has failed, no acks received 2021/06/06 23:16:05 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:16:06 [INFO] serf: EventMemberLeave: dev-consul-7467bf7b46-wmc2v 10.242.206.200 2021/06/06 23:16:08 [INFO] serf: EventMemberUpdate: dev-consul-server-1.dc1 2021/06/06 23:16:08 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.205 2021/06/06 23:16:08 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:16:15 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:16:21 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:16:25 [INFO] memberlist: Marking dev-consul-server-1 as failed, suspect timeout reached (2 peer confirmations) 2021/06/06 23:16:25 [INFO] serf: EventMemberFailed: dev-consul-server-1 10.242.206.205 2021/06/06 23:16:25 [INFO] consul: Removing LAN server dev-consul-server-1 (Addr: tcp/10.242.206.205:8300) (DC: dc1) 2021/06/06 23:16:33 [INFO] memberlist: Suspect dev-consul-server-1 has failed, no acks received 2021/06/06 23:16:39 [ERR] memberlist: Conflicting address for dev-consul-server-1. Mine: 10.242.206.205:8301 Theirs: 10.242.206.237:8301 2021/06/06 23:16:39 [WARN] serf: Name conflict for 'dev-consul-server-1' both 10.242.206.205:8301 and 10.242.206.237:8301 are claiming 2021/06/06 23:16:39 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.206.205:8302 Theirs: 10.242.206.237:8302 2021/06/06 23:16:39 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.206.205:8302 and 10.242.206.237:8302 are claiming 2021/06/06 23:16:39 [ERR] memberlist: Conflicting address for dev-consul-server-1. Mine: 10.242.206.205:8301 Theirs: 10.242.206.237:8301 2021/06/06 23:16:39 [WARN] serf: Name conflict for 'dev-consul-server-1' both 10.242.206.205:8301 and 10.242.206.237:8301 are claiming 2021/06/06 23:16:39 [ERR] memberlist: Conflicting address for dev-consul-server-1. Mine: 10.242.206.205:8301 Theirs: 10.242.206.237:8301 2021/06/06 23:16:39 [WARN] serf: Name conflict for 'dev-consul-server-1' both 10.242.206.205:8301 and 10.242.206.237:8301 are claiming 2021/06/06 23:16:39 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.206.205:8302 Theirs: 10.242.206.237:8302 2021/06/06 23:16:39 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.206.205:8302 and 10.242.206.237:8302 are claiming 2021/06/06 23:16:40 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.206.205:8302 Theirs: 10.242.206.237:8302 2021/06/06 23:16:40 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.206.205:8302 and 10.242.206.237:8302 are claiming 2021/06/06 23:16:53 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:17:23 [INFO] memberlist: Marking dev-consul-server-1.dc1 as failed, suspect timeout reached (0 peer confirmations) 2021/06/06 23:17:23 [INFO] serf: EventMemberFailed: dev-consul-server-1.dc1 10.242.206.205 2021/06/06 23:17:23 [INFO] consul: Handled member-failed event for server "dev-consul-server-1.dc1" in area "wan" 2021/06/06 23:17:24 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.206.205:8302 Theirs: 10.242.206.237:8302 2021/06/06 23:17:24 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.206.205:8302 and 10.242.206.237:8302 are claiming 2021/06/06 23:17:25 [ERR] memberlist: Conflicting address for dev-consul-server-1.dc1. Mine: 10.242.206.205:8302 Theirs: 10.242.206.237:8302 2021/06/06 23:17:25 [WARN] serf: Name conflict for 'dev-consul-server-1.dc1' both 10.242.206.205:8302 and 10.242.206.237:8302 are claiming 2021/06/06 23:17:33 [INFO] memberlist: Suspect dev-consul-server-1.dc1 has failed, no acks received 2021/06/06 23:17:36 [INFO] serf: EventMemberJoin: dev-consul-server-1 10.242.206.237 2021/06/06 23:17:36 [INFO] consul: Adding LAN server dev-consul-server-1 (Addr: tcp/10.242.206.237:8300) (DC: dc1) 2021/06/06 23:18:45 [INFO] serf: attempting reconnect to dev-consul-server-1.dc1 10.242.206.205:8302 2021/06/06 23:19:09 [INFO] serf: EventMemberJoin: dev-consul-server-1.dc1 10.242.206.237 2021/06/06 23:19:09 [INFO] consul: Handled member-join event for server "dev-consul-server-1.dc1" in area "wan"